Network storage

Upcoming. This page describes the master branch. Network storage is not part of release 0.0.8, and details may change.

Hedgehog can store a file on the Unigrid network without trusting any single machine with it. The file is encrypted on your own node, cut into small pieces and spread over the network’s gridnodes, which are the nodes that announce themselves as available servers. One random secret, the fingerprint, is all you need to read or delete the file later, and the network keeps no record of who stored what.

How it works

Storing. storage-put hands the file to your local daemon. The daemon draws a fresh fingerprint, encrypts the file with keys derived from it, and adds redundancy with an erasure code, so that a file can be rebuilt from only part of what was stored. The pieces are signed and sent to gridnodes, and the fingerprint is printed. How the coding works is the subject of Erasure coding.

Finding the pieces. Every piece belongs to a group, and a group’s name is derived from the fingerprint. Each gridnode scores itself against that name with a hash, and the highest scorers hold the group. Any node with the same list of gridnodes computes the same answer, so there is no index or directory to keep, and the groups of one file land on unrelated sets of gridnodes.

Reading. Given the fingerprint, the daemon works out where the file’s description is stored, fetches it, and then collects pieces group by group. Every piece is checked against a signature and a Merkle proof before it is used, so a read returns the stored bytes or fails; it never returns other bytes.

Staying alive without the owner. Gridnodes check the groups they hold once an hour and take turns doing it. When a group has lost enough pieces, a gridnode rebuilds the missing ones from the survivors and hands them to free gridnodes. It needs no key to do this, and it cannot produce a piece the owner did not sign.

Deleting. Only the fingerprint yields the keys that sign a valid delete. Gridnodes drop the pieces and keep the signed delete as a tombstone for 30 days, which stops stale copies from being stored or repaired again. Tombstones also spread through the repair checks, so a gridnode that was offline during the delete learns of it later.

What the design guarantees

• Confidentiality. Gridnodes only ever hold AES-256-GCM ciphertext, or redundancy computed from it, and never the key.
• Integrity. Forged or damaged pieces are rejected and counted as missing.
• Durability. A group survives the loss of any 8 of its 24 guaranteed pieces. A full stripe of 32 data chunks also survives the loss of any 16 of its 48 groups, and a file keeps three copies of its description.
• Delete authority. Nobody without the fingerprint can read or delete a file. The holders of the network’s spork keys cannot either; they only set storage parameters.

Key facts

Fact	Default
Chunk size and fragment size	1 MiB and 64 KiB
Pieces per group	24 guaranteed, plus up to 8 extras
Repair starts when a group is down to	20 distinct pieces
Gridnodes needed to accept an upload	at least 24 active
Gridnodes ranked per group	40 (32 pieces plus 8 spares)
Quota per gridnode	10 GiB, of which extras may use 20 %
Repair interval	60 minutes
Tombstone lifetime	30 days
Largest upload through the REST interface	64 GiB
Fingerprint	32 random bytes, written as 50 characters

Every figure except the upload limit and the fingerprint format is a default of the storage spork, a signed network parameter that two network keys must agree on. It is described in Grid sporks. Storage stays switched off until that spork exists. Changing it affects new uploads, and repair and quotas for old files too.

Using it

The daemon exposes POST, GET and DELETE /storage on its REST interface, and the command line wraps them:

hedgehog cli storage-put ./report.pdf          # prints the fingerprint
hedgehog cli storage-get -f -o ./report-copy.pdf
hedgehog cli storage-delete -f

With -f and no value the command prompts for the fingerprint without echo, which keeps it out of shell history. Pieces travel over the existing peer connections described in Peer-to-peer network protocol.

What it is not, today

• There are no accounts, payments or per-user limits, and nothing expires. A file stays stored, and repaired, until its fingerprint holder deletes it. A lost fingerprint means a lost file.
• A fingerprint grants everything, delete included. There is no read-only form.
• Gridnodes do not yet lock collateral, so any key can announce itself as a gridnode.
• The S3-compatible /bucket and /storage-object endpoints are a separate feature. They keep plain files in the local data directory and do not use this network storage.

In the source

Paths are relative to application/src/main/java/org/unigrid/hedgehog/.

• service/storage/StorageService.java is the entry point for store, read and delete.
• service/storage/StorageUpload.java and service/storage/Retrieval.java handle one upload and one read.
• model/storage/placement/Placement.java holds the whole of the gridnode ranking.
• service/storage/FragmentKeeper.java is the gridnode side: it decides what to accept and serve.
• service/storage/GroupRepairer.java is the self-healing round.
• model/spork/StorageSpork.java defines the parameters above.
• service/storage/StorageNetworkTest.java in the test tree runs twenty daemons storing, losing, repairing and deleting a file; see Build, testing and native image.

The design rationale and durability analysis are in documentation/storage-white-paper/sharded-and-redundant-storage.tex. For the surrounding system, start at Home or the Architecture overview.