Legacy chain snapshot

Before Hedgehog existed, the Unigrid foundation ran an earlier blockchain, and its block files are the only record of who held what. The legacy chain snapshot turns that history into one compact, signed file that a Hedgehog node can download, check against the board keys and use to answer balance and transaction-history questions, without anyone having to trust the host that served the file.

How it works

The snapshot is produced once per release by whoever holds the legacy data directory, and every node afterwards only consumes it.

flowchart LR
    A["Legacy blk*.dat files"] --> B["Link blocks into one chain"]
    B --> C["Replay the chain into balances and history"]
    C --> D["Write snapshot file"]
    D --> E["Foundation key signs it"]
    E --> F["Published with the release"]
    F --> G["Node downloads and verifies it"]

Rebuilding the chain. The legacy daemon kept every block it ever saw, including blocks on branches that lost. Reading the files in order would count coins moved by blocks the network never settled on, so the converter instead links every block to its parent by hash and keeps only the path from the genesis block to the deepest block. In the run that produced the current snapshot, 145,942 of the 3,318,609 stored blocks were discarded this way. A block whose parent is not stored and that is not the genesis block is dropped together with its descendants. Block files that hold no block are refused with The block files hold no block that starts with the network magic, and block files from which no block leads back to the genesis block with No block could be linked back to the genesis block.

Replaying it. The kept chain is walked block by block to build the balance and the dated history of every address. Coins minted into the zerocoin pool are tallied separately rather than credited to an address.

The file. The result is a single file laid out so that it can be memory-mapped. Addresses are sorted by hash, so a balance lookup is a binary search, and each address’s history is one contiguous run of entries. Nothing is loaded into the heap, and a lookup does not read the whole file.

Signing. Signing hashes the file with SHA-512 and signs the digest with an ECDSA key on the P-521 curve. The signature is appended after the content, so an unsigned local rebuild and the signed release file are identical over everything they share. Anyone can therefore rebuild the snapshot from the legacy data and compare it with the published one. A node reports a snapshot as signed only if the signature verifies against a key that was ever a board key: the keys the release is built with, the four foundation keys unless --network-keys replaces them, and every key the chain of changes made active. A key that the board switched off since still vouches for the snapshot it signed, as it does for an entry of a signature log. The same key machinery signs the network-wide settings described in Grid sporks.

Distribution. hedgehog bootstrap fetch downloads bootstrap.dat.gz from the node’s own GitHub release. It first fetches a SHA-256 hash file and its detached signature, which must verify against the release key bundled in the program, and installs the snapshot only if the downloaded bytes match that hash. The file goes to a temporary name and is renamed into place only after verification, so a failed download never replaces a working snapshot. That name is snapshot-pid<process id>-<number>.part beside the snapshot. Before the next download, a partial file is removed when no running process of the id in its name can still be writing it, or when its name carries no process id. bootstrap fetch and a node that downloads as it starts can therefore write to the same directory at the same time, but only on one machine and in one container: a process cannot see one that runs elsewhere, and removes a file that such a process still writes.

A daemon checks as it starts whether it holds a snapshot it can open. A file that it refuses counts as no snapshot: one that is cut off, of another format version, signed by no key that the node knows as a board key, or ending in bytes that are no whole signature. The node then downloads the snapshot of its release in the background, and the refused file stays where it is until the download has been verified. A snapshot without a signature that opens, such as one built with bootstrap import, is kept. The board keys that judge the file include the changes that the release ships and that the spork database holds, because the daemon reads its spork database before it checks the file. A failed download, one whose hash or signature does not verify included, is tried again after 10 seconds, then after twice the previous pause up to 10 minutes, and from then on every 10 minutes. The file is judged again before every attempt, so this goes on until the node holds a snapshot it can open, such as one that bootstrap fetch installed, or the refused file once the node has learned of the board key that signed it. A file signed by a board key that the node learns of only while it runs is still replaced when a download succeeds before then. The progress, and the failure between two attempts, are reported on GET /status. The node keeps running either way, and stopping it ends the download or the pause.

Using it. The same data is available from the command line (bootstrap info, balance and history) and over HTTP, see REST: legacy chain. The balance endpoint also adds funds that the MintStorage spork has promised to an address but the chain has not minted yet, see Grid sporks. The ledger takes its legacy balances from this snapshot: its genesis commits to the snapshot’s hash and total, and a claim moves a balance to a ledger account.

Key facts

Fact	Value
Chain tip of the current snapshot	Height 3,172,666
Addresses in the snapshot	28,431, of which 3,296 still hold coins
Ledger entries	16,799,121
Total unspent	15,900,032.04557988
Snapshot size	556,361,944 bytes, roughly 314 MB as `bootstrap.dat.gz`
Amount precision	8 decimal places
Default snapshot file	`bootstrap.dat` in the per-platform user data directory
Retry of a failed download	After 10 seconds, doubling up to 10 minutes, then every 10 minutes until the node holds a snapshot it can open
Trusted signers	Every key that was ever a board key: the built-in four (`--network-keys` replaces them) and those the chain of changes made active
REST history paging	`limit` defaults to 100 and is capped at 1000

Commands

Command	Purpose
`hedgehog bootstrap import -b <dir>`	Convert the legacy `blk*.dat` files into a snapshot
`hedgehog bootstrap sign -k <key> [--force] [--spork-db <file>]`	Append a foundation signature
`hedgehog bootstrap fetch`	Download, verify and install a published snapshot
`hedgehog bootstrap info`, `balance`, `history`	Inspect a snapshot locally

An unsigned snapshot can be inspected locally and a daemon uses one with a warning, but fetch installs only a signed one. A file with a signature that does not verify is refused everywhere, and so is one that ends in bytes that are no whole signature, until bootstrap sign --force cuts those bytes off and signs the file. bootstrap sign judges its key by the longer of the chain the release ships and this machine’s record of the board keys, or by the chain of the spork database given with --spork-db <file> when that holds more of it. A key that is not one the network trusts is refused with That key is not one the network trusts, so nothing would accept the result. Without --force, a file that ends in bytes that are no whole signature is refused with <file> ends in bytes that are no whole signature, as a signing that was cut short leaves them, pass --force to cut them off and sign. A whole signature that does not verify is refused with or without --force: <file> already carries a signature that does not verify, so its contents cannot be trusted enough to sign. Each of these refusals ends the command with exit code 2. A valid signature is replaced only with --force, and without it the command prints <file> is already signed, pass --force to replace it and exits with 1.

Signing makes the new signature while the old one is still in place, so a signing that fails before then leaves the file as it was. It then cuts off what follows the content, writes the new signature block in its place and forces both to the disk. A signing stopped during that write leaves either the content without a signature, or the content and the start of the block, which --force repairs.

bootstrap import writes the snapshot to bootstrap-build-pid<process id>-<number>.tmp beside the target and replaces the target only after the new file has been read back and checked. A build that runs out of disk, is killed or fails its check therefore leaves the previous snapshot untouched, and replacing a snapshot needs room for a second copy of it. Before it writes, the build removes the partial files of earlier builds whose process no longer runs; downloads and builds each remove only partial files of their own kind. A snapshot that is replaced keeps its permissions, so a node under another account can still read it. When the target is a symbolic link to a file, the file it names is rebuilt beside where it is, and the link stays. A damaged block, like block files that hold no chain, ends the build with one line and exit code 1, as in The block at offset <offset> of <file> is damaged and cannot be read. A bootstrap command that finds a file or directory missing, or may not use it, names it and the reason, as in Cannot access <file>: no such file or directory or Cannot access <file>: permission denied. When the move into place fails, import and fetch name where the snapshot was to go, as in Cannot write <file>: <reason>.

In the source

application/src/main/java/org/unigrid/hedgehog/command/bootstrap/ holds the six bootstrap commands.
application/src/main/java/org/unigrid/hedgehog/model/bootstrap/SnapshotBuilder.java runs the conversion pipeline from block files to snapshot.
application/src/main/java/org/unigrid/hedgehog/model/bootstrap/ChainLinker.java rebuilds the active chain and drops the stale branches.
application/src/main/java/org/unigrid/hedgehog/model/bootstrap/SnapshotFormat.java defines the on-disk layout.
application/src/main/java/org/unigrid/hedgehog/model/bootstrap/SnapshotSignature.java signs and verifies a snapshot.
application/src/main/java/org/unigrid/hedgehog/model/bootstrap/SnapshotDownload.java is the verified download and install.
application/src/main/java/org/unigrid/hedgehog/model/bootstrap/SnapshotInstaller.java decides as the daemon starts whether it holds a snapshot it can open, and downloads the snapshot of the release until it has one.
application/src/main/java/org/unigrid/hedgehog/model/bootstrap/PartialFiles.java names the partial files of downloads and builds, and removes those left behind.
application/src/main/java/org/unigrid/hedgehog/server/rest/BootstrapResource.java is the HTTP interface.