Ledger in the daemon

Part of Ledger implementation. The ledger module knows nothing of options, files, sporks or threads; it asks for them through interfaces. This page covers the classes of the application module that supply them: the options and their checks, the genesis and key files, the snapshot, the authority over the sporks, and LedgerService, which puts it all together when the daemon runs with --ledger. How an operator uses them is on Running a ledger node and Ledger sporks.

flowchart TD
    Daemon -->|before the container| Setup[LedgerSetup]
    Service[LedgerService] --> Genesis[GenesisFile]
    Service --> Check[SnapshotCheck]
    Service --> Runtime[LedgerRuntime]
    Service --> Nodes[LedgerNodes]
    Service --> Authority[SporkLedgerAuthority]
    Service --> Driver[LedgerDriver]
    Service --> Gateway[LedgerGateway]
    Nodes --> Snapshot[LedgerSnapshot]
    Authority --> Sporks[SporkDatabase]
    Authority --> Lag[SporkLag]
    Authority --> Entry[MintEntry]
    Gateway --> Driver

Off by default

--ledger is a boolean option of daemon, off by default. With it off nothing in the daemon changes: the node offers no ledger protocol, installs no ledger pipeline and starts no ledger threads, and LedgerService does nothing. Network.hasLedger() follows the option, and the protocol list follows that, see Ledger networking.

Before the container starts

Daemon.run() calls LedgerSetup.validate() before the container starts, so a bad setup stops the daemon with a readable message and exit code 2. It does nothing unless --ledger is given, and then:

1. loads the key file of --ledger-key, if given;
2. parses --ledger-trusted as <height>:<128 hex characters>, with a height of at least 1;
3. loads the genesis, from --ledger-genesis or from the bundled resource, and refuses to start without one: --ledger needs a genesis: give --ledger-genesis <file> (no genesis is bundled with this release);
4. warns, without stopping, when the key is not one of the genesis validators, with The key <public key> is not a validator of this genesis, this node signs once the validators vote it in.

The genesis file

GenesisFile reads the genesis as one JSON object with exactly the fields time, roundMillis, validators, snapshotHash and snapshotTotal, see Running a ledger node. It is read from the file given with --ledger-genesis, or else from the resource /ledger-genesis.json when the build holds one. No build holds one yet.

The read is strict, so that a typo cannot pass:

• the file must be a regular file, read through links, and at most 1 MiB, whatever size it claims;
• a field that is missing, repeated or unknown is refused, and so is text after the object;
• time is 0 or more, roundMillis 60000 or more, snapshotTotal 0 or more, all whole numbers;
• validators holds 1 to 1000 keys of 64 hex characters each, each an Ed25519 public key, with no key twice; their order is the proposing order;
• snapshotHash is 128 hex characters.

A message names the file and the field and never copies the content, since an operator can point the option at the wrong file, such as a key file. GenesisFile.write writes a genesis back in the same form, and chainIdHex gives the chain id that the service logs at start.

The key file

LedgerKeyFile holds the seed of a validator: one line of 64 hexadecimal characters.

• create refuses an existing path, writes a temporary file that is created owner-only rather than restricted afterwards, and publishes it with a hard link, which fails if the path is taken, so a key is never replaced. Only a file system without hard links gets a plain move.
• load refuses a file that a group or others can read, with the advice to run chmod 600, wherever POSIX permissions exist, and reads at most 128 bytes.
• A seed is never printed or logged. publicKeyHex gives the public key.

OwnerOnlyFile holds the permission check and the owner-only creation, and the gridnode key uses it too. hedgehog util ledger-key-generate <file> calls create and prints the public key.

The service

LedgerService is an @Eager @ApplicationScoped bean. It is built on the thread of the container, which only reads the genesis and does not wait for anything. The daemon is whole without the ledger: whatever stops the ledger from starting is logged with the name of the file, as The ledger is not running, the node goes on without it: <reason>, and the node runs on as it was.

Starting.

1. Load the genesis and log The ledger is of the chain <chain id>.
2. Start the two threads of the ledger, see The threads.
3. Every 5 seconds, look for the bootstrap snapshot, which may still be downloading, and log once a minute that the ledger waits for it.
4. Once it is there, check it against the genesis with SnapshotCheck, on the timer thread. A snapshot that is not the one of the genesis stops the ledger.
5. Under a lock, so that a stop either comes before it and nothing starts, or after it and closes it: load the seed, build the snapshot adapter, the authority, the policy of the peer limiter and the node factory, build the driver, attach it to the gateway and start it.
6. Every 5 seconds, look whether the driver has halted. The driver logs The ledger driver halted: <reason> as it halts; the service closes it and builds it again in the same process over the same files after 1 minute, then after 2 and after 4, logged The ledger halted, it starts again in <n> ms: <reason>.

A ledger that halts after 3 restarts within 6 hours stays halted, and no further restart follows: The ledger halted after 3 starts again, it stays halted and serves the blocks it has, the node goes on: <reason>.

The replay of the block log runs on the events thread of the driver, not on the thread of the container. A task of a timer that throws would be lost without a word, so every task is wrapped and its failure is turned into the refusal above.

sequenceDiagram
    participant W as Container thread
    participant S as LedgerService
    participant T as ledger-timers
    participant B as BootstrapSnapshot
    participant E as ledger-events
    participant D as LedgerDriver
    W->>S: start()
    S->>S: load the genesis, log the chain id
    S->>T: poll now
    S-->>W: back at once
    loop every 5 seconds until it is there
        T->>B: getReader()
    end
    T->>T: SnapshotCheck: hash and total
    T->>D: build, attach to the gateway, start
    D->>E: build the observer: open and replay the log
    E->>E: sync, then promote to a signing node
    loop every 5 seconds
        T->>D: halted?
    end

Reading. read(function) applies a function to the newest ledger the driver has built, on the thread of the caller, and is empty before there is one. It is for reads only. height(), supply(), balanceOf(address) and tipHash() are reads of this kind. They still answer after the driver halted or the service stopped. submit(transaction) is the only way in: it hands a transaction to the driver, which also gossips it, and answers with the ledger’s reason for a refusal, or The ledger is not running. The answer is a future. A submit that is made while the driver closes is answered The ledger is not running and not left waiting. status() gives a Report:

Field	Meaning
running	A driver is started, the service has not stopped and the driver has not halted
halted, haltReason	Whether the driver halted, and why
signing	Running, the newest node was built to sign, and the key is one of the validators of the newest ledger
height, tipHash	Of the newest ledger, 0 and empty before there is one
droppedInbound	What the driver’s queue has dropped
unsetSporks, laggingSporks	While running, the ledger sporks that are not set, and those of which a newer version is known and not here yet. Nothing that depends on them is minted, claimed or paid for

LedgerResource reads all of this: GET /ledger is the report together with the chain, the supply, the round and when it ends, the validators that sign blocks, the set that the round’s votes are judged by, and the minimum fee of the fee spork, see REST interface. A halted ledger shows running false, halted true with its reason, and the height and tip hash of the newest ledger; GET /ledger shows the same. A node signs only while its key is among the validators: it starts when they vote it in, and a key that is voted out stops signing.

Stopping. On shutdown, in this order: the gateway lets go of the driver, so that nothing new reaches a driver that is closing; the driver closes its node and is waited for, up to 10 seconds; the threads end, also waited for up to 10 seconds; and last the block log is closed.

The threads

LedgerRuntime is the Serial and the Scheduler of the driver: one daemon thread ledger-events that takes the driver’s events one at a time, and one daemon thread ledger-timers that waits for its timers. Both are started by the thread that builds the runtime, not by whichever thread first gives them work. On close the waiting timers are dropped, and a task that is running on the events thread is finished, not interrupted.

The nodes

LedgerNodes is the NodeFactory of the driver, over the files blocks.log and sign.state in the folder of the chain below ledger in the data directory, named by the first 16 characters of the chain id (LedgerService.chainDirectoryOf), see Ledger storage. Each create closes the block log of the node before, opens it again, replays it into a fresh LedgerState built from the genesis, the authority and the snapshot, and opens the sign state only for a signing node. The replay is the cost of a node, and a failure to open or replay names the file. The newest ledger is kept for the readers of the service, and stays readable after its node is gone.

The snapshot

LedgerSnapshot is the ledger’s Snapshot over the SnapshotReader of the bootstrap snapshot. It holds the one reader it is given for its whole life: the ledger replays its block log through the snapshot and must judge every claim against the same balances, so a snapshot file that changes under a running ledger is not followed. It also gives the MintSchedule, from the tip height and tip time of the same reader, so a mint is never due against another snapshot. The reader answers from a read-only memory map and is safe from any thread.

SnapshotCheck.verify decides once, at start, whether the file is the one the genesis commits to: its content hash, the SHA-512 of the file without its signature, and the sum of its balances must both be the genesis’s. The signature of the snapshot is not asked for, since the hash in the genesis is the commitment. A mismatch is reported with both values, such as The snapshot <file> adds up to <n> units, the genesis expects <m>.

SnapshotReader answers tipHeight(), tipTimeMillis(), which reads the block time of the tip as unsigned seconds, and balanceUnits(hash), which is 0 for an address that is not in the table.

The authority

SporkLedgerAuthority is the node’s LedgerAuthority, read from the sporks. It is policy and may differ between nodes: the ledger asks it only of what is not certified, and a block that most validators signed stands whatever it says.

Question	Answered yes when
due(time)	Lists every entry of the mint storage spork that is due by time, can be minted, and fits alone under the maximum supply on top of the committed supply and under what its address is granted on top of what it has had. Sorted by reference
authorizes(mint, time, supply, minted)	The mint is exactly one of the entries due by time for its recipient, its amount fits under the maximum supply on top of supply, and under what the spork grants the recipient on top of minted, the mints made to it so far
authorizes(claim)	The blacklist spork is set and does not list the claimed address
authorizes(transfer)	The fee spork is set and the transfer pays at least its minimum

It also answers leastNewAccountAmount(): the least amount that a transfer to an address the ledger has no account for must carry. That is the fee spork’s amount to a new account, or 0 when no fee spork is set. The state asks it only after authorizes(transfer) answered yes.

How authorizes(mint, time, supply, minted) decides:

flowchart TD
    Ask[a mint to judge] --> Db{spork database there?}
    Db -->|no| No[no]
    Db -->|yes| Supply{mint supply spork set<br/>and not behind?}
    Supply -->|no| No
    Supply -->|yes| Room{amount within<br/>maximum minus supply?}
    Room -->|no| No
    Room -->|yes| Storage{mint storage spork set<br/>and not behind?}
    Storage -->|no| No
    Storage -->|yes| Entry{an entry for this recipient<br/>that is due by the time<br/>and gives exactly this mint?}
    Entry -->|no| No
    Entry -->|yes| Grant{amount within what the<br/>spork grants the address<br/>minus what it has had?}
    Grant -->|no| No
    Grant -->|yes| Yes[yes]

A node that does not know answers no, which only delays a block:

• a spork that is not set: no mints without a mint storage and a mint supply spork, no claims without a blacklist spork, even an empty one, and no transfers without a fee spork;
• a spork that a newer one is known to replace, see Spork lag: the lag is asked first, so a spork stored in between is read in its newer form;
• any trouble while answering. No exception leaves: a failure is logged once at debug level and answered no.

A spork can change at any time, so each call reads the database once. One thing is kept between calls: the entries of the mint storage spork, decoded and found by reference, together with their due times and the grant of each address. The ledger asks about every due mint at every tick and for every block it checks, and decoding thousands of entries each time would take seconds. The slot is kept for the spork object it was made from and found again only by that same object. That is safe because a spork is replaced whole when a newer one is accepted and never changed in place. A cause that is logged once a second would fill the log, so each is told once, up to 256 of them.

What an address is granted is the sum of the amounts of all the entries of the mint storage spork that can mint for it: every entry of an account, and for a legacy address those above the snapshot tip. A mint is refused when the mints made to the address so far plus this one would pass that sum. An entry already mints once, by its reference, but a later spork can replace an entry by another for the same address, with a new number and so a new reference, and the cap keeps that from paying the address twice for what the spork grants it once. The total comes from Ledger.mintedTo, which the service passes in as a supplier for the listing, and the ledger passes in for a mint it checks. Each spork object is summed once, each reference once, and a sum that a long cannot hold is cut at the largest. So a spork that moves an entry of an address that was paid, to another height or time, mints nothing more. Only an entry that fits whole in what the spork grants the address beyond what it has had is minted.

The maximum supply of the mint supply spork is cut at eight decimals, so it is never above what the spork says, and a maximum that a long cannot count is more than there can be. The room is max(0, maximum - supply).

Mint entries

MintEntry reads an entry of the mint storage spork as a mint of the ledger. The spork keeps its shape, an address string and a number of 32 bits that the model holds as an int and the wire as a u32. The number is read unsigned, and what it says depends on the address:

Address string	Kind	The number is	Due at
A legacy base58 address	OLD	A height of the legacy chain	MintSchedule.dueTime(height). An entry at or below the snapshot tip never mints
An Ed25519 account, 64 lower-case hex characters	NEW	Unix time in whole seconds, to 2106-02-07	That time. The snapshot tip plays no part, since no snapshot balance exists for an account
Anything else			Never: the entry is skipped and logged once

Upper-case hex is not an account, which leaves one spelling per account and so one reference per grant.

• units() gives the amount in units of 1e-8, cut at eight decimals and never rounded up. An amount that is zero, negative, below one unit or beyond a long is no mint, and the entry is skipped.
• reference() is Reference.ofSpork(recipient, number), see Ledger encoding.
• heightOf(address, number) is the one rule for the command line and the REST interface: a number from 0 to 4,294,967,295 for an account, and up to 2,147,483,647 for a legacy address.

The ledger sporks

The ledger has two governed sporks, each with a data class, a wire encoder and decoder, a field in SporkDatabase, the periodic publish, a REST resource and a command:

Spork	Id	Data	Wire after the spork header
LedgerBlacklist	1040	legacyAddresses: a set of hash160s as 40 lower-case hex characters, at most 4096	A count of 2 bytes, then the 20-byte hashes in ascending order
LedgerFee	1050	minimumTransferFee: coins, 0 to 1, at most eight decimals; minimumNewAccountAmount: coins, 0 to 100, at most eight decimals	The fee, then the amount to a new account, each in units of 1e-8 as an unsigned 8-byte number

The encoders validate the data first, since the signatures cover these bytes and what cannot be decoded again must not be signed. The decoders refuse a blacklist of more than 4096 entries, a fee above one coin and an amount to a new account above a hundred coins, read unsigned, so a value with the high bit set is not taken for a small amount. A fee spork carries both numbers, and the decoder reads both whatever the frame holds: one that stops after the fee is a corrupt frame, not one with an amount to a new account of 0. The lower-case hex gives one spelling per address, so equal lists sign the same bytes.

Amounts are bounded. The amounts of the mint storage, mint supply and vesting storage sporks are text on the wire, and anyone can send a frame. A number such as 1E+999999999 is a few bytes, but writes out as a billion digits when it is signed, which takes gigabytes. SporkAmount therefore takes an amount of at most 8 decimals, not counting trailing zeros, and 12 digits before the point, in at most 48 characters, and refuses a scale that would write out longer than that. A frame that holds more is corrupt. Real amounts fit with room to spare, since the ledger counts in units of 1e-8 in a long. The REST resource of the mint storage refuses such an amount with the reason before it signs anything.

GridSpork.Type carries a factory for each type, and SporkDatabase reads and writes each type through a table of slots instead of a switch. Its fields are volatile, since network threads store sporks while the ledger’s threads read them. An older spork.db still loads.

Only to ledger peers. A node of release 0.0.8 reads the two bytes of a spork’s type, and for a type it does not know it logs an error and reads the rest of the frame on as more types: hundreds of error lines for one spork, a chance of taking garbage for a spork, and with an odd frame length the next frame read out of step. SporkRecipients therefore sends the blacklist and the fee only over a stream whose connection settled on the ledger protocol, which a node of an earlier release never does. Everything that writes a spork to a stream goes through it: the periodic publish and new streams, the relay of a spork that came in, and the REST resources. A node without --ledger neither sends nor relays the two, but may still store one that a peer sent it. A spork that is not set is not sent at all.

Spork lag

SporkLag remembers, per spork type, the newest time stamp the network keys have been seen to vouch for, whether this node accepted that spork or not. PublishSporkChannelHandler records every spork it receives before it decides on it, so a spork that this node turns down still tells it that the network has moved on. isBehind(type, database) is true when that time stamp is newer than the stored spork’s, or when the type is not stored at all.

A spork sent to this node in wire form is recorded too, once its signature checks out, with the source local: /gridspork/wire records one before it decides whether it stores it or holds it as a proposal. A proposal taken in this way waits for the co-signature like every other, and the node does not judge by the old spork meanwhile. That matters most for the blacklist, where a claim taken on the old list cannot be taken back; the price is a temporary lockout of what the spork governs.

• A spork counts only when the signature check that acceptance relies on passes: signed by two different current network keys, or, as a proposal, signed once by a current network key while a proposal of that age would still be held. A peer without a network key cannot make a node wait.
• The checks that compare a spork with the stored one, a log that extends the stored log and a newer time, are left out on purpose: a validly signed spork that fails them is a rival version of the history, and the node waits for it to settle.
• A check costs a signature verification and anyone can send a frame, so it is made only for a spork newer than the stored one, and only when VerifySlots gives a turn: one check a second for each spork type and sending peer, and at most 32 a second for a type whoever sends. A flood of forged sporks from one peer uses up only that peer’s turns. A spork that is turned away is not recorded, and since every peer sends the sporks again and again, a genuine one gets its turn. A spork recorded as local, one a board member signed and sent in wire form, takes no turn: the holder of the REST token submits it once, and a proposal that is turned away would not be sent again by anyone.
• A time stamp more than 10 minutes ahead of the node’s clock is recorded as 10 minutes ahead, and a proposal counts only for the 60 minutes a node holds it, so a replayed old proposal cannot hold a node back.
• The memory is not stored. A restarted node learns it again from the sporks its peers send on every new stream and every three minutes.

Tests

LedgerFleet runs real daemons in one process: each node is a container of its own with a QUIC server, its streams to the others, and a LedgerService that the container starts as the daemon does. They share one genesis and one bootstrap snapshot. The files of a node are on a virtual file system and survive a stop, so a test can restart a node over its files. What is faked is the surroundings of the service, through LedgerService.setup(): the directory, the key file, the genesis, the clock and the speed. A fleet is at most the 20 servers of BaseServerTest. A virtual file system gives every lock to whoever asks, so the lock of a sign state is not tested there.

LedgerFleetTest starts every node on the ledger protocol, keeps each node’s files apart, produces a block on every node after a transaction, delivers sporks through the receive path and refuses one that two network keys have not signed, catches up a stopped node and a whole network restarted over its files, syncs an observer that joins late, and makes a mint due by the clock. LegacyWallet signs claims as the legacy wallet does.

Further tests cover each class above: LedgerServiceTest and its CDI tests, LedgerNodesTest, LedgerRuntimeTest, GenesisFileTest, LedgerKeyFileTest, LedgerSetupTest, LedgerSnapshotTest, SnapshotCheckTest, SporkLedgerAuthorityTest, MintEntryTest, SporkLagTest, the spork, integrity, resource and command tests of the two new sporks, and DaemonTest.

In the source

• application/src/main/java/org/unigrid/hedgehog/service/ledger/LedgerService.java: the ledger as a part of the daemon
• application/src/main/java/org/unigrid/hedgehog/server/rest/LedgerResource.java: the REST side of reading the ledger and submitting to it
• application/src/main/java/org/unigrid/hedgehog/command/cli/ledger/: the commands of hedgehog cli ledger
• application/src/main/java/org/unigrid/hedgehog/model/bootstrap/LedgerText.java: addresses and coin amounts as people write them
• application/src/main/java/org/unigrid/hedgehog/service/ledger/LedgerNodes.java: the node factory over the files
• application/src/main/java/org/unigrid/hedgehog/service/ledger/LedgerRuntime.java: the two threads
• application/src/main/java/org/unigrid/hedgehog/service/ledger/LedgerSetup.java: the checks before the container starts
• application/src/main/java/org/unigrid/hedgehog/service/ledger/GenesisFile.java: the genesis as JSON
• application/src/main/java/org/unigrid/hedgehog/service/ledger/LedgerKeyFile.java: the validator’s seed, with model/util/OwnerOnlyFile.java
• application/src/main/java/org/unigrid/hedgehog/service/ledger/LedgerSnapshot.java and SnapshotCheck.java: the snapshot behind the ledger
• application/src/main/java/org/unigrid/hedgehog/service/ledger/SporkLedgerAuthority.java: the policy from the sporks
• application/src/main/java/org/unigrid/hedgehog/model/bootstrap/MintEntry.java: an entry of the mint spork as a mint
• application/src/main/java/org/unigrid/hedgehog/model/spork/LedgerBlacklist.java and LedgerFee.java: the two sporks, with their encoders and decoders beside them
• application/src/main/java/org/unigrid/hedgehog/model/spork/SporkLag.java: which sporks are known to be out of date
• application/src/main/java/org/unigrid/hedgehog/model/spork/SporkAmount.java and VerifySlots.java: the bounds on an amount in a spork, and the turns for checking signatures
• application/src/main/java/org/unigrid/hedgehog/model/network/handler/SporkRecipients.java: which streams may receive a spork
• application/src/main/java/org/unigrid/hedgehog/server/rest/LedgerBlacklistResource.java, LedgerFeeResource.java and MintStorageResource.java: the REST side of the sporks
• application/src/main/java/org/unigrid/hedgehog/command/cli/spork/LedgerBlacklist.java, LedgerFee.java and MintStorage.java: the commands
• application/src/main/java/org/unigrid/hedgehog/command/option/LedgerOptions.java: the command-line options
• application/src/main/java/org/unigrid/hedgehog/command/util/LedgerKeyGenerate.java: util ledger-key-generate
• application/src/test/java/org/unigrid/hedgehog/service/ledger/LedgerFleet.java: real daemons in one process