Peer-to-peer network protocol

After 0.0.8. The GRIDNODE packet, gridnode announcements, the storage, ledger and chain packets, the chain stream and the identifiers hedgehog-proxy/0.0.2, hedgehog/0.0.9 and hedgehog-ledger/0.0.2 are on the master branch and not part of a release yet. Release 0.0.8 offers hedgehog/0.0.4 and gridspork/0.0.4. Release 0.0.9 adds hedgehog/0.0.9, and carries the rest only with the hidden --preview switch, see Stages a node carries.

Every Hedgehog node is both a client and a server: it keeps encrypted connections open to other nodes and uses them to find more peers, agree on network-wide settings, advertise gridnodes and move storage requests around. There is no central server; a new node needs only the six seed addresses to join.

The idea in short

• Transport. Nodes talk QUIC, an encrypted protocol that runs over UDP. The dialing node opens one bidirectional stream, the main stream, and everything between two nodes travels on it in both directions. When the connection settled on the proxy or the ledger protocol, which only nodes run with --preview offer, it also opens a second, stream 4, for the chain traffic of the SOCKS proxy. A connection allows at most 16 streams and 16 MiB in flight on the connection and on each stream; it is closed after 15 minutes without traffic, and a dial that takes more than 2 seconds fails. On a connection that a peer opened, the node serves the first main stream for as long as the connection lives. Any further main stream the peer opens, also after the first one closed, is closed before anything is set up for it, and the third such stream closes the connection; the chain stream does not count. Such a connection is also closed once it has held no open stream for 20 seconds, counted from when it was made and from the close of its last stream. A node closes a connection that it dialed as soon as the peer ends the main stream, after it has handled what arrived before the end, and logs Closed the connection to <peer>, which ended the stream at info level; the peer stays known and the next round dials it again. A node that stops closes every connection, those its peers dialed included.
• Messages. Each message is a small binary frame with an eight-byte header: a magic number, a packet type and the payload length. The receiving side reads the type and hands the payload to the matching decoder and handler.
• Joining. A node dials the seeds (seed1.unigrid.org to seed6.unigrid.org), learns more peers from them and keeps dialing the peers it knows. A node that cannot be reached is dropped from the list, but one an operator configured stays configured and is dialed again; a dropped node returns if a seed or another peer announces it again.
• Compatibility. Both sides offer the protocol identifiers hedgehog/0.0.9, hedgehog/0.0.4 and gridspork/0.0.4 during the handshake, in that order; a node run with --preview offers hedgehog-proxy/0.0.2 ahead of them, and one that also runs a ledger hedgehog-ledger/0.0.2 ahead of that. A node that shares none of them cannot connect, and the other side logs that it is running an incompatible release. What the proxy and ledger identifiers tell a peer is on SOCKS proxy.

What nodes say to each other

Twelve packet types are live on the main stream of a node run with --preview, six more on a connection that settled on the ledger protocol, and eight on the chain stream. A node run without it takes four: HELLO, PING, PUBLISH_PEERS and PUBLISH_SPORK.

Group	Packets	Purpose
Presence	HELLO, PING	The dialing node announces the port it listens on; both sides measure latency
Topology	PUBLISH_PEERS, GRIDNODE	Share the list of known nodes and the signed list of active gridnodes
Settings	PUBLISH_SPORK	Spread grid sporks, the signed network-wide settings (see Grid sporks)
Storage	Seven packets, from STORE_FRAGMENT to STORAGE_ACK	Store, fetch, count and delete data fragments (see Network storage)
Ledger	Six packets, from LEDGER_TRANSACTION to LEDGER_BLOCK_BATCH	Transactions, consensus, status and block sync of the ledger
Chains	Eight packets, from JOIN_CHAIN to CHAIN_STEM, on the chain stream	The entries, heads and stem of the chains of the SOCKS proxy (see SOCKS proxy chains)

The storage packets can travel in either direction, so whichever node dialed, either side can ask and either side can answer. Each request carries an id, and the reply is matched to it by id, stream and type.

sequenceDiagram
    participant A as Dialing node
    participant B as Remote node

    A->>B: QUIC handshake (offers hedgehog/0.0.9, hedgehog/0.0.4, gridspork/0.0.4)
    A->>B: Open one bidirectional stream
    A->>B: HELLO with the port A listens on
    Note over B: B records A as a known node, for the first HELLO of the connection
    A->>B: Board keys, other stored sporks, held proposals
    B->>A: Board keys, other stored sporks, held proposals
    loop while connected
        A-->>B: PING, PUBLISH_PEERS, PUBLISH_SPORK, and GRIDNODE with preview
        B-->>A: PING, PUBLISH_PEERS, PUBLISH_SPORK, and GRIDNODE with preview
    end

Stages a node carries

What a node takes on the main stream follows from the protocols it offers. Every node carries the presence, topology and settings stages, except for the GRIDNODE packet. Only a node run with --preview offers hedgehog-proxy/0.0.2, and only such a node carries the gridnode and storage stages and announces gridnodes; one that also runs --ledger offers hedgehog-ledger/0.0.2 and carries the ledger stages. A node run without --preview never settles on a connection that takes chain traffic, so it opens no chain stream, and a second stream that a peer opens is a further main stream, which is closed (see Transport).

The frame decoder of the main stream knows which stages the node runs. On a node run without --preview it passes over the body of a GRIDNODE frame and of the seven storage frames as the body arrives, and on a node that offers no ledger protocol it does the same for the six ledger frames. It keeps only a count of the bytes still to come, and the stream stays open. The header of such a frame is still held to the size bounds of its type, so a frame outside them closes the stream. Both ends close the main pipeline with UnclaimedFrameHandler. A decoder hands a frame it does not own on as it is, so any other frame that no stage takes, such as ASK_PEERS or ASK_NODE_DETAILS, is dropped there rather than run into the frame behind it. In the size table of the main stream a chain packet holds nothing, so a chain packet with a body on any stream but the chain stream closes that stream. Only a connection that settled on the proxy or the ledger protocol has a chain stream.

How the network stays in sync

• Peers. Every three minutes each node sends its whole list of known nodes to every connection. Receivers add the ones they have not seen, so knowledge of the network spreads without any directory. A peer list frame carries IP literals only, so a host name is dropped without a DNS lookup. An entry whose address carries a named IPv6 scope, a % followed by anything but a number such as fe80::1%eth0, is skipped without asking the operating system for the interfaces of the host, and so is an entry that holds any character other than hex digits, . and : before the scope, an address in brackets among them; a numeric scope is kept. A node reads the first 1,024 entries of a frame, skipped ones included, each address at most 64 bytes, and ignores the rest. A peer with a public address cannot add loopback, private, link-local, unspecified, multicast or unique-local addresses, so a private network on its own LAN or on loopback still works. Only the first HELLO of a connection, with a port above 0, names the peer’s node and adds it to the topology; a later HELLO on any stream of that connection is ignored. A node known only by a stream it opened to this node, one that is not configured, is no seed and has no connection that this node dialed, leaves the topology as soon as that stream closes. The topology holds at most 1,000 known nodes and always keeps the seeds. Only the connections a node dialed count toward its 32 outbound connections and set the network groups; a node known only through a connection a peer opened is still dialed. It dials the nodes an operator configured, the seeds and those added with node-add, outside that limit, before the others, which it takes in random order, at most one in each network group, an IPv4 /16 or an IPv6 /32. A configured node stays configured when a dial fails. node-remove, the REST DELETE, clears a node added with node-add; a seed that it removes is a seed again once a round resolves a seed name to it.

  The node dials in rounds. In every round it looks up the seed names, and the addresses they resolve to are the seeds of that round, dialed like configured nodes and let in past the 1,000-node cap. A name whose lookup fails keeps the address it last resolved to. A seed address that no seed name resolves to any more leaves the topology unless it was added with node-add; it also stays while a connection or an open stream still ties it to the node, and then as a node a peer told of. A dial that fails ends the dial of that node only, and the round goes on with the next node. After a round the node waits 33 seconds plus 3 seconds for each known node up to 32, which is 33 to 129 seconds. A round that fails with an exception is logged at warning level as A round of the topology thread failed with <class of the failure>, with its trace at debug level, and the next round follows after the wait of the last round that went through.

• Sporks. A node sends the spork of the board keys first, then its other stored sporks, on a new stream and every three minutes, and passes on each spork it accepts and that counts. An ordinary spork is accepted only when two different keys active now signed its head, it follows on the stored one under the rules of Grid sporks, and every entry of its history was signed by a key that was ever active; the board keys spork is judged by its chain instead, see Board keys. A co-signed spork stamped more than two minutes ahead of the clock is dropped, as is one whose head does not take the place of the stored spork. A proposal with one signature is held only when it was made over the stored spork, or over nothing when the node stores none, and is forwarded in the same way; a node that lags takes it after it has caught up to the version it was made from. The board keys, vesting storage and storage sporks go only to a peer whose connection settled on hedgehog/0.0.9, the proxy protocol or the ledger protocol, see Board keys on the wire. The communication spork goes only over the proxy or the ledger protocol, and the ledger sporks only over the ledger protocol. A node run without --preview takes and sends no storage, ledger or communication spork, see Grid sporks.
• Gridnodes. Each gridnode signs an announcement of its address, its status and its sealing key with its own key, see Gridnode announcements on the wire. A receiver accepts an entry only if the signature verifies, the timestamp is recent, and the entry is newer than the one it already has. It then forwards the entry to every peer except the sender. Entries that are not refreshed for 30 minutes expire.
• Latency. Both sides ping every 20 seconds: a NAT, Docker’s among them, may forget a UDP flow that has been quiet for 30 seconds. The node that dialed records the round trip, and the REST interface reports it with each node (see REST interface). A ping answer the node never asked for, or a second answer, is ignored. A peer that sends nothing at all while three pings in a row go unanswered, about a minute, has crashed or hangs: the node closes its connection, as a peer that crashed sends no close. Any data from the peer counts as an answer, as one can wait behind a large frame on the stream. A ping is answered only while the stream takes more data; otherwise the answer is dropped. The ping, peer list and spork schedules skip a stream that takes no more, so no ping is sent on it. While such a stream stays full, each 20-second ping period in which less than 16 KiB left it adds the shortfall to what its peer owes; a period with more pays the debt off, never below zero, and the surplus is not kept. Once 48 KiB are owed, the node closes the connection and logs at info level Closed the connection to <peer>, which fell 49152 bytes behind the 16384 bytes that it is to read between pings while its stream took no more. The debt is cleared as soon as the stream takes more again. The same rule holds for a frame that has not arrived whole: from the read that brought its header, what arrived of the frame is looked at every 20 seconds, and the stream is closed once the frame owes 48 KiB, so three periods with nothing close it after about a minute. A peer that moves at least 16 KiB in every period is kept however long the transfer takes.

Board keys on the wire

The spork of the board keys carries the chain that every node and command line verifies. Its data begins with a count of changes in one byte, then each change holds its sequence (4 bytes), the time it was made (8 bytes, Unix milliseconds), the digest it follows on (64 bytes), the keys it makes active (a count in one byte, then each key behind a 2-byte length in lower-case hex, at most 262 characters) and its approvals (a count in one byte, then each key behind a 2-byte length, followed by its signature behind a 2-byte length, at most 255 bytes). A chain holds at most 64 changes, each 2 to 16 keys, with at most 32 approvals, and the decoder refuses a count above its cap before it reads what the count covers.

A node of release 0.0.8 reads the two bytes of the type of a spork, and for a type it does not know it logs an error and leaves the rest of the frame unread, which it then reads on as more types. The board keys spork therefore goes only to a peer whose connection settled on hedgehog/0.0.9, hedgehog-proxy/0.0.2 or the ledger protocol, none of which release 0.0.8 offers. So do the storage spork, whose type that release does not know either, and the vesting spork, whose schedules it reads without the three fields added since. The other way round, a node of release 0.0.8 sends its vesting spork (type 1020) in the older layout. On a connection that did not settle on hedgehog/0.0.9, hedgehog-proxy/0.0.2 or the ledger protocol, a node skips such a frame whole, as it skips a spork of a type it does not know, and keeps the stream open; over a connection that did, a vesting spork is read as before. The board keys spork is published before any other spork, as every other spork is judged by the keys it makes active. A node of 0.0.8 refuses every spork that a key it does not know signed, so it must be upgraded before the board keys change. It cannot read a spork.db that holds a board keys spork either, and moves the file aside as spork.db.unreadable-<epoch seconds>. The changes themselves are described in Changing the board keys.

Gridnode announcements on the wire

A GRIDNODE packet (2030) holds one announcement, big-endian:

Field	Bytes
Status	1: ACTIVE 1, INACTIVE 2
Timestamp	8, Unix milliseconds
Gridnode id	A u16 length, then 262: the gridnode’s P-521 public key in hex
Host	A u16 length, then at most 255 of UTF-8: the announced host:port
Sealing key	32: the X25519 public key a client seals a SOCKS proxy session to
Signature	A u16 length, then at most 150

The signature is ECDSA with SHA-512 under the gridnode key, over hh-gridnode-v2 in ASCII, the id behind a u16 length, the status (u8), the host behind a u16 length, the sealing key and the timestamp (u64). The sealing key is signed with the rest, so it is bound to the gridnode, and an announcement without one does not verify.

Security

Connections are encrypted and integrity-protected by QUIC, but nodes do not authenticate each other at the transport level: each server creates a fresh self-signed certificate at start-up and clients accept any certificate. Trust therefore sits in the data rather than in the connection. Sporks and gridnode announcements are verified against signatures, and the network keys that govern sporks are described in Grid sporks. The server also uses QUIC address-validation tokens, which are tied to the peer address and become invalid when the node restarts. One remote address may hold at most eight connections at a time; loopback is exempt, so a forwarder or several nodes on one host still work. A node also admits at most 2,000 inbound connections in all, set with --netmaxinbound (8 or more), which a seed, as every node dials it, may need to raise. Only connections that peers open count, not those the node dials, and loopback is exempt again; an address that holds its eight is refused for that alone. While the ceiling makes it refuse new peers, the node logs a warning at most once a minute: The ceiling of <ceiling> inbound connections is reached, new peers are refused; a seed may need more: option --netmaxinbound.

For what peers send, checking is bounded: a node makes at most 256 signature checks a second, 128 of one type, with three quarters of each for the connections it dialed and one quarter for those peers opened. A network group, or a configured peer on a dialed connection, has a share of its own: on a dialed connection 12 checks a second of a type and 24 over all types, on one a peer opened 8 of a type and 8 over all types; a pool’s node ceiling takes at least 8 sources to spend. A signature that verified is not checked again while the node remembers it. What arrives over REST is not counted against that budget.

Key facts

Item	Value
Default port	52883 (--netport or -p)
Default bind address	0.0.0.0 (--nethost or -H)
Seed nodes	seed1.unigrid.org to seed6.unigrid.org, looked up every round, a failed lookup keeps the last address; turn off with --no-seeds
Protocol identifiers	hedgehog/0.0.9, hedgehog/0.0.4 and gridspork/0.0.4; with --preview hedgehog-proxy/0.0.2 ahead of them, and hedgehog-ledger/0.0.2 first with a ledger
Streams	The main stream, one per connection; on a connection a peer opened, a further main stream is closed and the third further one closes the connection; stream 4 for chain traffic over the proxy or the ledger protocol, so only between nodes run with --preview
Unclaimed frames	Bodies of frames of stages the node does without are passed over as they arrive; other frames that no stage takes are dropped at the end of the main pipeline
Frame	8-byte header, magic 0xBABE; the size cap depends on the packet type
Spork frame	At most 16 MiB
Board keys spork	Published first, only to a peer that agreed on hedgehog/0.0.9, the proxy or the ledger protocol, as are the vesting storage and storage sporks
Board keys chain	At most 64 changes, each 2 to 16 keys, with at most 32 approvals
Storage fragment frame	At most 256 MiB
Ledger transaction	At most 163 bytes, behind a 4-byte length
Ledger consensus message	At most 528,441 bytes, behind a 4-byte length
Ledger block batch	At most 8 MiB plus 64 KiB, behind a request id and a 4-byte length
QUIC limits	16 streams, 16 MiB in flight, 15 minutes idle timeout, 2 second connect timeout
Connections per address	At most 8; loopback is exempt
Inbound connections in all	At most 2,000 (--netmaxinbound, 8 or more); only those peers open count; loopback is exempt; a warning at most once a minute while peers are refused for it
Connection without a stream	Closed after 20 seconds, on connections a peer opened
Known nodes	At most 1,000; the seeds are always kept
Outbound connections	At most 32 dialed, not counting configured nodes, which are dialed as well; among the others, one in each network group (IPv4 /16 or IPv6 /32)
Peer lists	At most the first 1,024 entries of a frame and 64 bytes per address; IP literals only, none with a named IPv6 scope or in brackets
Ping	Every 20 seconds; an answer not asked for is ignored; a peer that sends nothing during three unanswered pings is closed; answered only while the stream takes more
Slow streams	Closed once 48 KiB behind 16 KiB a 20-second period: the connection of a full stream, the stream of an unfinished frame
Peers and sporks	Published every 3 minutes
Gridnode announcements	Every 5 minutes; expire after 30 minutes
Reconnect pause	33 seconds plus 3 seconds per known node, counted up to 32: 33 to 129 seconds
Storage request timeout	10 seconds

Related pages

The Architecture overview shows where the network layer sits in the daemon, and Erasure coding explains what the stored fragments are.

In the source

• application/src/main/java/org/unigrid/hedgehog/server/p2p/P2PServer.java builds the QUIC server
• application/src/main/java/org/unigrid/hedgehog/transport/initializer/PeerPipeline.java builds the stages of a peer stream for the server and the client side
• application/src/main/java/org/unigrid/hedgehog/client/p2p/P2PClient.java is the outbound connection
• application/src/main/java/org/unigrid/hedgehog/server/p2p/TopologyThread.java dials known nodes and prunes unreachable ones
• application/src/main/java/org/unigrid/hedgehog/model/Network.java holds protocol identifiers, seeds and limits, and hasPreview and hasLedger, which choose the stages
• application/src/main/java/org/unigrid/hedgehog/transport/handler/UnclaimedFrameHandler.java drops the frames no stage takes
• application/src/main/java/org/unigrid/hedgehog/model/network/Packet.java defines the packet types
• application/src/main/java/org/unigrid/hedgehog/model/network/Topology.java is the registry of known nodes and gridnodes
• application/src/main/java/org/unigrid/hedgehog/model/network/ConnectionLimiter.java caps the connections from one address and in all
• application/src/main/java/org/unigrid/hedgehog/model/option/NetOptions.java holds the network options, --netmaxinbound among them
• application/src/main/java/org/unigrid/hedgehog/transport/handler/StreamlessConnectionHandler.java closes a connection that holds no stream for 20 seconds
• application/src/main/java/org/unigrid/hedgehog/transport/handler/EndedStreamHandler.java closes a dialed connection whose peer ends the main stream
• application/src/main/java/org/unigrid/hedgehog/model/network/codec/FrameDecoder.java bounds each frame by its packet type
• application/src/main/java/org/unigrid/hedgehog/model/spork/BoardKeysEncoder.java and BoardKeysDecoder.java are the wire form of the board keys chain
• application/src/main/java/org/unigrid/hedgehog/model/gridnode/GridnodeSignature.java is what a gridnode announcement signs, and model/network/codec/GridnodeEncoder.java and GridnodeDecoder.java its wire form
• application/src/main/java/org/unigrid/hedgehog/transport/handler/SporkRecipients.java decides which peer each spork type may go to, and SporkProtocol.java whether the connection reads the board keys and the vesting spork of this release
• application/src/main/java/org/unigrid/hedgehog/model/network/PeerProtocol.java tells whether a connection speaks the proxy, and transport/initializer/StreamRouter.java sets up the main and the chain stream and closes further main streams
• application/src/main/java/org/unigrid/hedgehog/transport/handler/ contains the handler for each packet
• application/src/main/java/org/unigrid/hedgehog/transport/schedule/ contains the periodic sends, the sporks among them with the board keys first