Peer-to-peer network protocol

After 0.0.8. The GRIDNODE packet, gridnode announcements and the seven storage packets are on the master branch and not in release 0.0.8, which uses the same protocol identifiers without them.

Every Hedgehog node is both a client and a server: it keeps encrypted connections open to other nodes and uses them to find more peers, agree on network-wide settings, advertise gridnodes and move storage requests around. There is no central server; a new node needs only the six seed addresses to join. This page is a first overview of how that communication works, for readers who want the idea before the code.

The idea in short

  • Transport. Nodes talk QUIC, a modern encrypted protocol that runs over UDP. One connection carries one bidirectional stream, and everything between two nodes travels on that stream in both directions. A connection allows at most 16 streams and 16 MiB in flight on the connection and on each stream; it is closed after 15 minutes without traffic, and a dial that takes more than 2 seconds fails.
  • Messages. Each message is a small binary frame with an eight-byte header: a magic number, a packet type and the payload length. The receiving side reads the type and hands the payload to the matching decoder and handler.
  • Joining. A node dials the seeds (seed1.unigrid.org to seed6.unigrid.org), learns more peers from them and keeps dialing the peers it knows. A node that cannot be reached is dropped from the list; it returns if a seed or another peer announces it again.
  • Compatibility. Both sides offer the protocol identifiers hedgehog/0.0.4 and gridspork/0.0.4 during the handshake. A node that shares none of them cannot connect, and the other side logs that it is running an incompatible release. The socks-proxy branch offers other identifiers, see SOCKS proxy.

What nodes say to each other

Twelve packet types are live on a connection. They fall into four groups.

Group Packets Purpose
Presence HELLO, PING The dialing node announces the port it listens on; both sides measure latency
Topology PUBLISH_PEERS, GRIDNODE Share the list of known nodes and the signed list of active gridnodes
Settings PUBLISH_SPORK Spread grid sporks, the signed network-wide settings (see Grid sporks)
Storage Seven packets, from STORE_FRAGMENT to STORAGE_ACK Store, fetch, count and delete data fragments (see Network storage)

The storage packets can travel in either direction, so whichever node dialed, either side can ask and either side can answer. Each request carries an id, and the reply is matched to it by id, stream and type.

sequenceDiagram
    participant A as Dialing node
    participant B as Remote node

    A->>B: QUIC handshake (offers hedgehog/0.0.4 and gridspork/0.0.4)
    A->>B: Open one bidirectional stream
    A->>B: HELLO with the port A listens on
    Note over B: B records A as a known node
    A->>B: Stored sporks and held proposals
    B->>A: Stored sporks and held proposals
    loop while connected
        A-->>B: PING, PUBLISH_PEERS, PUBLISH_SPORK, GRIDNODE
        B-->>A: PING, PUBLISH_PEERS, PUBLISH_SPORK, GRIDNODE
    end

How the network stays in sync

  • Peers. Every three minutes each node sends its whole list of known nodes to every connection. Receivers add the ones they have not seen, so knowledge of the network spreads without any directory. A peer list frame carries IP literals only, so a host name is dropped without a DNS lookup. A node takes at most 1,024 peers from it, each address at most 64 bytes, and skips the entries past the cap. A peer with a public address cannot add loopback, private, link-local, unspecified, multicast or unique-local addresses, so a private network on its own LAN or on loopback still works. The topology holds at most 1,000 known nodes and always keeps the seeds. A node keeps at most 32 outbound connections and dials the seeds before the other nodes, which it takes in random order. On the s3 branch, a HELLO from a known peer replaces the connection that peer made before, while a connection the node dialed itself stays.
  • Sporks. A spork is accepted only when it carries two signatures from different network keys and extends the signature history of the one it replaces. A node that accepts one passes it on to all of its connections. Proposals that have only one signature are held and forwarded in the same way.
  • Gridnodes. Each gridnode signs an announcement of its address and status with its own key. A receiver accepts an entry only if the signature verifies, the timestamp is recent, and the entry is newer than the one it already has. It then forwards the entry to every peer except the sender. Entries that are not refreshed for 30 minutes expire.
  • Latency. Both sides ping every 20 seconds: a NAT, Docker’s among them, may forget a UDP flow that has been quiet for 30 seconds. The node that dialed records the round trip, and the REST interface reports it with each node (see REST interface). A ping answer the node never asked for, or a second answer, is ignored. On the s3 branch the QUIC idle timeout is 1 minute, so a dead or restarted peer is noticed within about a minute.

Security

Connections are encrypted and integrity-protected by QUIC, but nodes do not authenticate each other at the transport level: each server creates a fresh self-signed certificate at start-up and clients accept any certificate. Trust therefore sits in the data rather than in the connection. Sporks and gridnode announcements are verified against signatures, and the network keys that govern sporks are described in Grid sporks. The server also uses QUIC address-validation tokens, which are tied to the peer address and become invalid when the node restarts. One remote address may hold at most eight connections at a time; loopback is exempt, so a forwarder or several nodes on one host still work.

Key facts

Item Value
Default port 52883 (--netport or -p)
Default bind address 0.0.0.0 (--nethost or -H)
Seed nodes seed1.unigrid.org to seed6.unigrid.org; turn off with --no-seeds
Protocol identifiers hedgehog/0.0.4, gridspork/0.0.4; hedgehog/0.0.5 first on the socks-proxy branch
Frame 8-byte header, magic 0xBABE; the size cap depends on the packet type
Spork frame At most 16 MiB
Storage fragment frame At most 256 MiB
Ledger transaction At most 163 bytes, behind a 4-byte length
Ledger consensus message At most 528,441 bytes, behind a 4-byte length
Ledger block batch At most 8 MiB plus 64 KiB, behind a request id and a 4-byte length
QUIC limits 16 streams, 16 MiB in flight, 15 minutes idle timeout (1 minute on the s3 branch), 2 second connect timeout
Connections per address At most 8; loopback is exempt
Known nodes At most 1,000; the seeds are always kept
Outbound connections At most 32, the seeds first
Peer lists At most 1,024 peers taken from a frame and 64 bytes per address; IP literals only
Ping Every 20 seconds; an answer not asked for is ignored
Peers and sporks Published every 3 minutes
Gridnode announcements Every 5 minutes; expire after 30 minutes
Reconnect pause 30 seconds plus 3 seconds per known node
Storage request timeout 10 seconds

The Architecture overview shows where the network layer sits in the daemon, and Erasure coding explains what the stored fragments are.

In the source

  • application/src/main/java/org/unigrid/hedgehog/server/p2p/P2PServer.java builds the QUIC server and its pipeline
  • application/src/main/java/org/unigrid/hedgehog/client/p2p/P2PClient.java is the outbound connection
  • application/src/main/java/org/unigrid/hedgehog/server/p2p/TopologyThread.java dials known nodes and prunes unreachable ones
  • application/src/main/java/org/unigrid/hedgehog/model/Network.java holds protocol identifiers, seeds and limits
  • application/src/main/java/org/unigrid/hedgehog/model/network/Packet.java defines the packet types
  • application/src/main/java/org/unigrid/hedgehog/model/network/Topology.java is the registry of known nodes and gridnodes
  • application/src/main/java/org/unigrid/hedgehog/model/network/ConnectionLimiter.java caps the connections from one address
  • application/src/main/java/org/unigrid/hedgehog/model/network/codec/FrameDecoder.java bounds each frame by its packet type
  • application/src/main/java/org/unigrid/hedgehog/model/network/handler/ contains the handler for each packet
  • application/src/main/java/org/unigrid/hedgehog/model/network/schedule/ contains the periodic sends