Peer-to-peer network protocol

After 0.0.8. The GRIDNODE packet, gridnode announcements and the seven storage packets are on the master branch and not in release 0.0.8, which uses the same protocol identifiers without them.

Every Hedgehog node is both a client and a server: it keeps encrypted connections open to other nodes and uses them to find more peers, agree on network-wide settings, advertise gridnodes and move storage requests around. There is no central server; a new node needs only the six seed addresses to join. This page is a first overview of how that communication works, for readers who want the idea before the code.

The idea in short

  • Transport. Nodes talk QUIC, a modern encrypted protocol that runs over UDP. One connection carries one bidirectional stream, and everything between two nodes travels on that stream in both directions.
  • Messages. Each message is a small binary frame with an eight-byte header: a magic number, a packet type and the payload length. The receiving side reads the type and hands the payload to the matching decoder and handler.
  • Joining. A node dials the seeds (seed1.unigrid.org to seed6.unigrid.org), learns more peers from them and keeps dialing the peers it knows. A node that cannot be reached is dropped from the list; it returns if a seed or another peer announces it again.
  • Compatibility. Both sides offer the protocol identifiers hedgehog/0.0.4 and gridspork/0.0.4 during the handshake. A node that shares none of them cannot connect, and the other side logs that it is running an incompatible release.

What nodes say to each other

Twelve packet types are live on a connection. They fall into four groups.

Group Packets Purpose
Presence HELLO, PING The dialing node announces the port it listens on; both sides measure latency
Topology PUBLISH_PEERS, GRIDNODE Share the list of known nodes and the signed list of active gridnodes
Settings PUBLISH_SPORK Spread grid sporks, the signed network-wide settings (see Grid sporks)
Storage Seven packets, from STORE_FRAGMENT to STORAGE_ACK Store, fetch, count and delete data fragments (see Network storage)

The storage packets can travel in either direction, so whichever node dialed, either side can ask and either side can answer. Each request carries an id, and the reply is matched to it by id, stream and type.

sequenceDiagram
    participant A as Dialing node
    participant B as Remote node

    A->>B: QUIC handshake (offers hedgehog/0.0.4 and gridspork/0.0.4)
    A->>B: Open one bidirectional stream
    A->>B: HELLO with the port A listens on
    Note over B: B records A as a known node
    A->>B: Stored sporks and held proposals
    B->>A: Stored sporks and held proposals
    loop while connected
        A-->>B: PING, PUBLISH_PEERS, PUBLISH_SPORK, GRIDNODE
        B-->>A: PING, PUBLISH_PEERS, PUBLISH_SPORK, GRIDNODE
    end

How the network stays in sync

  • Peers. Every three minutes each node sends its whole list of known nodes to every connection. Receivers add the ones they have not seen, so knowledge of the network spreads without any directory.
  • Sporks. A spork is accepted only when it carries two signatures from different network keys and extends the signature history of the one it replaces. A node that accepts one passes it on to all of its connections. Proposals that have only one signature are held and forwarded in the same way.
  • Gridnodes. Each gridnode signs an announcement of its address and status with its own key. A receiver accepts an entry only if the signature verifies, the timestamp is recent, and the entry is newer than the one it already has. It then forwards the entry to every peer except the sender. Entries that are not refreshed for 30 minutes expire.
  • Latency. Both sides ping every three minutes. The node that dialed records the round trip, and the REST interface reports it with each node (see REST interface).

Security

Connections are encrypted and integrity-protected by QUIC, but nodes do not authenticate each other at the transport level: each server creates a fresh self-signed certificate at start-up and clients accept any certificate. Trust therefore sits in the data rather than in the connection. Sporks and gridnode announcements are verified against signatures, and the network keys that govern sporks are described in Grid sporks. The server also uses QUIC address-validation tokens, which are tied to the peer address and become invalid when the node restarts.

Key facts

Item Value
Default port 52883 (--netport or -p)
Default bind address 0.0.0.0 (--nethost or -H)
Seed nodes seed1.unigrid.org to seed6.unigrid.org; turn off with --no-seeds
Protocol identifiers hedgehog/0.0.4, gridspork/0.0.4
Frame 8-byte header, magic 0xBABE, at most 256 MiB
Connection limits 512 streams, 15 minutes idle timeout, 2 second connect timeout
Ping, peers, sporks Every 3 minutes
Gridnode announcements Every 5 minutes; expire after 30 minutes
Reconnect pause 30 seconds plus 3 seconds per known node
Storage request timeout 10 seconds

The Architecture overview shows where the network layer sits in the daemon, and Erasure coding explains what the stored fragments are.

In the source

  • application/src/main/java/org/unigrid/hedgehog/server/p2p/P2PServer.java builds the QUIC server and its pipeline
  • application/src/main/java/org/unigrid/hedgehog/client/p2p/P2PClient.java is the outbound connection
  • application/src/main/java/org/unigrid/hedgehog/server/p2p/TopologyThread.java dials known nodes and prunes unreachable ones
  • application/src/main/java/org/unigrid/hedgehog/model/Network.java holds protocol identifiers, seeds and limits
  • application/src/main/java/org/unigrid/hedgehog/model/network/Packet.java defines the packet types
  • application/src/main/java/org/unigrid/hedgehog/model/network/Topology.java is the registry of known nodes and gridnodes
  • application/src/main/java/org/unigrid/hedgehog/model/network/handler/ contains the handler for each packet
  • application/src/main/java/org/unigrid/hedgehog/model/network/schedule/ contains the periodic sends