CDI container and component lifecycle

After 0.0.8. The storage repair job, the S3 endpoint, the SOCKS proxy and its chains, and the ledger are on the master branch and not part of a release yet. Release 0.0.8 starts only the peer-to-peer listener and the REST interface eagerly, and so does release 0.0.9 without the hidden --preview switch.

A Hedgehog node is a single program that runs several long-lived services at once: a peer-to-peer listener, a local web interface, a record of the network’s parameters and, when switched on, a storage repair job, the S3 endpoint, the SOCKS proxy and the ledger. This page describes the part that creates those services, starts them in the right order and shuts them down cleanly.

Hedgehog is written in Java and delegates this work to a standard dependency-injection container (CDI, implemented by Weld). Each service is declared as a managed component, the container builds them and hands each one the others it needs, and nothing in the node wires itself together by hand. The wider picture is in the Architecture overview.

How a node starts

The hedgehog daemon command reads its options, starts the container and then simply waits. Everything that matters happens while the container starts:

sequenceDiagram
    autonumber
    participant M as Hedgehog.main
    participant PC as picocli
    participant W as Weld container
    participant EE as EagerExtension
    participant S as P2PServer, RestServer<br/>and others
    participant D as Daemon

    M->>PC: execute(args)
    PC->>PC: bind NetOptions / RestOptions
    PC->>W: Daemon.run() starts the container
    W->>W: discover components
    W->>EE: ProcessBean for every component
    EE->>EE: remember those marked @Eager
    W->>EE: AfterDeploymentValidation
    EE->>S: create now (@PostConstruct)
    S->>S: bind P2P and REST ports,<br/>start topology thread
    W->>W: install shutdown hook
    W->>D: ContainerInitialized
    D->>D: read the spork database,<br/>start snapshot-install
    W-->>PC: container returned, main thread waits

The container creates components lazily, on first use, so a component that must be running from the start is marked @Eager. A small extension, EagerExtension, finds those components once discovery is finished and creates them before the container reports that it is ready. This is how the peer-to-peer server and the REST server are already running when startup completes. The S3 gateway, the SOCKS server and the ledger service are created the same way, and do nothing further when --no-s3, --socks-port 0 or the absence of --ledger switches them off.

When the container fires ContainerInitialized, Daemon reads the spork database on the main thread and then starts the virtual thread snapshot-install, which downloads the legacy chain snapshot when the node holds none it can open. Startup does not wait for the download, but it waits for the spork database: the install judges a snapshot by the board keys the node holds, and the changes to them that the release ships and that the database stores count only from that read on.

A component of a part that the release leaves off is also marked @Preview: the ledger service, the S3 gateway, the storage repair job, the chain host and exit and the SOCKS server. EagerExtension creates an @Eager @ApplicationScoped component marked @Preview only when the daemon runs with --preview, so a node without it builds no tunnel, GeoIP lookup or storage directories at start and runs none of their threads. Such a component is not vetoed: one that another injects is still created on its first use. PreviewSetup, which Daemon.run() calls before the container starts, refuses --ledger, --gridnode and a --socks-port other than 0 without --preview.

A component that throws while the container starts ends the start, and the daemon then prints one line on standard error and exits with code 1. When the peer-to-peer or REST server cannot bind its port, the line names the address, the server and the options that set the address, as in Cannot listen on <host>:<port> for P2P (--nethost, --netport): <reason> or Cannot listen on <host>:<port> for REST (--resthost, --restport): <reason>. For any other failure the line is the innermost cause. The whole trace is logged at error level as The node failed to start and shows with -v. Other commands that fail as they run still show their trace.

Shared state and its guards

The list of known peers (Topology) and the map from open connections to peers (ChannelMap) are used by many threads. Their methods carry @Protected and @Lock annotations, and ProtectedInterceptor turns those into a read-write lock: any number of readers at once, or a single writer. There is one lock per component instance, so the topology and the channel map do not block each other. The gridnodes of the topology have a read-write lock of their own, because an offer reaches them through calls on the topology itself, which the container does not intercept.

Some objects are not created by the container at all. Network handlers and scheduled tasks are built by the networking library, and the REST resources by the web framework, so they look their dependencies up in the container on demand (CDIUtil, CDIBridgeResource). The same pattern lets the REST layer reach the peer network described in Peer-to-peer network protocol.

Some values are produced rather than declared. The application directory is created per use, and the spork database is shared by every user of it and loaded from disk on first use. Sporks are the signed network parameters described in Grid sporks; the container saves the database again when it shuts down.

How a node stops

POST /stop on the local REST interface, or the hedgehog cli stop command, is answered with 202 and stops the node once that answer is written, or two seconds after the request at the latest, by waking the waiting main thread. The stop is remembered, so one that the REST server takes while the container is still starting ends the node once it has started.

The program exits, the container’s shutdown hook runs, and each component’s @PreDestroy method closes its sockets and threads; that of Daemon interrupts snapshot-install, which ends a download or the pause before the next attempt. The spork database is saved at that point. What the components log as they shut down reaches the console at the level that -v sets, such as the warning of a last save of the spork database that fails, which -vv shows. Only Weld’s line Weld SE container <id> shut down by shutdown hook is left out.

Key facts

Item Value
Peer-to-peer listener QUIC over UDP, default 0.0.0.0:52883
REST interface default localhost:52884
Components started eagerly P2PServer, RestServer; with --preview also RepairService, S3Gateway, SocksServer, LedgerService, ChainHost, ChainExit
Event-loop threads 4 per server
Wait for the answer to a stop 2 seconds at most
Exit code when a component fails to start 1
QUIC data in flight 16 MiB per connection and stream
QUIC streams and idle timeout 16 streams, 15 minutes
hedgehog subcommands bootstrap, cli, daemon, s3, util

In tests

The test suite starts real containers instead of mocking components. Each test class gets its own container, and the server tests start 20 independent ones to check that nodes bind distinct ports. A test container does not start the eager components unless it asks for the extension, so tests start their servers explicitly. A test of a part that the release leaves off turns --preview on with PreviewMockUp. How the suite is built and run is covered in Build, testing and native image.

In the source

  • application/src/main/java/org/unigrid/hedgehog/Hedgehog.java - the entry point and its subcommands.
  • application/src/main/java/org/unigrid/hedgehog/command/Daemon.java - the daemon command.
  • application/src/main/java/org/unigrid/hedgehog/model/cdi/CDIContext.java - starts the container and waits.
  • application/src/main/java/org/unigrid/hedgehog/model/cdi/EagerExtension.java - eager component startup.
  • application/src/main/java/org/unigrid/hedgehog/model/cdi/Preview.java - the marker of the components that start only with --preview.
  • application/src/main/java/org/unigrid/hedgehog/model/PreviewSetup.java - the refusal of the options of the parts left off.
  • application/src/main/java/org/unigrid/hedgehog/model/cdi/ProtectedInterceptor.java - the read-write guard.
  • application/src/main/java/org/unigrid/hedgehog/server/p2p/P2PServer.java - the peer-to-peer listener.
  • application/src/main/java/org/unigrid/hedgehog/server/rest/RestServer.java - the REST server.
  • application/src/main/java/org/unigrid/hedgehog/server/ListenException.java - a bind that failed, naming the address, the server and its options.
  • application/src/main/java/org/unigrid/hedgehog/producer/ - producers for shared objects.