Event Graph Recovery

This page is for operators running an application that stores data in EventGraph, for example darkirc. The safest recovery model is: stop the process, preserve the datastore, restart once to let automatic repairs run, and only then decide whether a restore or clean resync is needed.

Data layout

EventGraph stores several sled trees in the application’s datastore. For DarkIRC this is the --datastore path, defaulting to ~/.local/share/darkfi/darkirc_db. This is not the same as the P2P p2p_datastore path.

Important trees are:

DarkIRC also stores local account secrets in the same sled database under darkirc_account_* and darkirc_account_default. Treat the datastore as secret material. A clean resync can rebuild network DAG state, but it will not recover local account nullifiers and trapdoors unless they were backed up or exported with NickServ INFO <account_name>.

First response

1. Stop the application. Do not edit or copy sled files while the process is running.
2. Copy the whole datastore directory and the config file before changing anything.
3. Keep the exact binary, config, and logs that produced the failure.
4. Start the node once with the same config and watch the event graph logs.

Startup already performs important recovery work. If it succeeds and logs a completed rebuild, prefer keeping the repaired datastore over manual surgery.

Automatic repairs

The static DAG is authoritative for RLN identity state. At startup, EventGraph scans static-dag, sorts static events by (layer, event_id), and compares the result with:

• rln-identity-leaves
• rln-historical-roots-ordered
• rln-historical-roots-by-value

If these side tables are stale or incomplete, startup clears and rebuilds them from static-dag. Expected log lines include:

[EVENTGRAPH] RLN state audit: ... consistent=false
[EVENTGRAPH] Rebuilding RLN state: ...
[EVENTGRAPH] RLN state rebuild complete

Startup also audits static-dag-blobs. Missing pregenerated registration blobs are deterministic and are repaired with the genesis guard blob. Slash blobs and future staked-registration blobs are not reconstructible and must not be fabricated.

Common failures

Corrupted RLN roots or identity leaves

Symptoms:

• RLN state audit reports consistent=false.
• Signal verification rejects roots that should be historical.
• Startup logs RLN identity leaf audit failed.

Safe response:

• Restart with the same config and let the automatic rebuild complete.
• Do not delete static-dag; it is the source used to rebuild the RLN side tables.
• If rebuild fails because static-dag itself is unreadable, restore the datastore from backup or perform a clean resync from healthy peers.

Missing static blobs

Symptoms:

• static blob audit reports repaired > 0.
• static blob audit reports unrecoverable > 0.
• Logs mention static event ... is missing its RLN blob and cannot be reconstructed.

Safe response:

• If only repaired is nonzero, startup restored missing pregenerated guard blobs and no manual action is needed.
• If unrecoverable is nonzero, restore a backup that still has static-dag-blobs, or rebuild the node from peers that can serve complete static events and blobs.
• Do not invent slash or staked-registration blobs. A node that cannot serve original static blobs cannot safely help late joiners verify that history.

Missing rotating DAG blobs

Symptoms:

• Peers cannot sync current rotating events from this node.
• Logs mention declining to serve an event because the blob is missing.
• A local non-genesis event exists in a <timestamp> tree but its RLN signal blob is absent from dag-blobs.

Safe response:

• Restore from a datastore backup made before the blob loss.
• If the lost slot is disposable, start from a clean datastore with the same config and sync from healthy peers.
• Do not copy only dag-blobs from another network or config. RLN signal proofs are bound to the event and application domain.

Corrupted rotating DAG or header trees

Symptoms:

• Startup fails while opening DAG slots.
• History ordering or content sync fails on corrupt event/header records.
• Sync repeatedly returns DagSyncFailed for one slot.

Safe response:

• For current history, restore a full datastore backup if local account data matters.
• For disposable rotating history, move the datastore aside and let the node create a fresh store, then sync from peers.
• Do not edit individual sled tree files in place. The <timestamp> and headers_<timestamp> trees must agree with each other, and event blobs must agree with the event bodies.

Config mismatch

Symptoms:

• Layer-1 headers are rejected as referencing a foreign genesis.
• Peers return headers that fail HeaderIsInvalid.
• Static or rotating sync fails across otherwise reachable peers.
• RLN signals from peers never verify.

Safe response:

• Confirm every node in the network uses the same event graph consensus parameters: initial_genesis, hours_rotation, genesis_contents, and the app-provided pregenerated RLN commitment set.
• For DarkIRC these constants are compiled into bin/darkirc/src/main.rs and bin/darkirc/src/genesis_commits.rs.
• Do not reuse a datastore across networks with different consensus parameters. Fix the binary/config first, then start with a datastore created under the same parameters.

Clean resync

Use clean resync only after preserving the original datastore.

1. Stop the node.
2. Back up the datastore and config.
3. If local DarkIRC accounts matter, export each account with NickServ INFO <account_name> before discarding the datastore.
4. Start the node with an empty datastore and the same event graph consensus configuration as the network.
5. Let static sync and rotating DAG sync complete from healthy peers.

A clean resync is safe for public network state only if enough peers still serve complete events and blobs. It is not a replacement for backing up local account secrets.