thorwatch

How a churn works

THORChain holds user funds in shared vaults. No single node has the key: the key is split between up to 20 nodes (TSS — threshold signatures). Every few days the network replaces part of its validator set and creates fresh vault keys. That rotation is a churn. It runs fully automatically — nobody presses a button. The steps:

1. Pick the new team. The network removes a few nodes (the oldest, the worst-behaved, and — only when the active set is at capacity — the lowest-bonded) and selects up to 4 replacements — the standby nodes with the largest bonds. Those get status Ready.

2. Keygen — make new keys. The new member set runs TSS ceremonies to create fresh vaults. Every member must respond correctly, in multiple rounds, within a timeout. One overloaded, offline, or misconfigured node fails the ceremony for its whole vault. If exactly one member stalls a group, the protocol can name it: it fines it 720 slash points, jails it ~7.2 hours, and the next round runs without it. If two or more members of the same group are broken, nobody can be blamed, and the same groups fail again until an operator fixes a node or the member set changes. Attempts come every 720 blocks (~72 min), but a new keygen starts at most every second attempt. The new set goes live only when every group made a vault and all vaults list the same chains.

3. Migration — move the money. The old vaults send every asset to the new vaults, in rounds spread over hours. This doubles as a live fire drill: every churn proves that funds can actually leave the old keys. Outgoing nodes must stay online until their retiring vault has moved all its funds — going dark mid-migration bleeds slash points.

4. Settle up. The set rotates, the cycle's node rewards pay out, leaving nodes can unbond, new nodes start earning.

Why do it at all? Keys never guard funds for long, exits are tested in production, weak performers rotate out, and the bond market stays competitive — active slots are continuously auctioned to the highest bond that can prove its machine works on demand.

Who leaves, who joins

A churn is due every 43,200 blocks (~3 days), counted from the block where the new vaults went live, which is also the start of the fund migration (not its end). Attempts are skipped while an old vault still holds funds, so a slow migration can delay the next churn. At each churn the network marks some active nodes to leave:

• the oldest node (longest in the active set);
• the worst performer (most slash points relative to the others);
• up to 3 nodes on an old software version;
• the lowest bond — but only when the active set is full (120 nodes, mimir DESIREDVALIDATORSET). Below that size this rule does not fire;
• every node that asked to leave or was forced out.

Up to 4 standby nodes with the largest bonds join (status Ready). A candidate in jail at selection time is skipped. The live forecast is in Churn → Churn-out forecast.

Keygen retries and migration rounds

Keygen retries. If a keygen ceremony fails, the network retries every 720 blocks (~72 min); a new keygen starts at most every second attempt. The groups are built from the member set, so they stay the same from round to round until the set changes (a node is jailed, leaves or joins). Outcomes in Churn → Keygen attempts:
• Pending — ceremony in flight.
• Failed — exactly ONE broken member was in a group; TSS could blame it and jail it (720 points, ~7.2 h).
• No result — TWO OR MORE broken members shared a group; blame cannot be assigned and nothing concludes.
• Partial — some of the new vaults were created, not all.
• No keygen started — the tick passed without a new ceremony (no keygen block stored on-chain): the previous round's keygen was still pending, or churning was halted. Rounds therefore often run every second tick.
• Success — every group had zero (or one, then jailed) broken members.
So a group with one broken member is fixed after one round (that member is jailed and the groups change). A group with two or more broken members fails the same way every round, with no blame, until an operator repairs a node. The page names such a group under Keygen attempts.
How many vaults? ceil(keygen members ÷ ASGARDSIZE), where ASGARDSIZE (mimir, 20 today) is the maximum per vault. The members are the next set (active nodes, minus those leaving, plus the joiners), split into near-equal groups: 103 members → 6 vaults of 17–18. From 101 members on there are 6 vaults. A separate XMR monovault keygen with all members comes on top.
Keys made is not the same as switched in. Each round makes a full new set of vaults (InitVault). The network switches to it only if every vault in the set lists exactly the same chains. A vault lists a chain only when enough of its members attest a healthy scanner for it, so while a new chain (for example ZEC) is scanned by only part of the network, some vaults include it and others do not. That set is then skipped, the next round discards it (InactiveVault) and starts again. Rows marked inferred predate the tracker and come from on-chain jail records.

Migration rounds. After the new vaults exist, the old vaults send their funds over in rounds. A round starts every 720 blocks counted from the churn block, plus 5–6 blocks for outbound scheduling. Pool actions in the same block can shift a round by a few blocks. The countdown in Rotation & migration shows this window. Retiring nodes must stay online until their old vault is empty.

Slash points & jail

Slash points dock rewards, not bond. At churn payout each node earns proportionally to blocks active − slash points: one point = one block of rewards forfeited. Well-run nodes still accrue a baseline (network jitter, slow observations) — zero is impossible; the spread between operators is the signal.

Jail ≠ eviction. Jail only suspends signing duties; set membership changes only at a churn. A jailed candidate misses this churn (≈3 days of income to its bonders) but keeps its bond and its queue position.

What actually hits the bond (rare, severe — this is the network's security model, not an accident): an unauthorized outbound is slashed at 1.5× the tx value; double-signing is a severe bond slash. Routine downtime never touches principal.

Why "no result" happens — the blame limit: blame is a group verdict: each member builds a local blame list, and points are assigned only when the group AGREES on the culprit. That agreement is only reachable when exactly ONE member of a ceremony misbehaves. With two or more broken members in the same group, the local blame lists disagree and nobody can be blamed — the ceremony just never concludes. Groups change only when the member set changes, so a group with one bad node is cleared after one round (the node is jailed, 720 pts, ~7.2 h), but a group with two or more bad nodes keeps failing with nobody blamed until an operator repairs a node. That is when a churn stalls for many hours.

Reading this page: short jails (~60 blocks) on active nodes = routine signing trouble; long jails (~4,320 blocks) on Ready/standby candidates = failed entry ceremonies. A rolling population of short jails across the set means the signing layer is unstable — expect keygen retries to keep failing until the jail stream dries up.

Chain scanners (bifrost)

Each node runs bifrost next to thornode. For every connected chain, bifrost has a scanner: it reads each new block from the node's own chain daemon (bitcoind, zcashd, …) and reports the transactions for THORChain's vaults. The network accepts a transaction only when about 2/3 of nodes report the same thing, and it signs outbounds only when 2/3 of a vault's members agree. A node with a broken scanner cannot help with either.

Two views of scanner health. (1) On-chain: the last block height each node reported per chain (observe_chains). It only moves when the node reports a transaction, so a chain with no vault yet (ZEC, XMR) shows nothing there. (2) Bifrost: each node's own health endpoint (:6040/status/scanner), which this page asks every minute. It shows whether the scanner is healthy even before the chain goes live. Nodes behind a proxy do not answer and count as not healthy.

New chains. A new chain needs about 2/3 of active nodes with a healthy scanner before it can safely be switched on, because observations and signatures both need 2/3 agreement.

Block time and node operations

THORChain's own block time (~6 s) is the clock for everything the nodes do. Churn interval (43,200 blocks), retry attempts (720), jails (60 / 4,320), migration rounds (720) and the signing timeout (300 blocks) are all counted in THORChain blocks. If THORChain blocks slow down, all of these stretch in real time. Nodes also post their work as THORChain transactions (observations, keygen and keysign reports), so very full or slow blocks delay that work too.

Other chains' block times affect deposits and payouts, not signing. An inbound counts only after enough confirmations on its own chain, so a slow Bitcoin block delays swaps that start on Bitcoin. An outbound leaves the queue only after it is seen in a block on the destination chain. The signing itself (TSS keysign between the vault's members) does not wait for the other chain.

How to tell a slow chain from a signing problem. An outbound that is not signed within 300 THORChain blocks (~30 min, SigningTransactionPeriod) is taken back, the vault that failed is slashed, and the item is rescheduled. So when the oldest queued item keeps climbing to ~300 blocks and resetting, the vault's signers are failing, whatever the destination chain's block time is. A slow destination chain instead shows as items that were signed (they leave the signing stage) but wait for a block.

Queues

Outbound queue = funds the network owes users (swap outputs, refunds, migrations). The vault that holds the funds must sign each one (TSS keysign, 2/3 of its members). A healthy chain clears it in minutes. A failed keysign jails the blamed members for 60 blocks and the outbound is tried again.

Scheduled outbounds = large outbounds held back on purpose and released at a later block (outbound throttling). This limits how much value can leave per block. It is normal, not a stuck queue.

Swap queue = swaps waiting to execute, including streaming swaps that run as many small sub-swaps over several blocks.

Reading a stuck queue. Many old items on one chain with a halt flag = the network stopped signing on purpose (funds are safe, they wait). Many old items on one vault only = that vault's signers have a problem.

Halts (mimir)

Mimir = network settings set by node vote or by admin keys. The halts on this page:

• HALTCHURNING — no churn runs;
• HALT<CHAIN>CHAIN — the network stops processing that chain;
• HALTSIGNING<CHAIN> — no outbounds are signed on that chain; the queue grows;
• HALT<CHAIN>TRADING — no swaps with that chain's assets;
• SOLVENCYHALT<CHAIN>CHAIN — set automatically when a vault holds less on-chain than the network's records say;
• HALTWASMGLOBAL — the smart-contract (DEX) layer stops.

Event types

Support thorwatch