High availability
The LeilFS master holds the whole filesystem namespace in RAM and is the single coordinator of the cluster. High availability (HA) protects against loss of the master by keeping one or more shadow masters that receive metadata in real time, and — optionally — by using the uRaft consensus daemon to fail over automatically.
This page builds on the basic master/shadow setup in Service configuration; read that first for the master and shadow personalities.
Shadow masters
A shadow master runs the leil-master service with PERSONALITY = shadow in leil-master.cfg. It connects to the active master, continuously downloads changelog entries, and keeps an up‑to‑date in‑memory copy of the metadata so it can take over quickly.
# /etc/saunafs/leil-master.cfg on a shadow host
PERSONALITY = shadow
MASTER_HOST = 192.168.10.10 # active master IP
MASTER_PORT = 9419
DATA_PATH = /var/lib/saunafs
Keep leil-exports.cfg, leil-goals.cfg and leil-topology.cfg identical on every master and shadow. A shadow that is promoted must serve exports and goals exactly as the former master did.
Shadow masters alone give you a warm standby but do not fail over on their own — promotion is manual (see below), unless you add uRaft.
Automatic failover with uRaft
uRaft is a LeilFS implementation of the Raft consensus algorithm. It runs as the saunafs-uraft service alongside each master/shadow, elects exactly one leader among the nodes, and moves a floating IP onto the elected leader so clients always reach the current master at one address.
For automatic HA, the master runs with PERSONALITY = ha-cluster-managed and uRaft controls which node is active.
uRaft is configured in its own file (leil-uraft.cfg), separate from leil-master.cfg. Key options:
| Option | Meaning | Default |
|---|---|---|
URAFT_ID | This node's id — 0‑based, matches its position in the node‑address list | — |
URAFT_NODE_ADDRESS | Address (and optional port) of a uRaft node; repeated once per node | — |
URAFT_PORT | UDP port for uRaft peer messages | 9427 |
URAFT_STATUS_PORT | TCP port for the uRaft status endpoint | 9428 |
LOCAL_MASTER_ADDRESS | Address of the local master this uRaft node manages | localhost |
LOCAL_MASTER_MATOCL_PORT | Client port of the local master | 9421 |
ELECTION_TIMEOUT_MIN / ELECTION_TIMEOUT_MAX | Election timeout window (ms) | 400 / 600 |
HEARTBEAT_PERIOD | Milliseconds between heartbeats | 20 |
QUORUM_LOSS_GRACE_HEARTBEATS | Missed heartbeats tolerated before quorum is lost | 5 |
URAFT_FLOATING_IP / URAFT_FLOATING_NETMASK / URAFT_FLOATING_IFACE | Floating IP moved onto the elected leader, its netmask and the interface to manage it | — |
URAFT_ELECTOR_MODE | 0 = can become leader (default); 1 = elector‑only (votes but never leads) | 0 |
Every uRaft node must list the same set of URAFT_NODE_ADDRESS entries in the same order, and each node's URAFT_ID must match its own position in that list.
Failover process
When the active master becomes unreachable, uRaft holds an election. The shadow with the most up‑to‑date metadata (highest changelog position) that also wins votes from a majority (quorum) of nodes becomes the new master, and the floating IP moves to it. Failover typically completes within a few seconds, bounded by the election‑timeout window.
After a failover, do not bring the old master back as an active master. Investigate the cause first, then rejoin it to the cluster as a shadow. Two nodes both believing they are the master ("split brain") can corrupt metadata.
Without uRaft, promote a shadow manually: stop the failed master, ensure the chosen shadow has the latest changelog, switch its PERSONALITY to master, and repoint clients (or move the service IP) to it.
Recommended HA topology
| Topology | Notes |
|---|---|
| 1 master + 1 shadow + 1 metalogger | Minimal production HA. The shadow provides fast takeover; the metalogger adds an independent metadata backup. A 2‑node quorum cannot elect on its own, so promotion may be manual. |
| 1 master + 2 shadows | Best consensus HA. A 3‑node quorum tolerates the loss of one node and can elect automatically with uRaft. |
| 1 master + 2 shadows + 1+ metaloggers | Recommended for critical systems — full redundancy at both the consensus and backup levels. |
uRaft needs a majority of nodes to elect a leader. Three nodes tolerate one failure; two nodes cannot form a quorum after a single failure. Always size the cluster with an odd number of consensus nodes.
If every master is lost, recover from a metalogger — see Metadata recovery.