Metadata recovery

The master holds the authoritative filesystem metadata. If the active master fails, recovery depends on what redundancy is in place: a shadow master for fast takeover, or a metalogger for a full rebuild when no master survives.

caution

This is a break‑glass procedure. Work on a copy of the metadata where possible, and never run two masters against the same cluster at once — a split brain can corrupt metadata irreversibly.

---

If a shadow master is available

1. Stop the failed master so it cannot rejoin as an active master.
2. Promote the shadow with the most up‑to‑date metadata:
   • With uRaft, election and the floating‑IP move happen automatically once the failed node drops out of quorum.
   • Without uRaft, promote manually — confirm the shadow has the latest changelog, set PERSONALITY = master in its leil-master.cfg, and restart it.
3. Verify filesystem consistency (mount a client, check that recent files and directories are present).
4. Re‑establish at least one new shadow before returning to normal production, so you are not left without redundancy.

---

If no master survives — rebuild from a metalogger

1. Stop all remaining LeilFS services.

2. Copy the metalogger's DATA_PATH contents (metadata image + changelogs) to the new master host, into the master's DATA_PATH.

3. Reconstruct the metadata by replaying the changelogs:

   sudo leil-master -a

   leil-master -a (--auto-recovery) applies the retained changelog entries on top of the last metadata image to rebuild state up to the last entry the metalogger received.

4. Start the new master:

   sudo systemctl start saunafs-master

5. Verify filesystem consistency, then re‑establish shadow master(s) and metalogger(s) before returning to production.

note

Recovery restores metadata up to the last changelog entry the metalogger received. Any writes the master accepted after the metalogger's last sync are not recoverable — size META_DOWNLOAD_FREQ and BACK_LOGS in leil-metalogger.cfg accordingly.