Raft breaks in predictable ways.

Introduction

If you're digging into distributed systems, consensus is one of the first big topics you run into, and Raft is usually where people start. A couple weeks ago I sat down to actually learn how it works, and the first thing I did was read the Raft paper, a pretty approachable 18 pages.

I also like breaking things on purpose, so that's what this piece is: a five-node Raft cluster, three fault scenarios I threw at it, and the ugly parts that only show up once you actually stress the thing instead of just reading the paper.

Correct under ideal conditions is not the same as correct.

What we're watching for

Three scripted failure scenarios, each targeting a different way this can go wrong. No touching anything mid-run, I let each one play out end to end, then log the results so I can rerun it and get the same answer.

Building a Raft

Get a stable five-node cluster running, five being the smallest size that can tolerate a failure and still hold quorum. Each node runs as its own process, all equal participants in consensus.

Node responsibilities

• Raft state
  • currentTerm to track logical time
  • log[] to store replicated commands
  • commitIndex and lastApplied for state-machine progress
• RPC endpoints
  • RequestVote for leader election
  • AppendEntries for log replication and heartbeats
• State machine, a minimal key-value store that applies each entry once it's committed. Later experiments check applied state, not just the log, since a committed entry sitting unapplied would otherwise look like a healthy cluster.

Timing configuration

Election timeout and heartbeat interval are tunable, not hardcoded, since they're the levers that decide whether leader churn actually happens once faults show up in phase two.

type Config struct {
    ElectionTimeoutMin time.Duration
    ElectionTimeoutMax time.Duration
    HeartbeatInterval  time.Duration
}

Cluster deployment

Five nodes, Docker Compose, one container each on a shared private network. Any instability here just gets amplified once failures are introduced, so it has to be solid first.

The network layer

Take control of every message between nodes so failures can be injected deliberately. Real networks delay, drop, and block messages, so to simulate that we decouple Raft logic from direct communication, every message routes through one intermediary.

one control point, where we drop (packet loss), delay (latency), or block (partitions).

Implementation

An application-level transport interface inside the Go code routes every RPC through it, instead of raw HTTP or gRPC calls:

type Transport interface {
    Send(to string, msg Message) error
}

The controller isn't embedded per-node, it's a single centralized service in its own container that every node's Transport implementation calls into. Five independent copies of the partition matrix could disagree with each other, one node seeing a link as blocked while another still sees it open. One shared controller means one view of the network's state, so a partition is a partition everywhere at once, not a race between five inconsistent opinions:

type Network struct {
    mu        sync.RWMutex
    blocked   map[string]map[string]bool
    latency   time.Duration
    dropRate  float64
}

Before sending: check if the link is blocked, apply the random drop probability, apply the artificial delay, then forward if allowed. Every drop, delay, and block gets logged with a timestamp, the from/to pair, and the action, so later phases can line up injected faults against observed Raft behavior instead of guessing at cause and effect after the fact.

Client harness

Give the cluster something that behaves like a real client: it submits writes, deals with leadership changes, and keeps an honest record of what it believes actually succeeded. This is what turns silent data loss from theoretical into detectable.

Client responsibilities

• Leader tracking, submits writes to whichever node it currently believes is leader, and follows the redirect when a follower points it elsewhere
• Outcome recording, every submission ends in one of three states: submitted, committed, or timeout
• Client ledger, a record of every write the client believes was acknowledged, kept independently of the cluster's own log

type LedgerEntry struct {
    Key       string
    Value     string
    Outcome   string
    Timestamp time.Time
}

The ledger earns its keep at analysis time, by cross-checking it against the cluster's committed log after a run. A write the client logged as committed that never shows up in the applied state machine is exactly the silent data loss this whole simulation exists to catch.

Results

[1] Leader churn

Does flapping a partial partition cause runaway elections, and how bad does it get?

1. Term number over time

First question: does the term keep climbing the whole time the partition is unstable, or does the cluster eventually settle?

• Before the fault, everything sits flat at term 0. Healthy baseline: one election, then settle.
• Right after the marker, the term starts climbing almost immediately.
• From there it's a near-continuous staircase, roughly 0 to ~40. About 30 terms in ~70 seconds, so an election every 2–3 seconds.
• All five lines move together. It isn't just the cut-off nodes (N4/N5) racing ahead. The whole cluster's term gets dragged up.

2. Commit index over time

Same run, now looking at how much actually got committed while elections were firing.

• During the heaviest churn, commit index barely moved. A slow creep from ~2.55K to ~2.7K over about 90 seconds, a fraction of normal throughput.
• It didn't flatline completely either. Writes still trickled through between elections.

Total outage would have been a boring result. What showed up instead is an availability tax that scales with churn rate: more elections per minute, less commit progress in that window.

[2] Stalled writes

When a leader can't reach a quorum, do the writes it can't commit fail loudly, or get silently swallowed?

1. Write outcome rate

Every submission ends as committed or timeout. Plotting both rates side by side shows exactly what the client experiences the moment the leader loses its majority.

• Baseline sits around 1.5–1.65 writes/sec committed, with the green line notching down and back up as leadership hands off. Zero timeouts.
• At the marker, the two lines swap. Committed collapses to 0 and timeout spikes to its peak of ~2 writes/sec, the client hammering a leader that logs the write but can't get it past 2-of-5.
• Then a low timeout plateau, roughly 0.3 writes/sec, holds while committed stays pinned at 0. Nothing is getting through, and the client knows it.
• Once the majority side elects a new leader, green jumps straight back to ~1.65 and the timeouts vanish. A second shorter timeout patch around 13:32 lines up with the crash-and-re-elect window before it settles for good.

The point of this one isn't that writes failed, it's how they failed. Every write the leader couldn't commit surfaced as a timeout in the client ledger, never as a false committed. When N1's uncommitted entry got overwritten on rejoin, no acknowledged write went with it. That's the difference between a stall and silent data loss.

[3] Log divergence

A fully isolated leader keeps appending. How far does its log drift from the real one, and does it ever come back?

1. Divergence magnitude

For each follower, count how many log entries differ from the current leader's log at the same positions. Zero means everyone agrees; anything above zero is a fork waiting to be reconciled.

• Most of the run sits flat at 0. The brief single-entry blips from N2, N3, and N5 before the partition are ordinary handoff lag, one entry out of step for a tick, then reconciled.
• Inside the isolation window (shaded), N1 holds a steady divergence of 2 entries, the writes it appended alone while cut off from everyone. Those entries were never going to commit.
• The divergence is bounded. It doesn't run away, it caps at exactly the number of entries the isolated leader wrote by itself.
• The instant the partition heals, N1 drops straight from 2 back to 0. N3's AppendEntries backtracks past the mismatch and forces N1 to discard its divergent tail and copy the committed log instead.

This is the log-matching property doing exactly what we expect. A leader that lost its quorum forked its log, but the fork was bounded and self-correcting: erased the moment a real leader could talk to it again, with no operator intervention.

Conclusion

Three faults, three different failure modes. The interesting question was never does it fail, everything fails, but how. The term staircase, the writes that stalled into timeouts, the leader that forked its log and then had it repaired, all of it is behavior we expect.

The dangerous system is the one that looks healthy while quietly dropping an acknowledged write, and that's exactly the failure the client ledger existed to catch and never did.