← Back

Molina documentation

Test a network partition in a container cluster

Servers in a cluster talk to agree on who is in charge. Cut the connection and each side sees only part of the group. That is when two leaders appear, or a node removes itself.

Molina puts each node in its own world on one virtual network and cuts the links. The cut happens at virtual time, at a request chosen before the run, so the same failure can be run again.

What happened in the runs

Three Akka.NET nodes ran at their own settings, with a split-brain resolver that downs a node unreachable for twenty seconds. Seed 7 cut node 2 off for 100,000 ms of virtual time. The switch dropped 42 frames on 4 links. The two other nodes removed node 2 and no one else. In the main run's report alone, the cut-off node removed every member, itself included, and named itself leader.

The partition was replayed in 3 pairs. In every pair all 1,476 frames had the same outcome, the same delivery time for delivered frames and the same contents, and both runs cut the same 42 frames. A NATS pair under a partition had 102 frames numbered and 9 cut, and matched byte for byte in 3 of 3 replays.

What it does not prove

Three pairs leave room for a replay that differs more often than not. One seed, one machine, one hypervisor build. A result tells you what happened in those runs.

The Akka.NET record The NATS record