With Great Uptime Comes Great Duplication
Functional Programming Isn't Just for Academics — Part 22
I have asked all of my neighbors to call the fire department if they see smoke billowing out of my house. If they send too many trucks because too many called, my neighbors will have something to laugh about while they watch my house burn down. We probably do not want them sending all the trucks to one side of town in case a fire breaks out elsewhere, or shelling out more diesel than they need to rolling unneeded trucks, but as a homeowner, I do not worry about idempotency. That is the fire department's problem.
Redundancy, fail-over, redelivery, supervision, automatic retry: each of them exists so that when something goes wrong the work still happens, and the way each of them accomplishes that is by attempting it another time. A system that reroutes around a dead node, redelivers an unacknowledged message, and replays an interrupted job is a system that does important things more than once. Not as a malfunction. On purpose, as a sign of health. Every mechanism you bought to keep work from being lost is a mechanism for doing it again. Which means the better your infrastructure gets, the more correctness you owe your model.
When a customer is charged twice on your system and your infrastructure, you go looking for the broken thing, and you may not find one. The queue delivered the message. The worker that crashed mid-charge was restarted by its supervisor in under a second, exactly as designed, and the restarted worker did what it was told: it charged the card. Every recovery mechanism worked. The transaction is what failed. Every component behaved the way you paid for it to behave. The second charge is not the price of a fragile system. It is the price of a robust one.
