Retries are a load multiplier

Retries feel free. A request failed, so try again; the second attempt usually works. The trouble starts when every layer of a system thinks the same way.

The arithmetic

Three layers, each retrying three times, turn one user request into as many as 27 calls to the bottom of the stack. That happens exactly when the bottom is already failing — which is why outages so often get worse right after they begin.

Retry like you mean it

• Retry at one layer. Usually the one closest to the user, which knows whether the work still matters.
• Back off exponentially, with jitter. Without jitter, clients that failed together retry together.
• Budget retries. Allow them while they stay under, say, 10% of traffic; past that, fail fast.
• Only retry what is safe. Idempotent operations, or ones carrying an idempotency key.

Deadlines beat timeouts

Pass a deadline down with the request instead of giving each hop its own timeout. A service that knows the caller gave up two seconds ago can stop working on the answer.

The goal is not to never fail. It is to fail in a way that lets the system recover.