Every dependency call needs a time budget

Resilience starts by accepting that dependencies will be slow, unavailable, or ambiguous. A timeout bounds how long you wait; a retry decides whether another attempt is safe; backoff prevents synchronized pressure; a circuit breaker stops sending work to a dependency that is already failing. They must be designed together with idempotency and an end-to-end latency budget.

Timeouts, retries, backoff, and circuit breaking are different tools

  • Set connect and request timeouts below the caller’s total deadline.
  • Retry only errors likely to be transient and only operations known to be retry-safe.
  • Use exponential backoff with jitter rather than immediate fixed retries.
  • Limit total attempts and concurrent retry volume with a retry budget.
  • Circuit breakers protect resources but require fallback or explicit unavailable behavior.

A bounded retry lifecycle

The caller checks the deadline and circuit, attempts the dependency, classifies the failure, waits with jitter only when safe, and stops at a bounded budget.

Diagram

Bounded retry with backoff and circuit state

The caller checks the deadline and circuit, attempts the dependency, classifies the failure, waits with jitter only when safe, and stops at a bounded budget.

Making a flaky provider survivable

Bound the attempt by a caller-owned deadline

A small retry policy should be explicit about attempts, timeout, and which errors qualify.

resilient-call.tstypescript
const policy = { attempts: 2, timeoutMs: 600 };

for (let attempt = 1; attempt <= policy.attempts; attempt++) {
  try {
    return await callProvider({ timeoutMs: policy.timeoutMs });
  } catch (error) {
    if (!isTransient(error) || attempt === policy.attempts) throw error;
    await sleep(withJitter(100 * 2 ** (attempt - 1)));
  }
}

Retry storms and other resilience failures

Resilience checklist

  • Write the caller deadline before dependency timeouts.
  • Classify retryable errors explicitly.
  • Make retried mutations idempotent.
  • Use jitter and a total retry budget.
  • Observe timeout rate, retry count, circuit state, and dependency latency.