Fifteen years building and scaling distributed systems, across the industry's move from bare-metal servers and monoliths to cloud-native, Kubernetes and GitOps. I take on the work teams postpone: paying down technical debt, building deployment from nothing, and keeping systems running.
Reference architecture. Client code is under NDA, so this page describes the design and the reasoning behind it rather than a specific deployment.
Background processing for everything that should not happen inside a request: imports, exports, notifications, report generation. Handlers are idempotent because at-least-once delivery means a job will eventually run twice, and pretending otherwise just relocates the bug.
Retries use exponential backoff with a cap, and anything exhausting its retries lands in a dead-letter queue with the original payload and failure reason. The dead-letter queue has an owner and a dashboard, because an unread one is just a slower way to lose data.