Retries Have an Evil Twin: Duplicates

Duplication is presented as a normal outcome of retries in distributed systems, not an edge case. If the same payment, order, or message is replayed, the app can create corrupted data and repeated downstream effects unless it is built to handle at-least-once delivery.

The post compares four ways to handle duplicates at the application layer. A unique database constraint can block repeated inserts with an idempotency key. In-memory tracking is fast but only fits single-instance or short-lived processes. Redis offers shared deduplication across nodes with TTL-based cleanup, and broker-level duplicate detection can stop repeated delivery before the app sees it. The main takeaway is to combine layers where the flow is critical.

Reading notes

#

• Retries and queues can make the same work arrive more than once.
• At-least-once delivery means messages and requests may be processed multiple times.
• Duplicate processing can charge users twice, create repeated records, and trigger repeated events.
• Without duplication control, data becomes unreliable and operations spend time fixing inconsistencies.
• A database UNIQUE constraint plus an idempotency key can stop repeated inserts and return the original result.
• In-memory deduplication is fast, but it only works safely in single-instance setups without frequent restarts.
• Redis can store request IDs with TTL and coordinate deduplication across nodes.
• Broker-side duplicate detection can discard repeated messages before they reach the application.
• The post recommends using the database for strong guarantees, Redis for coordination, and brokers for message-level protection.
• TTL-based deduplication is presented as a practical starting point for many systems.