The Myth of "Exactly-Once" Delivery

In distributed systems, the fallacies of distributed computing dictate that networks will partition, consumers will crash, and acknowledgments will get dropped. While message brokers (like Apache Kafka, RabbitMQ, or AWS SQS) often advertise "exactly-once processing semantics," this guarantee only exists within the closed ecosystem of the broker itself. Once an event leaves the broker and causes side effects (such as writing to a database or calling a payment gateway), you are operating strictly under at-least-once delivery.

The consensus across modern distributed architecture is straightforward: Do not design for exactly-once delivery; design for at-least-once delivery paired with idempotent consumers.

Why Naive Redis Deduplication Fails

A common first attempt at deduplication looks like this:

  1. Consumer receives eventId.
  2. Consumer checks if eventId exists in Redis.
  3. If no, execute payment and set eventId in Redis.

This approach exposes your system to two critical failure modes:

  • Race Conditions: If two consumer instances receive duplicate messages simultaneously, both can execute the "Check" step before either writes the "Set" step. Both will process the event.
  • Dual-Write & Crash Anomalies: If the consumer processes the payment and crashes before setting the key in Redis (or acknowledging the broker), the redelivered message will re-execute the payment. Conversely, setting the Redis key first means a crash during payment processing permanently drops the event without execution.

The 3 Production-Grade Deduplication Patterns

Pattern 1: The Transactional Idempotency Table (The Gold Standard)

If your primary business operation updates a relational or document database supporting ACID transactions, the database must be your single source of truth. You can achieve atomic deduplication by tying the business logic and the event consumption record into a single transaction.

CREATE TABLE processed_events (
    event_id VARCHAR(64) PRIMARY KEY,
    event_type VARCHAR(64) NOT NULL,
    processed_at TIMESTAMP WITH TIME ZONE DEFAULT NOW()
);

-- Inside a database transaction:
BEGIN;

-- Step 1: Attempt to record the event
INSERT INTO processed_events (event_id, event_type)
VALUES ('evt_12345', 'PAYMENT_COMPLETED')
ON CONFLICT (event_id) DO NOTHING;

-- Step 2: Conditionally execute the business logic
-- Check if the row was actually inserted. If duplicate, ROLLBACK / skip.
UPDATE accounts 
SET balance = balance + 1000 
WHERE user_id = 'user_42';

COMMIT;

If the consumer crashes midway, the transaction rolls back entirely, leaving the database clean for redelivery. If duplicate messages arrive concurrently, database unique constraints guarantee that only one transaction succeeds.

Pattern 2: State Machine & Natural Idempotency

Whenever possible, design your domain mutations to be naturally idempotent. Instead of processing an event as an arithmetic delta, structure it as an explicit state transition.

  • Non-idempotent: UPDATE orders SET status = 'SHIPPED', stock = stock - 1;
  • Idempotent: UPDATE orders SET status = 'SHIPPED' WHERE order_id = 'ord_123' AND status = 'PENDING';

By checking the row count of affected records, your service can detect whether the state transition has already occurred and safely acknowledge the broker.

Pattern 3: Distributed Locks with Multi-Phase Status (For External APIs)

When an event triggers an external side-effect (such as charging a card through Stripe) where database transactions cannot span across systems, combine an atomic distributed lock with a status-tracking table:

async function processPaymentEvent(event: PaymentEvent): Promise<void> {
  const { eventId, userId, amount } = event;

  // 1. Atomic Redis Lock with TTL to prevent concurrent races
  const acquired = await redis.set(`lock:${eventId}`, 'LOCKED', 'NX', 'PX', 30000);
  if (!acquired) {
    // Another worker is actively processing or recently processed this event
    return;
  }

  try {
    // 2. Check persistent DB idempotency store
    const existing = await db.query('SELECT status FROM payment_requests WHERE event_id = $1', [eventId]);
    if (existing.rows.length > 0) {
      return; // Already processed or currently processing
    }

    // 3. Mark as IN_PROGRESS
    await db.query('INSERT INTO payment_requests (event_id, status) VALUES ($1, $2)', [eventId, 'IN_PROGRESS']);

    // 4. Call External Gateway using eventId as Idempotency Key
    const chargeResult = await paymentGateway.charge({
      idempotencyKey: eventId,
      amount,
      userId
    });

    // 5. Update DB Status
    await db.query('UPDATE payment_requests SET status = $1 WHERE event_id = $2', ['COMPLETED', eventId]);
  } finally {
    // Release lock
    await redis.del(`lock:${eventId}`);
  }
}

Answering Key Architectural Questions

1. Where should deduplication live?

Deduplication must live at the persistence layer of the bounded context handling the side effect. A distributed cache like Redis can be used as an upstream, fast-rejection filter to preserve database CPU cycles, but the database's unique constraints remain the ultimate authority.

2. How to handle 10M+ events per day efficiently?

Ten million events per day equals approximately 115 events/second on average, which is well within standard relational database capacities. However, unbounded growth of an idempotency table will eventually degrade index performance. To scale:

  • Partition by Date: Partition the processed_events table by week or month and drop old partitions beyond your broker’s maximum retention window (e.g., 7 days).
  • Database TTL: Use databases with native TTL features (like MongoDB or DynamoDB) or an automated cleanup job (PostgreSQL pg_cron).
  • Partitioned Queues: Ensure events with the same partition key (e.g., userId or orderId) route to the same consumer partition to prevent concurrency conflicts on identical entities.

Summary Checklist

  • Always assume at-least-once delivery; treat exactly-once as an internal broker optimization.
  • Never rely on non-atomic "Check-Then-Act" operations across network boundaries.
  • Use database unique constraints (event_id) inside ACID transactions for state mutations.
  • Forward your event_id as an idempotency key to all downstream APIs.
  • Prune idempotency records older than the maximum retention window of your message broker.