Skip to content

Dead letter queue

When a Kafka handler raises, its event goes to the dead letter queue, and the worker's DLQ job later sends it back to the one consumer group that failed. The queue is the dlq_messages MongoDB collection, not a Kafka topic. FastStream has no dead letter queue for Kafka, so two small middlewares in app/events/middlewares.py connect the subscribers to it.

What happens to a failed event

Every subscriber except the SSE bridge uses AckPolicy.ACK, which commits the offset after the handler returns, whether it succeeded or raised. The execution scheduler and notification routers also carry DeadLetterMiddleware:

class DeadLetterMiddleware(BaseMiddleware):
    """Saves an event whose handler raised to the DLQ, for the dlq-processor to retry in this consumer group.

    The error still propagates, so FastStream logs it and the telemetry middleware counts it.
    """

    async def consume_scope(
        self,
        call_next: Callable[[StreamMessage[Any]], Awaitable[Any]],
        msg: StreamMessage[Any],
    ) -> Any:
        try:
            return await call_next(msg)
        except Exception as e:
            container: AsyncContainer = self.context.get_local("dishka")
            manager = await container.get(DLQManager)
            event = DomainEventAdapter.validate_python(await msg.decode())
            await manager.handle_message(
                DLQMessageDocument(
                    event=event,
                    original_topic=msg.raw_message.topic,
                    consumer_group=self.context.get_local("handler_").group_id,
                    error=str(e),
                    retry_count=int(msg.headers.get(DLQ_RETRY_HEADER, 0)),
                    producer_id=event.metadata.service_name,
                )
            )
            raise


class RetryTargetMiddleware(BaseMiddleware):
    """Skips a DLQ retry that's addressed to another consumer group."""

    async def consume_scope(
        self,
        call_next: Callable[[StreamMessage[Any]], Awaitable[Any]],
        msg: StreamMessage[Any],
    ) -> Any:
        target = msg.headers.get(DLQ_GROUP_HEADER)
        if target and target != self.context.get_local("handler_").group_id:
            return None
        return await call_next(msg)

When the handler raises, the middleware passes the event to DLQManager.handle_message with its topic, the subscriber's consumer group, the error and the retry count, then re-raises. FastStream logs the error, the telemetry middleware counts it in messaging_process_duration_seconds under error_type, and the offset is committed. Where the router also has an idempotency guard, the guard releases the event's key on the way out, so the retry isn't turned away as a duplicate.

handle_message drops the message if it's older than 7 days or its event id starts with test- (outside the test environment). Otherwise it saves the row as pending, publishes dlq_message_received, and then either discards the message, when its retries are used up, or schedules the next retry by the event type's policy.

The SSE bridge has no DLQ middleware. It acks before it runs (ACK_FIRST), and pushing a stale status to a browser minutes later would do more harm than missing it.

Retries

The DLQ processor runs every 10 seconds. It picks up rows with status scheduled whose next_retry_at has passed, marks each one retried with its retry count plus one, and republishes the event to its original topic.

Every consumer group on that topic receives the retried event, so the publish carries two headers: x-dlq-consumer-group names the group that failed and x-dlq-retry-count the attempt. RetryTargetMiddleware, which every broker has, makes the other groups skip it. If the retry fails again, DeadLetterMiddleware saves it with that retry count, so the same row goes back to pending and gets scheduled or discarded. A retry that succeeds leaves the row at retried. If the republish itself fails, the row goes back to scheduled and the next cycle tries again.

The row is marked before the publish on purpose. A retry can fail and save its new failure within milliseconds, and marking afterwards would overwrite that with retried.

One retry policy, RETRY_POLICY in app/dlq/models.py, covers every row, since only consumers of execution events save to the DLQ. The wait starts at 30 seconds and doubles with each failed attempt, up to 300 seconds, with 10% jitter. A message whose retry_count has reached 5 is discarded with reason max_retries_exceeded, and dlq_message_discarded goes out.

Storage

A row is one event in one consumer group. The unique index is on event.event_id plus consumer_group, so when two groups fail on the same event, each gets its own row and its own retry. A TTL index deletes rows 7 days after created_at.

A database created before consumer groups were added still has the old unique index idx_dlq_event_id on event.event_id alone, and Beanie never drops indexes. Until someone runs db.dlq_messages.dropIndex("idx_dlq_event_id") once, a second group that fails on the same event can't save its row.

The retry policy and the 7 day TTL are set in code, and config.toml has no DLQ keys. There's no HTTP API for the DLQ.

Key files

File Purpose
middlewares.py DeadLetterMiddleware and RetryTargetMiddleware
manager.py Intake, retry, discard
models.py Statuses, the retry policy and the two headers
dlq.py dlq_messages document and indexes
worker.py Worker entry point, schedules the retry job