DLQ processor¶
The DLQ processor has no Kafka subscriber. It retries the failed events that the consumer routers, in the worker and the API, save to the dlq_messages collection. Dead letter queue covers how they get there.
graph LR
Scheduler[APScheduler, every 10 s] --> Manager[DLQManager]
Manager -->|status scheduled, next_retry_at due| Mongo[(dlq_messages)]
Manager -->|republish| Original[(original topic)]
Manager -->|dlq_message_retried| Kafka[(Kafka)]
Every 10 seconds process_monitoring_cycle does two things. First it loads up to 100 rows with status scheduled whose next_retry_at has passed. For each one it sets the row to retried with retry_count plus one, republishes the event to its original topic keyed by event id, with headers naming the consumer group that failed and the retry count, and publishes dlq_message_retried. Then it counts the pending and scheduled rows per topic and reports each count on a dlq.queue_size span, which the collector keeps as the dlq_queue_size gauge. A retry that fails again comes back as a new failure for the same row.
The retry policy is set in app/dlq/models.py. There's no admin API for the DLQ.
Key files¶
| File | Purpose |
|---|---|
worker.py |
Entry point and APScheduler job |
manager.py |
process_due_retries, retry_message |