Idempotency¶
Kafka delivers at least once, so a consumer can see the same event twice. Handlers that must not run twice go through IdempotencyManager, which keeps one Redis key per event and lets only the first caller through. The API reuses the same manager for the Idempotency-Key header on POST /api/v1/execute.
Who uses it¶
| Caller | Events | Key strategy | TTL |
|---|---|---|---|
| Execution scheduler | execution_requested, execution_cancelled |
EVENT_BASED |
1 hour |
API, POST /api/v1/execute with Idempotency-Key |
none, see below | CUSTOM |
24 hours |
The other subscribers, sse-bridge-pool and notification-service, don't use it. The subscriptions live in backend/app/events/handlers.py, one KafkaRouter per consumer, and the guard is a middleware on the router, so it covers every subscriber on it.
Keys aren't scoped by consumer group, so two routers with the same strategy on the same event would skip each other's deliveries. Only one router uses the manager today.
Inside the worker, a finished pod doesn't need a key: the result processor's write only applies to an execution that is still active, so a pod reported twice stores its result once and publishes it once.
Keys¶
The manager builds a key from the event in one of three ways:
def _generate_key(
self, event: BaseEvent, key_strategy: KeyStrategy, custom_key: str | None = None, fields: set[str] | None = None
) -> str:
if key_strategy == KeyStrategy.EVENT_BASED:
key = f"{event.event_type}:{event.event_id}"
elif key_strategy == KeyStrategy.CONTENT_HASH:
event_dict = event.model_dump(mode="json")
event_dict.pop("event_id", None)
event_dict.pop("timestamp", None)
event_dict.pop("metadata", None)
if fields:
event_dict = {k: v for k, v in event_dict.items() if k in fields}
content = json.dumps(event_dict, sort_keys=True)
key = hashlib.sha256(content.encode()).hexdigest()
elif key_strategy == KeyStrategy.CUSTOM and custom_key:
key = f"{event.event_type}:{custom_key}"
else:
raise ValueError(f"Invalid key strategy: {key_strategy}")
return f"{self.config.key_prefix}:{key}"
EVENT_BASED is {event_type}:{event_id} and is the default when a caller passes no strategy. CONTENT_HASH is a SHA-256 of the event JSON with event_id, timestamp and metadata removed, so an event published twice under two different ids still counts as one; no router uses it today. CUSTOM is {event_type}:{custom_key} with a key the caller picks. Every key gets the idempotency: prefix.
How a check works¶
flowchart TB
IN[Event] --> GET{Key in Redis?}
GET -->|No| NX[SET NX EX: processing]
NX -->|Won| RUN[Run handler]
NX -->|Lost the race| GET
GET -->|completed or failed| DUP[Duplicate, skip]
GET -->|processing| AGE{Older than 300 s?}
AGE -->|No| DUP
AGE -->|Yes| RUN
RUN -->|OK| DONE[Mark completed]
RUN -->|Raises| FAIL[Release key, re-raise]
A record has one of three states:
class IdempotencyStatus(StringEnum):
PROCESSING = "processing"
COMPLETED = "completed"
FAILED = "failed"
check_and_reserve reads the key first. If it's missing, the manager writes a processing record with SET NX EX, so of two racing consumers only one wins. The loser reads the key again and gets treated as a duplicate.
async def insert_processing(self, record: IdempotencyRecord) -> None:
k = self._full_key(record.key)
doc = self._record_to_doc(record)
# SET NX with EX for atomic reservation
ok = await self._r.set(k, json.dumps(doc, default=_json_default), ex=record.ttl_seconds, nx=True)
if not ok:
# Mirror Mongo behavior so manager's DuplicateKeyError path is reused
raise DuplicateKeyError("Key already exists")
When SET NX loses, the repository raises pymongo's DuplicateKeyError, odd for a Redis store but it's what the manager catches. Each record is a JSON document with the status, the event type and id, timestamps, processing duration, the error and an optional cached result. An update is one SET XX KEEPTTL: it overwrites only a key that still exists and keeps its remaining lifetime, so a rewrite never brings back an expired key or drops an expiry.
A processing record older than processing_timeout_seconds (300) is taken to be a crashed consumer. The manager resets its created_at and lets the new caller run. completed and failed records both count as duplicates until the key expires. Neither the Kafka middleware nor the HTTP path writes failed today: both release the key when the work raises.
Kafka handlers get the guard from IdempotencyMiddleware, which each router takes with its key strategy and TTL:
class IdempotencyMiddleware(BaseMiddleware):
"""Runs a handler at most once per idempotency key while the key lives in Redis.
The key is reserved before the handler runs, then marked completed when it returns
or released when it raises, so a retry from the DLQ can run the handler again. A
delivery whose key is already taken is skipped, so Kafka commits its offset without
running the handler again.
"""
def __init__(self, msg: Any, /, *, key_strategy: KeyStrategy, ttl_seconds: int, **kwargs: Any) -> None:
super().__init__(msg, **kwargs)
self.key_strategy = key_strategy
self.ttl_seconds = ttl_seconds
async def consume_scope(
self,
call_next: Callable[[StreamMessage[Any]], Awaitable[Any]],
msg: StreamMessage[Any],
) -> Any:
# The broker-level Dishka middleware wraps this one and opens the request container
container: AsyncContainer = self.context.get_local("dishka")
idem = await container.get(IdempotencyManager)
event = DomainEventAdapter.validate_python(await msg.decode())
reservation = await idem.check_and_reserve(
event=event, key_strategy=self.key_strategy, ttl_seconds=self.ttl_seconds
)
if reservation.is_duplicate:
logger = await container.get(structlog.stdlib.BoundLogger)
logger.info("Duplicate event", event_type=event.event_type, event_id=event.event_id)
return None
try:
result = await call_next(msg)
except Exception:
await idem.release(event=event, key_strategy=self.key_strategy)
raise
await idem.mark_completed(event=event, key_strategy=self.key_strategy)
return result
It runs before FastStream hands the message to the handler, so it parses the raw body into the DomainEvent union itself and takes the manager from the request container that the broker's Dishka middleware opens around it.
A duplicate is logged and skipped, and the message is acked as usual. A handler that raises gets its key released, so nothing is left to block the event. The router's DeadLetterMiddleware saves it to the dead letter queue, and the retry runs the handler again.
HTTP Idempotency-Key¶
POST /api/v1/execute takes an optional Idempotency-Key header. Without it the request runs as usual. With it, ExecutionService.execute_script_idempotent reserves the custom key http:{user_id}:{key} for 24 hours, so two users can reuse the same header value without clashing. The Redis key ends up as idempotency:execution_requested:http:{user_id}:{key}.
On success the manager stores the new execution as JSON in the record. A repeat with the same key gets one of these:
| Repeat request arrives | Response |
|---|---|
| After the first one succeeded | 200 with the cached execution, same execution_id, nothing new runs |
| While the first one is still running | 409, "Duplicate request ' |
| After the first one failed | Runs again, since a failed request deletes its key |
| After the cached result went missing | 409, "Cached result for ' |
A client that hits an error can retry with the same key.
Configuration¶
class IdempotencyConfig(BaseModel):
key_prefix: str = "idempotency"
default_ttl_seconds: int = 3600
processing_timeout_seconds: int = 300
enable_result_caching: bool = True
max_result_size_bytes: int = 1048576
The DI provider builds IdempotencyConfig() with these defaults; nothing in config.toml changes them. Every caller passes its own TTL, so default_ttl_seconds only applies to a caller that doesn't. Result caching is used by the HTTP path only. Event handlers mark keys completed without a result, and a result over max_result_size_bytes is logged and not stored.
Metrics¶
IdempotencyManager puts each check_and_reserve on an idempotency.check span with a hit or miss, and leaves idempotency.duplicate_blocked and idempotency.processed spans for blocked duplicates and finished keys. The worker counts the idempotency:* keys in Redis with SCAN every 30 seconds for idempotency_keys_active. A miss is a lookup that found no record and went on to SET NX, whether or not it won it. The metrics reference lists the series and their labels.