Event storage¶
Every event the system produces is written to one MongoDB collection, events, before it goes to Kafka. Each document is an EventDocument. The base event fields sit at the top level, and so do the event-specific ones: there's no payload wrapper.
EventDocument¶
class EventDocument(Document):
"""Event document for event browsing/admin system.
Uses extra='allow' for flexible event data storage - event-specific fields
are stored directly at document level (no payload wrapper needed).
"""
event_id: Annotated[str, Indexed(unique=True)] = Field(default_factory=lambda: str(uuid4()))
event_type: EventType # Indexed via Settings.indexes
event_version: str = "1.0"
timestamp: Annotated[datetime, Indexed()] = Field(default_factory=lambda: datetime.now(timezone.utc))
aggregate_id: str | None = None
metadata: EventMetadata
stored_at: datetime = Field(default_factory=lambda: datetime.now(timezone.utc))
ttl_expires_at: datetime = Field(default_factory=lambda: datetime.now(timezone.utc) + timedelta(days=30))
# Most event types have execution_id (sparse-indexed)
execution_id: str | None = None
model_config = ConfigDict(from_attributes=True, extra="allow")
class Settings:
name = "events"
use_state_management = True
bson_encoders = {EventMetadata: dataclasses.asdict}
indexes = [
# Compound indexes for common query patterns
IndexModel([("event_type", ASCENDING), ("timestamp", DESCENDING)], name="idx_event_type_ts"),
IndexModel([("aggregate_id", ASCENDING), ("timestamp", DESCENDING)], name="idx_aggregate_ts"),
IndexModel([("metadata.user_id", ASCENDING), ("timestamp", DESCENDING)], name="idx_meta_user_ts"),
IndexModel([("metadata.service_name", ASCENDING), ("timestamp", DESCENDING)], name="idx_meta_service_ts"),
# Event-specific field indexes (sparse - only exist on relevant event types)
IndexModel([("execution_id", ASCENDING)], name="idx_execution_id", sparse=True),
IndexModel([("pod_name", ASCENDING)], name="idx_pod_name", sparse=True),
# TTL index (expireAfterSeconds=0 means use ttl_expires_at value directly)
IndexModel([("ttl_expires_at", ASCENDING)], name="idx_ttl", expireAfterSeconds=0),
# Additional compound indexes for query optimization
IndexModel([("event_type", ASCENDING), ("aggregate_id", ASCENDING)], name="idx_events_type_agg"),
IndexModel([("aggregate_id", ASCENDING), ("timestamp", ASCENDING)], name="idx_events_agg_ts"),
IndexModel([("event_type", ASCENDING), ("timestamp", ASCENDING)], name="idx_events_type_ts_asc"),
IndexModel([("metadata.user_id", ASCENDING), ("timestamp", ASCENDING)], name="idx_events_user_ts"),
IndexModel([("metadata.user_id", ASCENDING), ("event_type", ASCENDING)], name="idx_events_user_type"),
IndexModel(
[("event_type", ASCENDING), ("metadata.user_id", ASCENDING), ("timestamp", DESCENDING)],
name="idx_events_type_user_ts",
),
# Text search index
IndexModel(
[
("event_type", pymongo.TEXT),
("metadata.service_name", pymongo.TEXT),
("metadata.user_id", pymongo.TEXT),
("execution_id", pymongo.TEXT),
],
name="idx_text_search",
language_override="none",
default_language="english",
),
]
The class declares the fields every event shares, plus stored_at, ttl_expires_at and execution_id. Everything else comes in through extra="allow". A PodCreatedEvent stores pod_name and namespace next to event_type, a UserSettingsUpdatedEvent stores whichever settings changed. Only some event types have execution_id or pod_name, so both indexes are declared sparse. Only the pod_name one stays sparse in practice: execution_id is a declared field, so an event without one is stored with execution_id: null, and a sparse index still indexes nulls.
Queries hit event fields directly, with no prefix:
Write path¶
graph LR
Svc[Service] --> UP[UnifiedProducer.produce]
UP -->|1. store_event| Events[(events)]
UP -->|2. publish| Transport[KafkaEventTransport]
Transport --> Kafka[(Kafka topic = event_type)]
UnifiedProducer.produce() in app/events/core/producer.py does both steps. First EventRepository.store_event() dumps the event with model_dump(exclude_none=True), sets stored_at and inserts it. A duplicate event_id hits the unique index, and the repository logs it and returns, so storing the same event twice is a no-op. Replayed events keep their event_id, so a replay doesn't create new rows.
Then KafkaEventTransport.publish() sends the event to the topic named after its event_type. If that fails, the stored event stays as it is and the error goes back to the caller, or into the log for produce_or_log. Nothing re-sends it automatically, though a replay session with the Kafka target sends stored events again.
Every service publishes through UnifiedProducer.produce, which also opens a trace span and records the publish metrics.
Read path¶
Reads turn documents back into typed events with DomainEventAdapter.validate_python(doc). The adapter is a TypeAdapter over the DomainEvent union, discriminated by event_type, so each row comes back as its concrete class. BaseEvent has from_attributes=True, which lets it read the extra fields straight off the document.
Several repositories read the same collection:
| Repository | Reads events for |
|---|---|
EventRepository |
Execution event lists, user event history, statistics, aggregate replay info |
AdminEventsRepository |
Admin browse, detail, stats, export, replay preview |
ReplayRepository |
Replay sessions, and streaming the events they send |
UserSettingsRepository |
Rebuilding settings from user_settings_updated events |
Filters¶
Two filter dataclasses build the queries. Admin browse and export use EventFilter from app/domain/events/event_models.py. AdminEventsRepository._event_filter_conditions() turns it into Beanie conditions, and its search_text runs against the idx_text_search index.
Replay uses ReplayFilter, which builds a raw Mongo query itself:
@dataclass
class ReplayFilter:
# Event selection filters
event_ids: list[str] | None = None
execution_id: str | None = None
aggregate_id: str | None = None
event_types: list[EventType] | None = None
exclude_event_types: list[EventType] | None = None
# Time range
start_time: datetime | None = None
end_time: datetime | None = None
# Metadata filters
user_id: str | None = None
service_name: str | None = None
def is_empty(self) -> bool:
return not any(
[
self.event_ids,
self.execution_id,
self.aggregate_id,
self.event_types,
self.start_time,
self.end_time,
self.user_id,
self.service_name,
]
)
def to_mongo_query(self) -> dict[str, Any]:
query: dict[str, Any] = {}
if self.event_ids:
query["event_id"] = {"$in": self.event_ids}
if self.execution_id:
query["execution_id"] = self.execution_id
if self.aggregate_id:
query["aggregate_id"] = self.aggregate_id
if self.event_types:
query["event_type"] = {"$in": list(self.event_types)}
if self.exclude_event_types:
query.setdefault("event_type", {})["$nin"] = list(self.exclude_event_types)
if self.start_time or self.end_time:
time_query: dict[str, Any] = {}
if self.start_time:
time_query["$gte"] = self.start_time
if self.end_time:
time_query["$lte"] = self.end_time
query["timestamp"] = time_query
if self.user_id:
query["metadata.user_id"] = self.user_id
if self.service_name:
query["metadata.service_name"] = self.service_name
return query
ReplayRepository uses to_mongo_query() to count and stream events for a replay session. AdminEventsRepository uses it for the admin replay endpoint: the dry-run preview, the event count and the related execution results.
Retention¶
Each event expires 30 days after it's stored. ttl_expires_at defaults to now plus 30 days, and the idx_ttl index with expireAfterSeconds=0 lets MongoDB delete the document at that time. The 30 days are hardcoded in EventDocument.
TTL expiry doesn't archive anything. Archiving only happens when an admin deletes an event through DELETE /api/v1/admin/events/{event_id}: AdminEventsService.delete_event() copies it into events_archive (EventArchiveDocument, same fields plus deleted_at, deleted_by and deletion_reason) and then deletes the original. The archive collection has no TTL index.
Related: User Settings Events for an event-sourced read model built on this collection, and Model Conversion for how the other repositories convert documents.