Notifications¶
NotificationService turns execution results into in-app notifications. It stores each one in MongoDB and pushes it to the user's open notification streams over SSE. It lives in the backend API process, not in the worker.
flowchart LR
Kafka[(Kafka)] --> NS[NotificationService]
NS --> DB[(MongoDB)]
NS --> SSE[SSE bus] --> Browser
Where they come from¶
dishka_lifespan includes notification_router, which subscribes to execution_completed, execution_failed and execution_timeout in the consumer group notification-service, starting from the latest offset. Each handler turns the event into a notification for the execution's owner, unless the owner switched it off in their settings: execution_completed covers completed runs, and execution_failed covers failed and timed-out ones.
| Event | Subject | Severity | Tags |
|---|---|---|---|
execution_completed |
Execution Completed: <id> |
medium | execution, completed, entity:execution, exec:<id> |
execution_failed |
Execution Failed: <id> |
high | execution, failed, entity:execution, exec:<id> |
execution_timeout |
Execution Timeout: <id> |
high | execution, timeout, entity:execution, exec:<id> |
The tag pattern is the convention for any producer: a category, an outcome, an entity: tag and a reference such as exec:<id>. The UI filters and links on them.
Fields and statuses¶
| Field | Values |
|---|---|
severity |
low, medium, high, urgent |
status |
pending, queued, sending, delivered, failed, read, clicked |
Besides those, a notification has subject, body, action_url, tags and a metadata dict.
A new notification is pending. Delivery claims it atomically by moving it to sending, then it ends as delivered or failed. Marking it read sets read. queued and clicked exist in the enum, but no code sets them today.
Delivery¶
A notification is delivered right after it's stored. The service publishes it to the SSE bus on sse:notif:{user_id}, and every open notification stream of that user gets it. A failed publish is retried with exponential backoff, up to max_retries (3) attempts and at most 30 seconds apart, before the notification is marked failed. It stays in the user's list either way, since the list reads MongoDB.
Throttling¶
Creation is throttled per user and severity, NOTIF_THROTTLE_MAX_PER_HOUR (5) per NOTIF_THROTTLE_WINDOW_HOURS (1). Past the limit create_notification raises NotificationThrottledError. That check is off when ENVIRONMENT is test.
@dataclass
class ThrottleCache:
"""Manages notification throttling with time windows."""
_entries: dict[str, list[datetime]] = field(default_factory=dict)
_lock: asyncio.Lock = field(default_factory=asyncio.Lock)
async def check_throttle(
self,
user_id: str,
severity: NotificationSeverity,
window_hours: int,
max_per_hour: int,
) -> bool:
"""Check if notification should be throttled."""
key = f"{user_id}:{severity}"
now = datetime.now(UTC)
window_start = now - timedelta(hours=window_hours)
async with self._lock:
if key not in self._entries:
self._entries[key] = []
# Clean old entries
self._entries[key] = [ts for ts in self._entries[key] if ts > window_start]
# Check limit
if len(self._entries[key]) >= max_per_hour:
return True
# Add new entry
self._entries[key].append(now)
return False
async def clear(self) -> None:
"""Clear all throttle entries."""
async with self._lock:
self._entries.clear()
The cache lives in memory, so each gunicorn worker counts on its own and a restart resets it.