Execution queue¶
The execution scheduler in the worker owns scheduling. It keeps pending executions in Redis, one sorted set per priority, and creates a pod only while fewer than max_concurrent_executions are running. Before any of that, the API caps how many executions one user can have queued or running at once. The queue itself is ExecutionQueueService in backend/app/services/execution_queue.py; the execution scheduler decides when to call it.
flowchart LR
API[API] -->|Lua: reserve user slot| USER[(exec_queue:user:id)]
API -->|execution_requested, execution_cancelled| Kafka[(Kafka)]
Kafka --> SCHED[Execution scheduler]
SCHED -->|ZADD| PENDING[(exec_queue:pending:priority
5 sorted sets)]
SCHED -->|Lua: prune, ZCARD, ZPOPMIN, ZADD| RUNNING[(exec_queue:running)]
SCHED -->|create pod| K8S[K8s worker]
MON[Pod monitor] -->|pod finished| SCHED
SCHED -->|ZREM| RUNNING
SCHED -->|ZREM| USER
Redis keys¶
| Key | Type | Holds | TTL |
|---|---|---|---|
exec_queue:pending:{priority} |
Sorted set | Pending ids for one priority, scored by enqueue time (epoch seconds) | none |
exec_queue:running |
Sorted set | Ids that hold a running slot, scored by the time the slot expires | none |
exec_queue:user:{user_id} |
Sorted set | One user's queued and running ids, scored by the time the reservation expires | 24 hours, refreshed on each reservation |
exec_queue:owner:{execution_id} |
String | The user an id was reserved for, so a release can find the user's set | 24 hours |
exec_queue:event:{execution_id} |
String | The ExecutionRequestedEvent as JSON |
24 hours |
There are five pending sets: critical, high, normal, low and background. A higher priority always goes first, and within one priority the oldest entry goes first.
Every execution submitted through POST /api/v1/execute starts as normal. Only an admin can change that, after the fact.
Per-user limit¶
ExecutionService.execute_script reserves a slot for the user before it writes the execution. The reservation is a Lua script on exec_queue:user:{user_id}: it drops entries whose time has passed, and if the user still holds max_executions_per_user (admin setting, default 100, 1 to 1000) it refuses, and the API answers 429 with ExecutionLimitExceededError. Otherwise it adds the new id, which is generated before the insert for this reason. Retries go through the same path. If the insert or the publish fails, the service gives the slot back.
A reservation lasts 24 hours, as long as a queued event does. The scheduler gives it back when the execution ends, a cancel or a pod that couldn't be created included. A release that never happens only holds the slot until it expires.
Scheduling¶
try_schedule is one Lua script that gets the running set and the five pending sets as keys. It first drops running entries whose expiry has passed. If ZCARD on the running set is then at the cap, it returns nothing. Otherwise it walks the pending sets from critical to background, pops the first entry it finds with ZPOPMIN, adds the id to the running set with its expiry, and returns the id, its score and the priority index. The count check and the pop happen in one round trip, so two callers can't both squeeze past the cap.
Once the service has loaded the event, the slot gets the execution's own pod deadline plus 300 seconds: the timeout the execution was submitted with, raised to its language's minimum. The user's reservation moves to the same deadline, so a job that waited long in the queue keeps its place in the user's count while it runs. Until the event is loaded, the slot assumes the longest timeout an admin can set. A normal run releases the slot long before it expires, so expiry only matters when a release got lost, for example a pod deleted by hand while the worker was down.
The service then loads the stored event. If that key already expired, the request can't run: the id leaves the running set and its user's reservation, queue.event_data_lost goes up, and the scheduler has the result processor store the execution as failed with error type system_error, then moves on to the next one.
schedule_pending reads max_concurrent_executions from the admin system settings (default 10, admins can set 1 to 100, cached for 60 seconds by RuntimeSettingsLoader). It calls try_schedule in a loop and has the k8s worker create the pod for each id it gets, until the queue is empty, the cap is reached or it pops an id whose event expired. That last case ends the pass even with entries still pending, and the next pass picks them up.
If the pod can't be created, the scheduler stores the execution as failed with a system_error, releases both slots and goes on with the next id.
These trigger a pass:
| Trigger | Queue action before scheduling |
|---|---|
execution_requested |
enqueue (guarded by idempotency) |
| the pod monitor sees a pod finish | release, release_user_slot |
execution_cancelled |
remove from every queue key, store cancelled, delete the pod, release, release_user_slot |
release is a ZREM on the running set plus deleting the event key, and release_user_slot looks up the owner and removes the id from that user's set, so both are safe on an id that holds nothing. On top of those, the worker runs schedule_pending every 10 seconds through APScheduler (app/worker.py), which catches up after a restart or a slot that expired.
Admin endpoints¶
All three need an admin user.
| Endpoint | What it does |
|---|---|
GET /api/v1/admin/executions/ |
Lists executions, filtered by status, priority or user |
PUT /api/v1/admin/executions/{execution_id}/priority |
Sets a new priority |
GET /api/v1/admin/executions/queue |
Returns queue_depth, active_count, max_concurrent and by_priority |
A priority change updates MongoDB first, then runs a second Lua script that finds the id in whichever pending set holds it and moves it to the new set with the same score. The entry keeps its age, so it lands among the new priority's entries by enqueue time. An execution that already left the queue only gets the MongoDB update.
The queue status reads Redis directly: queue_depth is the sum of ZCARD over the five pending sets, active_count counts the running entries that haven't expired, and by_priority has the ZCARD of each non-empty set.
Metrics¶
The queue's six series come from spans: queue.enqueue and queue.schedule from ExecutionQueueService, and queue.schedule_pending from every pass of the execution scheduler, which reads the depth and the running slots from Redis. So queue_depth and queue_active match the admin queue endpoint and survive restarts. The metrics reference lists the series and their labels.