K8s worker¶
The k8s worker turns an execution into Kubernetes objects. The execution scheduler calls it once a queued execution gets a slot, and again to delete the pod of a cancelled one. It consumes nothing from Kafka.
graph LR
Sched[Execution scheduler] -->|create_pod / delete_pod| Worker[K8s worker]
Worker -->|ConfigMap + Pod| K8s[Kubernetes API]
Worker -->|pod_created| Kafka[(Kafka)]
Creating a pod¶
For a new execution the worker creates a ConfigMap script-{execution_id} holding the user's script and entrypoint.sh, then a pod executor-{execution_id}. Once the pod exists, it patches the ConfigMap with an ownerReference to the pod, so Kubernetes deletes the ConfigMap together with the pod. If the Kubernetes API rejects that patch, the worker only logs a warning and the ConfigMap outlives the pod. Then it publishes pod_created; a Kafka error there is only logged, since the pod already exists. If anything else in that sequence fails, the error goes back to the scheduler, which stores the execution as failed with error type system_error and exit code -1. A pod that already exists (409) counts as created, so a retried request doesn't make a second one.
The image and command come from the execution_requested event, which the API filled in from the runtime registry (app/runtime_registry.py). The registry can also give a language its own pod settings: extra env vars, a bigger /tmp, and minimum CPU, memory and timeout. The pod builder takes the larger of the configured value and the language minimum. Only Go uses this today:
LANGUAGE_SPECS: dict[str, LanguageSpec] = {
"python": {
"versions": ["3.14", "3.13", "3.12", "3.11"],
"image_tpl": "python:{version}-slim",
"file_ext": "py",
"interpreter": ["python"],
},
"node": {
"versions": ["26", "24", "22"],
"image_tpl": "node:{version}-alpine",
"file_ext": "js",
"interpreter": ["node"],
},
"ruby": {
"versions": ["4.0", "3.4", "3.3"],
"image_tpl": "ruby:{version}-alpine",
"file_ext": "rb",
"interpreter": ["ruby"],
},
"bash": {
"versions": ["5.3"],
"image_tpl": "bash:{version}",
"file_ext": "sh",
"interpreter": ["bash"],
},
"go": {
"versions": ["1.27", "1.26"],
"image_tpl": "golang:{version}-alpine",
"file_ext": "go",
# exec keeps the PID, so the entrypoint measures the program instead of the go tool that built it
"interpreter": [
"sh",
"-c",
'go build -o /tmp/main -ldflags="-s -w" -gcflags=all=-dwarf=false "$0" && exec /tmp/main',
],
# A cold build compiles the runtime and std into GOCACHE: about 50Mi of /tmp, 230Mi RSS, 10s of CPU.
"pod": PodRuntimeConfig(
env={"HOME": "/tmp", "GOCACHE": "/tmp/go-cache"}, # NOSONAR # nosec B108
tmp_size="64Mi",
min_cpu="1000m",
min_memory="320Mi",
min_timeout_seconds=30,
),
},
}
CPU and memory limits and the timeout arrive in the event, taken from the admin system settings when the execution was requested. Requests come from K8S_POD_CPU_REQUEST and K8S_POD_MEMORY_REQUEST. The timeout becomes the pod's activeDeadlineSeconds.
For a cancelled execution the worker deletes the pod with no grace period and lets the ownerReference take the ConfigMap with it. A queued execution has no pod yet, so a 404 there is expected.
Pod security¶
| Control | Setting |
|---|---|
| User | runAsNonRoot, UID and GID 1000, hostUsers: false (user namespace) |
| Filesystem | Read-only root; writable /output (10Mi) and /tmp (10Mi, 64Mi for Go) emptyDirs |
| Privileges | allowPrivilegeEscalation: false, all capabilities dropped |
| Seccomp | RuntimeDefault |
| API access | automountServiceAccountToken: false, enableServiceLinks: false |
| Host namespaces | No host network, PID or IPC |
| Runtime class | K8S_POD_RUNTIME_CLASS_NAME if set (for example gVisor) |
The worker refuses to start if K8S_NAMESPACE is default.
Startup¶
Before the worker consumes anything, it applies two things to the executor namespace. The first is the executor-deny-all NetworkPolicy, which selects pods labeled component=executor and allows no ingress and no egress, DNS included. The second is the Pod Security Admission labels for the restricted profile (enforce, warn and audit). The worker is the only thing that writes this policy, and it takes effect only if the cluster's network plugin enforces NetworkPolicy.
It also applies the runtime-image-pre-puller DaemonSet, which pulls every image in the runtime registry on each node. The worker doesn't wait for those pulls, so an execution that arrives before they finish still waits for its image.
Every executor pod counts against the executor-quota ResourceQuota that the worker applies to its namespace on start (K8S_NAMESPACE_CPU_QUOTA and K8S_NAMESPACE_MEMORY_QUOTA, 32 CPU and 4Gi of requests). A pod over the quota is refused with 403 Forbidden, and the execution is stored as failed.
Key files¶
| File | Purpose |
|---|---|
worker.py |
Entry point, startup setup |
worker.py |
Pod creation and deletion, namespace policy, pre-puller |
pod_builder.py |
ConfigMap and pod spec |
runtime_registry.py |
Languages, images, per-language pod settings |