Skip to content

K8s worker

The k8s worker turns an execution into Kubernetes objects. The execution scheduler calls it once a queued execution gets a slot, and again to delete the pod of a cancelled one. It consumes nothing from Kafka.

graph LR
    Sched[Execution scheduler] -->|create_pod / delete_pod| Worker[K8s worker]
    Worker -->|ConfigMap + Pod| K8s[Kubernetes API]
    Worker -->|pod_created| Kafka[(Kafka)]

Creating a pod

For a new execution the worker creates a ConfigMap script-{execution_id} holding the user's script and entrypoint.sh, then a pod executor-{execution_id}. Once the pod exists, it patches the ConfigMap with an ownerReference to the pod, so Kubernetes deletes the ConfigMap together with the pod. If the Kubernetes API rejects that patch, the worker only logs a warning and the ConfigMap outlives the pod. Then it publishes pod_created; a Kafka error there is only logged, since the pod already exists. If anything else in that sequence fails, the error goes back to the scheduler, which stores the execution as failed with error type system_error and exit code -1. A pod that already exists (409) counts as created, so a retried request doesn't make a second one.

The image and command come from the execution_requested event, which the API filled in from the runtime registry (app/runtime_registry.py). The registry can also give a language its own pod settings: extra env vars, a bigger /tmp, and minimum CPU, memory and timeout. The pod builder takes the larger of the configured value and the language minimum. Only Go uses this today:

LANGUAGE_SPECS: dict[str, LanguageSpec] = {
    "python": {
        "versions": ["3.14", "3.13", "3.12", "3.11"],
        "image_tpl": "python:{version}-slim",
        "file_ext": "py",
        "interpreter": ["python"],
    },
    "node": {
        "versions": ["26", "24", "22"],
        "image_tpl": "node:{version}-alpine",
        "file_ext": "js",
        "interpreter": ["node"],
    },
    "ruby": {
        "versions": ["4.0", "3.4", "3.3"],
        "image_tpl": "ruby:{version}-alpine",
        "file_ext": "rb",
        "interpreter": ["ruby"],
    },
    "bash": {
        "versions": ["5.3"],
        "image_tpl": "bash:{version}",
        "file_ext": "sh",
        "interpreter": ["bash"],
    },
    "go": {
        "versions": ["1.27", "1.26"],
        "image_tpl": "golang:{version}-alpine",
        "file_ext": "go",
        # exec keeps the PID, so the entrypoint measures the program instead of the go tool that built it
        "interpreter": [
            "sh",
            "-c",
            'go build -o /tmp/main -ldflags="-s -w" -gcflags=all=-dwarf=false "$0" && exec /tmp/main',
        ],
        # A cold build compiles the runtime and std into GOCACHE: about 50Mi of /tmp, 230Mi RSS, 10s of CPU.
        "pod": PodRuntimeConfig(
            env={"HOME": "/tmp", "GOCACHE": "/tmp/go-cache"},  # NOSONAR  # nosec B108
            tmp_size="64Mi",
            min_cpu="1000m",
            min_memory="320Mi",
            min_timeout_seconds=30,
        ),
    },
}

CPU and memory limits and the timeout arrive in the event, taken from the admin system settings when the execution was requested. Requests come from K8S_POD_CPU_REQUEST and K8S_POD_MEMORY_REQUEST. The timeout becomes the pod's activeDeadlineSeconds.

For a cancelled execution the worker deletes the pod with no grace period and lets the ownerReference take the ConfigMap with it. A queued execution has no pod yet, so a 404 there is expected.

Pod security

Control Setting
User runAsNonRoot, UID and GID 1000, hostUsers: false (user namespace)
Filesystem Read-only root; writable /output (10Mi) and /tmp (10Mi, 64Mi for Go) emptyDirs
Privileges allowPrivilegeEscalation: false, all capabilities dropped
Seccomp RuntimeDefault
API access automountServiceAccountToken: false, enableServiceLinks: false
Host namespaces No host network, PID or IPC
Runtime class K8S_POD_RUNTIME_CLASS_NAME if set (for example gVisor)

The worker refuses to start if K8S_NAMESPACE is default.

Startup

Before the worker consumes anything, it applies two things to the executor namespace. The first is the executor-deny-all NetworkPolicy, which selects pods labeled component=executor and allows no ingress and no egress, DNS included. The second is the Pod Security Admission labels for the restricted profile (enforce, warn and audit). The worker is the only thing that writes this policy, and it takes effect only if the cluster's network plugin enforces NetworkPolicy.

It also applies the runtime-image-pre-puller DaemonSet, which pulls every image in the runtime registry on each node. The worker doesn't wait for those pulls, so an execution that arrives before they finish still waits for its image.

Every executor pod counts against the executor-quota ResourceQuota that the worker applies to its namespace on start (K8S_NAMESPACE_CPU_QUOTA and K8S_NAMESPACE_MEMORY_QUOTA, 32 CPU and 4Gi of requests). A pod over the quota is refused with 403 Forbidden, and the execution is stored as failed.

Key files

File Purpose
worker.py Entry point, startup setup
worker.py Pod creation and deletion, namespace policy, pre-puller
pod_builder.py ConfigMap and pod spec
runtime_registry.py Languages, images, per-language pod settings