Pod and namespace security¶
User code runs in executor pods in the integr8scode namespace. The pod builder locks down each pod, the k8s worker locks down the namespace at startup, and a ResourceQuota caps how much CPU and memory the pods can request in total.
Executor pods¶
pod_builder.py builds every executor pod the same way. The container runs as UID and GID 1000 with runAsNonRoot, a read-only root filesystem, allowPrivilegeEscalation: false, all capabilities dropped and the RuntimeDefault seccomp profile.
| Setting | Value | Why |
|---|---|---|
hostUsers |
false |
user namespaces: container UIDs map to unprivileged host UIDs |
automountServiceAccountToken |
false |
no token, so no Kubernetes API access |
enableServiceLinks |
false |
no service env vars to discover the cluster |
hostNetwork, hostPID, hostIPC |
false |
no host namespaces |
restartPolicy |
Never |
one-shot pods |
activeDeadlineSeconds |
the execution timeout, at least the language minimum | the kubelet kills the pod at the deadline |
runtimeClassName |
K8S_POD_RUNTIME_CLASS_NAME, unset by default |
lets you run pods under gVisor or another sandbox |
The script and entrypoint.sh come from a per-execution ConfigMap, mounted read-only. The only writable paths are two emptyDir volumes: /output (10Mi) and /tmp (10Mi, or more for languages that need it).
The timeout and the CPU and memory limits are the admin settings max_timeout_seconds, cpu_limit and memory_limit. The first read stores them in MongoDB, seeded from K8S_POD_EXECUTION_TIMEOUT (5 seconds), K8S_POD_CPU_LIMIT (1000m) and K8S_POD_MEMORY_LIMIT (128Mi) in config.toml. After that, editing those keys changes nothing until an admin resets the settings. A language can raise all three through its PodRuntimeConfig in runtime_registry.py. Today only Go does: at least 1000m CPU, 320Mi memory, 30 seconds and a 64Mi /tmp for its build cache. Requests always come from K8S_POD_CPU_REQUEST (200m) and K8S_POD_MEMORY_REQUEST (128Mi).
Namespace¶
The k8s worker calls ensure_namespace_security() when it starts. It server-side applies three objects, so each step is safe to repeat: the namespace with its Pod Security labels, the NetworkPolicy and the ResourceQuota.
The first creates or updates the executor-deny-all NetworkPolicy. It selects pods labelled component: executor, which every executor pod has, and denies all ingress and egress. The worker is the only owner of this policy.
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: executor-deny-all
namespace: integr8scode
labels: {app: integr8s, component: security}
spec:
podSelector:
matchLabels: {component: executor}
policyTypes: [Ingress, Egress]
ingress: []
egress: []
The second labels the namespace with the restricted Pod Security Standard for enforce, warn and audit. The API server then rejects any pod that asks for privileges the profile forbids.
The worker also applies a runtime-image-pre-puller DaemonSet that pulls every runtime image onto each node, so executor pods don't wait on a pull. Its containers also follow the restricted profile.
The worker reaches the cluster with k3s's own kubeconfig, mounted read-only from /etc/rancher/k3s into the worker container only, so it holds the cluster's admin rights. A service account limited to the namespace would not hold it back: the worker sets the namespace's Pod Security labels itself, and an account allowed to change those labels can lift the namespace out of restricted. The API container mounts no kubeconfig.
Quota¶
The executor-quota ResourceQuota caps the total CPU and memory that pods in the namespace request, not what they use: a default pod requests 200m CPU but may use up to its 1000m limit. The worker applies it with requests.cpu from K8S_NAMESPACE_CPU_QUOTA (32) and requests.memory from K8S_NAMESPACE_MEMORY_QUOTA (4Gi). The API server refuses a pod that would go over it with 403 Forbidden, and the execution scheduler stores that execution as failed with error type system_error.
How many executions run at once is decided by the execution scheduler's queue (max_concurrent_executions, 10 by default). Every pod requests 200m CPU and 128Mi, Go's included, since Go raises only its limits, so 10 executions request 2 CPU and 1.25Gi. The memory quota fills at 32 pods, which makes it a backstop that refuses pods only when the admin settings allow more than that at once.
Network policy enforcement¶
A NetworkPolicy only works if something enforces it. k3s does, through its embedded network policy controller, unless it was started with --disable-network-policy. On other clusters you need a CNI that enforces policies, such as Calico or Cilium; plain Flannel doesn't.
To let executors reach something specific, add a separate NetworkPolicy with explicit egress rules instead of editing executor-deny-all. The worker would overwrite your edit on its next start anyway.