Kubernetes

The manifests in deploy/kubernetes/ run Faber as a Deployment of N pods that share one Redis for the step cache:

             Ingress (TLS) ──▶ Service faber :3000
                                   │
              ┌────────────────────┼────────────────────┐
              ▼                    ▼                    ▼
        pod faber            pod faber            pod faber        Deployment, one pod per node,
        (node A)             (node B)             (node C)         privileged, dedicated nodes
              └────────────────────┼────────────────────┘
                                   ▼
                          Service redis :6379
                                   │
                              pod redis                            Deployment, no persistence

Manifest	Contents
Faber Deployment	The privileged Faber pods
Faber Service	ClusterIP in front of the pods, port 3000
Redis Deployment and Service	The shared cache tier
Secret	API_KEY (and the Redis password, if you set one)

Read the README.md next to the manifests for the exact apply order and the values to change.

The Faber pod

Faber needs the same things as in Docker: a privileged container, the host’s cgroup tree, and /sys/fs/cgroup mounted read-write.

spec:
  containers:
    - name: faber
      image: vgfractal/faber:0.3.0
      securityContext:
        privileged: true
        readOnlyRootFilesystem: true
      volumeMounts:
        - { name: cgroup, mountPath: /sys/fs/cgroup }
        - { name: tmp, mountPath: /tmp }
  volumes:
    - name: cgroup
      hostPath: { path: /sys/fs/cgroup, type: Directory }
    - name: tmp
      emptyDir: { medium: Memory, sizeLimit: 64Mi }

There is no pod field for the cgroup namespace. With containerd and CRI-O on cgroup v2, a privileged container is not given a private cgroup namespace, which is the equivalent of Docker’s --cgroupns=host. Faber checks this at startup and refuses to run if its cgroup namespace and the mounted tree disagree, so a misconfigured node fails loudly rather than running unconfined.

Other settings that matter:

• Resources. Set the memory limit from the sizing formula plus headroom, and request the same amount, so the scheduler places pods only where that memory exists. The pod limit is the backstop above Faber’s own aggregate cgroup limit.
• Probes. Point readiness and liveness at GET /api/v1/health. It answers 503 while the pod drains, so readiness takes it out of the Service first.
• Shutdown. Keep terminationGracePeriodSeconds above SHUTDOWN_TIMEOUT_MS.
• Reaping. Docker’s --init has no pod equivalent. Set shareProcessNamespace: true so the pod’s pause process reaps supervisors orphaned by an abnormal exit, or start Faber under an init such as tini.
• Configuration. Every pod gets the same environment: the same limits, FABER_TOOLCHAINS, FABER_IMAGE_ID and CACHE_NAMESPACE. Pods with different settings compute different cache keys and do not share layers.

Scheduling rules

One Faber pod per node, on nodes reserved for Faber

A Faber pod is root-equivalent on its node, and a sandbox escape through a kernel bug compromises the node and everything scheduled on it.

• Dedicated nodes. Taint a node pool for Faber and give the Deployment the matching toleration and node selector, so no other workload shares those nodes.
• One pod per node. Use a required pod anti-affinity on the hostname topology key. Each Faber process leases sandbox UIDs from SANDBOX_IDENTITY_BASE, and two pods on one kernel would hand out the same ones. If you must pack several pods per node, give each a separate range, which a Deployment cannot do; use one Deployment per range.
• Kernel. Nodes need Linux 5.8 or newer with cgroup v2 (the default on current distributions and managed Kubernetes images).

Redis

• One replica, no persistence: an empty cache only means misses.
• Set maxmemory and maxmemory-policy allkeys-lru so it evicts instead of refusing writes.
• Require a password (from the Secret) and restrict access with a NetworkPolicy so only Faber pods reach it. It stores workspace contents.
• If Redis is down, Faber keeps serving and treats every lookup as a miss.

CACHE_REDIS_URL points at the Service, for example redis://:$(REDIS_PASSWORD)@redis:6379/0.

Network

• Expose Faber through an Ingress or Gateway that terminates TLS, and only to your own backend. Faber has a single API key scope; it is not meant to be called by end users directly.
• Sandboxed tasks have no network at all, whatever the cluster’s network policy. NetworkPolicies protect Faber and Redis themselves.

Scaling

Each pod admits up to its own MAX_CONCURRENCY tasks and answers 503 with Retry-After beyond that. Scale on faber_execution_slots_in_use relative to faber_execution_slots, or on the rate of 503 responses (faber_execute_responses_total{status="503"}), rather than on CPU.

Uploaded files stay in the pod that received them. If clients use the file store, enable session affinity on the Service (sessionAffinity: ClientIP) or have clients re-upload on file_not_found.