Runbook

Requests are shed (503) although traffic is low

Check faber_execution_slots_in_use. A slot is held until its sandbox is really gone, so abandoned requests release theirs within the wall timeout. If slots stay held with no traffic, restart the container: startup kills every leftover sandbox and removes its cgroups and root directories.

Leaked cgroups or sandbox roots

Inside the container, request cgroups are faber/req-* beneath the container’s own cgroup, and sandbox roots are /tmp/faber/*. They should exist only while a request runs. Leftovers after a crash are reclaimed at the next start. To look:

docker exec faber sh -c 'find /sys/fs/cgroup -type d -name "req-*"; ls /tmp/faber'

OOM kills

• out_of_memory: a task exceeded MEMORY_MAX. Expected for hostile or oversized workloads.
• ancestor_out_of_memory: a request, service or container limit was hit first. Lower MAX_CONCURRENCY or raise the container’s memory limit; see sizing.
• The kernel log shows the Faber process itself being killed: the container limit is below the sizing figure.

Policy violations from a legitimate toolchain

The task dies with policy_violation (a denied syscall) or misbehaves after an ENOSYS (a syscall not on the v2 allowlist).

1. Reproduce with strace -f outside Faber to find the syscall.
2. Denied syscalls are denied on purpose; the toolchain has to do without, or the task needs compile_v2 instead of native_v2 (threads and processes).
3. For an unlisted syscall, allowing the v1 profiles (ALLOWED_SANDBOX_PROFILES=compile_v2,native_v2,compile_v1,native_v1) gives clients a denylist profile that runs it. That is weaker; the permanent fix is to add the syscall to the allowlist in the runtime and report it upstream.

Slow or hanging requests

Every task has a wall timeout and every request a deadline with a hard backstop, so a request cannot outlive OVERALL_TIMEOUT_MS by more than a few seconds. If requests are slow across the board, check host CPU and memory pressure and cpu_throttled_usec in task stats.

The step cache

Hit rate dropped to zero after a deploy. Expected if the image (FABER_IMAGE_ID, FABER_TOOLCHAINS) or any limit changed: they are part of every key, so old layers are no longer used. They expire after CACHE_TTL_SECS.

Instances do not share layers. Compare GET /info on each: limits, profiles and toolchains must match. Check that every instance has the same CACHE_NAMESPACE and FABER_IMAGE_ID, and runs on the same architecture.

Redis errors in the logs or metrics. Requests keep working, as misses. Check connectivity, credentials and Redis memory. A full Redis without an eviction policy refuses writes; set maxmemory-policy allkeys-lru.

A cached result is wrong. Usually a step marked cache: true that is not deterministic. Clear the namespace with DELETE /api/v1/cache (it affects every instance sharing the namespace) and fix the request so the step is no longer marked cacheable.

Shutdown and upgrades

SIGTERM stops new work, kills running sandboxes and exits within SHUTDOWN_TIMEOUT_MS; /health answers 503 meanwhile, and in-flight requests get 503 (shutting_down). To upgrade without failed requests, take the instance out of the load balancer, wait for faber_execution_slots_in_use to reach 0, then replace it.

Suspected sandbox escape

Treat the host as compromised: the Faber container is privileged.

1. Stop the container and take the host out of service.
2. Preserve the logs. Request IDs identify the requests involved; request bodies are not logged.
3. Rotate the API keys and the Redis password.
4. Clear the shared cache namespace, or the whole Redis.
5. Rebuild the host from a known-good image.
6. Report the finding privately, as described in the threat model.

Verifying a deployment

From a checkout of the repository, the same checks CI runs can be pointed at your images inside a disposable VM, never on your own kernel:

./scripts/vm.sh prepare          # once: builds the images in a disposable VM
./scripts/vm.sh test             # sandbox acceptance suites
./scripts/vm.sh test-docker      # production image smoke test
./scripts/vm.sh test-toolchains  # language toolchains under the default profile
./scripts/vm.sh soak             # mixed concurrent load, then leak checks

scripts/soak.sh and scripts/toolchain-test.sh also run against any reachable instance (API_URL, API_KEY). The soak sends hostile requests and expects the default limits; run it against staging, not production.