Skip to content
Faber Faber

Request lifecycle

This page follows one POST /api/v1/execute from the first byte to the response. The architecture page introduces the processes involved; the execution model describes requests and results from the client’s side.

The whole sequence

Section titled “The whole sequence”
  1. Admission.
  2. Client to API server: POST /api/v1/execute
  3. API server: Request ID, API key, rate limit, body deadline and size
  4. API server: Validate steps and profiles; fetch uploaded files
  5. API server: Step cache: longest cached prefix; wait for an identical request in flight
  6. API server: Reserve one slot per task of the widest step, or 503
  7. API server to Controller: spawn_blocking, holding the slots
  8. Sandbox setup.
  9. Controller: Create faber/req-*; lease a host UID/GID
  10. Controller to Jailer: start /proc/self/exe; job and files in a memfd
  11. Jailer: Join req-*/supervisor; oom_score_adj -999
  12. Jailer: New mount, network, UTS, IPC namespaces; pivot_root; mount /faber, /tmp, /dev; root read-only
  13. Jailer: Restore the cached workspace, if any
  14. Each step.
  15. Jailer to Supervisor: fork, one per task
  16. Supervisor: New PID namespace and its init; task-* cgroup; write the task's files
  17. Supervisor to Task: fork
  18. Task: Join task-*; own /proc; rlimits; user namespace; drop capabilities; Landlock; seccomp; exec
  19. Supervisor to Task: stdin
  20. Task to Supervisor: stdout, stderr
  21. Supervisor to Task: wall timeout or output limit: cgroup.kill
  22. Supervisor: Read stats and events; remove task-*; pick the outcome; kill init
  23. Supervisor to Jailer: task result
  24. Jailer to Controller: workspace snapshot, for cacheable steps
  25. Results and teardown.
  26. Jailer to Controller: result frame; the jailer exits
  27. Controller: Delete the sandbox root; kill and remove req-*; release the UID; free the slots
  28. Controller to API server: results and snapshots
  29. API server: Store cache layers; release the cache lock
  30. API server to Client: 200 with one result per task
  • Faber and the kernel (trusted)
  • Submitted code (untrusted)
  • Outside Faber
  • Starts or calls
  • Carries data
  • Kills
One request with one task per step. A parallel step forks one supervisor per task at once, and the jailer collects their results in order.

Admission

Section titled “Admission”

Everything here happens in the API server’s async handler, before any sandbox exists. A request that fails a check gets an error response and nothing runs.

  1. Every request gets an X-Request-Id and a log line. Its body must arrive within REQUEST_BODY_TIMEOUT_MS (otherwise 400) and fit EXECUTE_BODY_LIMIT_BYTES (otherwise 413).
  2. The API key is compared, in constant time, with every configured key (401). With RATE_LIMIT_PER_MINUTE set, the key’s token bucket must have a token left (429 with Retry-After).
  3. Validation. No steps or an empty parallel step is 400. More steps than MAX_STEPS_PER_REQUEST, or a step wider than MAX_PARALLEL_TASKS, is 422 limits_exceeded. A profile outside ALLOWED_SANDBOX_PROFILES is 422 profile_not_allowed. Tasks without a profile get the default one written into them here, so it is part of their cache key. File IDs must be SHA-256 hex and inline content must decode (400).
  4. Uploaded files the tasks reference are read from the file store. A missing one is 404 file_not_found; more than the workspace’s 128 MiB in total is 422.
  5. The step cache, when enabled, computes the keys of the leading run of cacheable steps, finds the longest cached prefix, and if the prefix is incomplete may wait for an identical request that is already computing the next layer (step cache internals). If every step is a hit, the response is built from the cache and no sandbox starts.
  6. Slots. The request takes as many task slots as its widest step has tasks, without waiting: if MAX_CONCURRENCY cannot cover them, the answer is 503 capacity_exhausted with Retry-After: 1. The cache wait comes before this, so a request waiting for another one’s result holds no slots.

The slot permit then moves into a blocking thread, the request’s controller, and is released only when that thread returns.

Sandbox setup

Section titled “Sandbox setup”

The controller creates the request cgroup faber/req-<pid>-<random>, with a memory and PID limit sized for its widest step and a supervisor leaf for Faber’s own processes, and leases a host UID/GID that no other running request has. It writes the job and every file blob into a memfd and starts the jailer with that memfd as stdin.

The jailer joins the supervisor leaf, then builds the sandbox: new mount, network, UTS and IPC namespaces, a root of read-only bind mounts under /tmp/faber/<random>, pivot_root into it, the device nodes, tmpfs mounts on /faber and /tmp, hostname faber, the old root detached, and the new root remounted read-only. The filesystem a task sees lists every mount. If the step cache replayed a prefix, the jailer recreates its workspace before the first step.

A failure anywhere in this phase ends the request with 500 sandbox_setup_failed. When the failure was restoring a cached workspace, the API server also deletes that cache layer, so the next request runs the step again instead of failing the same way.

Each step

Section titled “Each step”

Before a step, the jailer checks two things:

  • With stop_on_failure, if any earlier task did not complete with exit code 0, the step’s tasks are reported not_started (“Skipped because an earlier step failed”).
  • If the request’s deadline has passed, the step’s tasks are reported not_started too. Otherwise each task’s wall timeout is WALL_TIMEOUT_MS or the time left, whichever is shorter.

Then it forks one supervisor per task and collects their results in order. Each supervisor enters a new PID namespace and forks its init, creates the task cgroup, writes the task’s files into the workspace, and forks the task. The task joins its cgroup before anything else, so every byte and process it uses afterwards is counted, then locks itself down (see isolation layers) and calls exec.

While the task runs, the supervisor writes its stdin, drains stdout and stderr, and checks the clock every 10 ms. Output is kept up to OUTPUT_LIMIT_BYTES per stream and up to the task’s share of what is left of REQUEST_OUTPUT_LIMIT_BYTES (the remainder divided by the width of the step). When either runs out, or the wall timeout passes, the supervisor kills the whole task cgroup with cgroup.kill. Once the task has exited, it reads the cgroup’s statistics and events, removes the cgroup, decides the outcome, and kills the init.

After a step that is part of the cacheable run and finished cleanly, the jailer snapshots the workspace and streams the snapshot to the controller.

Results and teardown

Section titled “Results and teardown”

After the last step the jailer writes its result frame and exits. The controller, which has been reading the jailer’s output all along, decodes it, checks every snapshot blob against its SHA-256, then tears down: it deletes the sandbox root, kills whatever is left in the request cgroup and removes it, and only then releases the UID. Returning from the blocking thread drops the slot permit.

Back in the async handler, the API server stores the new cache layers, releases its cache lock (waking requests that waited for these layers), and answers 200 with one result per task.

Cancellation and deadlines

Section titled “Cancellation and deadlines”
  • Faber and the kernel (trusted)
  • Submitted code (untrusted)
  • Outside Faber
  • Starts or calls
  • Kills
Every way out ends in the same teardown, and the slots are freed only after it. A disconnecting client cannot free capacity while its sandbox still runs.

The controller waits for the jailer in a loop that polls its output pipe for at most 50 ms at a time, and on each pass checks:

  • Cancellation. The handler holds a guard that cancels the request when it is dropped, which is what happens when the client disconnects. The API server answers 503 cancelled, though nobody is left to read it.
  • Shutdown. SIGTERM sets a process-wide flag and kills every request cgroup at once. Running requests answer 503 shutting_down, and the server gives in-flight responses SHUTDOWN_TIMEOUT_MS to drain.
  • The hard deadline. The jailer enforces OVERALL_TIMEOUT_MS itself, between and within steps, and returns the steps it finished. The controller only steps in if the jailer is still running two seconds after the deadline, and answers 504 deadline_exceeded.

In all three cases it kills the jailer’s process group (the jailer, its supervisors and their inits) and the request cgroup, whose cgroup.kill reaches every task, then reaps the jailer and tears down as above. If the jailer exits while one of its descendants still holds the result pipe open, the controller kills the request cgroup too, so that the pipe closes.

Where this lives in the code

Section titled “Where this lives in the code”