Architecture and process model

Faber is one executable, /opt/faber in the image, and every process that serves a request runs it. The sandbox page describes the sandbox from the inside; this page describes the processes around it.

Five roles

Role	How many	Started by	Job
API server	One per container	Docker	Serves HTTP. Holds the API keys, the file store, the step cache and the task slots. One of its blocking threads acts as the controller of each running request
Jailer	One per running request	The controller, by starting the same executable again	Builds the request’s sandbox, runs its steps in order, snapshots the workspace for the cache
Supervisor	One per running task	The jailer, fork	Creates the task’s cgroup and files, starts the task in a PID namespace of its own, feeds its stdin, collects its output, enforces its wall and output limits
Init	One per running task	The supervisor, fork	PID 1 of the task’s PID namespace. Reaps orphaned processes until the supervisor kills it
Task	One per task	The supervisor, fork then exec	Your command

Only the last is untrusted. The jailer, the supervisors and the inits run as root in the container, and a task drops every privilege before it runs your command.

Who starts whom

• faber API serverroot Tokio runtime. The controller of each request runs on a blocking thread of its own.
  • faber-jailer started again from /proc/self/exeone per request Empty environment, own process group, the job on stdin. Takes over before main runs.
    • supervisor forkone per task Calls unshare(CLONE_NEWPID): its next children start in a new PID namespace.
      • init forkPID 1 Loops on waitpid to reap orphans. When it dies, the kernel kills everything left in the namespace.
      • task fork + execPID 2 Your command. Anything it starts and abandons is reparented to init.

• Faber and the kernel (trusted)
• Submitted code (untrusted)

The task is a child of its supervisor, not of the init: the supervisor creates the PID namespace, the init holds it open as PID 1, and the task is the second process started in it. A parallel step has one supervisor, init and task per task, side by side under the same jailer.

Why the jailer is a new process

The API server is multithreaded. A fork copies only the thread that called it, and locks held by other threads at that moment (the memory allocator’s, the logger’s) stay locked forever in the child. So the controller does not fork sandbox code out of the server. It starts /proc/self/exe again, as faber-jailer, with:

• an empty environment plus __FABER_JAILER=1. A function in the binary’s .init_array sees that variable and runs the jailer before main would start the server;
• stdin set to a memfd holding the job, stdout to a pipe back to the controller, and stderr inherited from the server. The jailer closes every other descriptor it may have inherited;
• a process group of its own, so the controller can signal the jailer and everything it forked at once.

The jailer therefore holds none of the server’s memory, environment or open files, the API keys included. It asks the kernel to kill it if its parent dies (PR_SET_PDEATHSIG) and exits at once if its parent is no longer the API server that started it. Supervisors and inits set the same death signal, so if the server dies, the whole chain goes with it.

Namespaces and cgroups

namespaces The container's As Docker created them, with the host's cgroup namespace.

cgroup <container>/faber-init API server root

namespaces: mount, network, UTS, IPC The request's Created by the jailer with unshare. PID namespace: still the container's.

cgroup faber/req-*/supervisor Jailer root, oom_score_adj -999

fork

cgroup faber/req-*/supervisor Supervisor root, oom_score_adj -999

namespace: PID The task's Created by the supervisor, one per task.

cgroup faber/req-*/supervisor Init, PID 1 root

namespaces: mount (a copy of the request's), user The task's own

cgroup faber/req-*/task-* Task, PID 2 65534 inside, a leased host UID outside

• Faber and the kernel (trusted)
• Submitted code (untrusted)

Each box is a set of namespaces the processes inside share; the tag on each process is its cgroup. The jailer creates the request's namespaces once; the supervisor and the task add theirs per task.

Process	Mount	Network, UTS, IPC	PID	User	Cgroup	Runs as
API server	container	container	container	initial	<container>/faber-init	root
Jailer	request	request	container	initial	faber/req-*/supervisor	root
Supervisor	request	request	container	initial	faber/req-*/supervisor	root
Init	request	request	task	initial	faber/req-*/supervisor	root
Task	task (copy of request)	request	task	task	faber/req-*/task-*	65534 inside, the request’s leased UID outside

Some details behind the table:

• faber-init. On its first start, Faber moves the container’s processes into <container>/faber-init: a cgroup that enables controllers for its children cannot hold processes itself. A restarted service finds itself there and reuses the hierarchy.
• The supervisor leaf. The request cgroup has controllers enabled, so the jailer cannot join it directly. It joins the request’s supervisor child instead, with a memory allowance of its own, and sets oom_score_adj to -999 so that within the request the OOM killer picks a task first. The supervisors and inits inherit both; the task sets its score back to 0.
• The task’s mount namespace. The task unshares a copy of the request’s mount namespace only to replace /proc and /sys with its own view (see the filesystem a task sees). Its mounts are otherwise the request’s.
• The user namespace is created per task, by the request’s leased host identity; isolation layers shows the handshake.

Cgroups and resource accounting has the whole cgroup tree with its limits.

How the processes talk

a thread of the API server Controller

the job, in a memfd on stdin

frames on stdout: snapshots, blobs, the result

one per request Jailer

fork

the task's result as JSON, over a pipe

one per task Supervisor

fork, exec; stdin

stdout, stderr; the status pipe

Task

• Faber and the kernel (trusted)
• Submitted code (untrusted)
• Starts or calls
• Carries data

Everything the jailer needs arrives in one memfd that is complete before the jailer starts, so handing it over never blocks. Its pages are charged to the API server, which holds the same bytes anyway, not to the request.

• The job. The memfd starts with a little-endian u64 length and a JSON header: the steps, the sandbox and cgroup configuration, the request cgroup path, the limits, the time left until the request’s deadline, the leased identity, which steps to snapshot and which workspace to restore, and an index of blobs. After the header comes the blob region: the content of uploaded files the tasks reference and of a cached workspace to restore. The jailer and the supervisors read blobs from it by offset, so processes sharing the descriptor never race on a file position.
• Frames. The jailer’s stdout is a sequence of frames, each a tag byte, a u64 length and a payload: a snapshot header (tag 1), the file contents it lists (tag 2), and finally the result (tag 3). Snapshots exist only when the step cache asked for them.
• The status pipe. A task’s setup between fork and exec reports a failure as a five-byte record: the stage that failed and the errno. The pipe is close-on-exec, so end-of-file means exec succeeded.
• Stdio. The supervisor polls the three pipes every 10 ms, writing stdin and draining stdout and stderr concurrently so that neither side can deadlock the other.

Lifetimes

A task ends, and its supervisor reads the cgroup’s statistics, removes the task cgroup, kills the init (which ends the PID namespace and anything left in it), writes the result to the jailer and exits. After the last step the jailer writes the result frame and exits. The controller then unmounts and deletes the sandbox root under /tmp/faber/, kills and removes the request cgroup, and releases the identity.

If the API server dies instead, the death signals take the jailer, supervisors and inits with it, and the next start of the service kills whatever is left in faber/req-* cgroups and deletes every root under /tmp/faber/.

Where this lives in the code

• src/main.rs: startup, reclaiming leftovers, sizing the service cgroup.
• crates/faber-api/src/handlers/execute.rs: runs the controller on a blocking thread.
• crates/faber-runtime/src/runtime/jailer.rs: starting the jailer, the job memfd, the output frames.
• crates/faber-runtime/src/runtime/core.rs: the controller loop, supervisors, inits and task setup.
• crates/faber-runtime/src/runtime/workspace.rs: blobs read by offset from the job.
• crates/faber-runtime/src/cgroup/core.rs and request.rs: faber-init, the request cgroup and its supervisor leaf.