Skip to content
Faber Faber

Step cache internals

The step cache page covers using the cache: opting in, what is stored, the configuration. This page covers how it works, from the key of a step to a layer shared by several nodes through Redis.

Keys

Section titled “Keys”

Only the leading run of cacheable steps is looked up and stored: from the first step up to the first one with a task that did not opt in. Each step of that run gets a key chained from the key before it, and the first from a digest of the service itself:

  • Faber and the kernel (trusted)
  • Outside Faber
  • Starts or calls
A key depends on everything before it. The same test step after a different compile step has a different key, and changing a limit or the image gives every step a new key, so nothing ever has to be invalidated.

The policy digest is the SHA-256 of a sorted-key JSON document holding: the format tag faber-layer-v1; the CPU architecture; CACHE_NAMESPACE; FABER_TOOLCHAINS; FABER_IMAGE_ID; the read-only paths; MEMORY_MAX, PIDS_MAX, CPU_MAX, the wall, CPU and overall time limits, both output limits, the workspace size and the tmpfs inode limit; the default sandbox profile; and the sandbox policy version, which is the crate version plus a revision bumped whenever a change to the sandbox could change a result.

A step’s canonical form is its JSON with object keys sorted at every level: each task’s cmd, args, env, stdin, working_dir and sandbox_profile (the server’s default already filled in), with every file replaced by {"sha256", "executable"} of its decoded bytes. A parallel step is an array in task order. The cache flag itself is not part of it. An inline string, base64 content and an uploaded file_id with the same bytes give the same key, since a file_id is the SHA-256 of the file.

What a layer holds

Section titled “What a layer holds”
  • Faber and the kernel (trusted)
  • Submitted code (untrusted)
  • Carries data
A layer is small; the bytes are in blobs, and layers that contain the same file share its blob.

A step is stored only if every task completed with exit code 0, outcome exited and no truncated output, and its workspace held nothing but directories and regular files within the size limits. Both parts are untrusted content: whatever the task printed and wrote.

Snapshot and restore in the jailer

Section titled “Snapshot and restore in the jailer”

The workspace only exists inside the sandbox, so the jailer is the one that captures it after a step and recreates it before the first step of a request that replays a prefix.

  1. Restore, before the first step.
  2. API server to Controller: manifest and blobs of the last hit
  3. Controller to Jailer: manifest in the job; blobs in the memfd
  4. Jailer: Validate the manifest, then create every directory and file under /faber the way task files are written: no links followed, owned by the task user, 0644 or 0755
  5. Snapshot, after each step of the cacheable run.
  6. Jailer: Only if the step was clean. Pass 1: walk /faber without following links, hash every regular file, refuse anything else, count files and bytes
  7. Jailer to Controller: snapshot header: manifest and the new blobs' hashes
  8. Jailer: Pass 2: stream each blob the controller does not have yet, once
  9. Jailer to Controller: blob frames
  10. Controller: Check every blob against its SHA-256; a mismatch makes the snapshot incomplete
  11. Store, after the sandbox is gone.
  12. Controller to API server: results and snapshots
  13. API server: For each step in order: store its blobs, then its layer; stop at the first that cannot be stored
  • Faber and the kernel (trusted)
  • Starts or calls
  • Carries data
Neither pass keeps file content in the jailer’s memory. A file that changes between the passes still streams the declared number of bytes, so the output stays parseable, and the hash check discards the snapshot.

A snapshot is incomplete, and the step not stored, when the workspace holds a symlink, FIFO, socket or device, a file name that is not UTF-8, more than CACHE_MAX_LAYER_FILES entries, or more than CACHE_MAX_LAYER_BYTES of content. The new content streamed across all snapshots of one request is capped at CACHE_MAX_LAYER_BYTES too. The first step that cannot be stored ends snapshotting for the request: a later layer could never be reached without it.

/tmp is not captured, and restored files keep only the executable bit: mode 0755 or 0644, directories 0755, all owned by the task user.

Tiers

Section titled “Tiers”
  • Faber and the kernel (trusted)
  • Outside Faber
  • Starts or calls
  • Carries data
The local tier is always there. Redis is there with CACHE_BACKEND=redis, and its failures read as misses.
  • Writes go to both tiers. In Redis, the blobs go first, existing ones only getting a fresh expiry, and the layer last, so that no node can see a layer whose content is not there yet.
  • Blob verification. Every blob read, from either tier, is hashed and compared with its key. A mismatch deletes the blob and counts as a miss. Layers are not hashed: they are parsed, must hold clean results and a valid manifest, and are deleted otherwise. Anyone who can write to Redis can therefore make Faber return any results; keep it private to Faber.
  • Lookup. Faber reads the layers of the run’s keys in order and stops at the first miss, or when the replayed output would exceed REQUEST_OUTPUT_LIMIT_BYTES. The workspace of the last hit is then needed in full; if any of its blobs is missing, it falls back to the hit before, and so on.
  • Clearing. DELETE /api/v1/cache empties the local tier and, with Redis, increments ns:generation and deletes every other key of the namespace. Every node reads the generation at the start of each lookup and drops its own local tier when it has changed.

Single flight

Section titled “Single flight”

When several requests need the same missing layer at the same time, one computes it and the others wait for it, once. In one process the waiting is a channel per key: the second request waits until the first has stored its layers or gone away. Across nodes it is a lock in Redis:

  1. Node A to Redis: SET ns:lock:k <node> NX PX <deadline>
  2. Redis to Node A: OK: A computes layer k
  3. Node B to Redis: the same SET
  4. Redis to Node B: nil: held by A
  5. Node B: Poll every 250 ms: has layer:k appeared, or lock:k gone?
  6. Node A: Run the sandbox
  7. Node A to Redis: blobs, then layer:k
  8. Node A to Redis: unlock, only if the lock is still A's
  9. Node B: Look up again: replay layer k, run the rest
  • Faber and the kernel (trusted)
  • Outside Faber
  • Starts or calls
  • Carries data
The lock and every wait last at most the overall timeout plus 15 seconds. A request waits at most once; if the layer is still missing afterwards, it runs the step itself. A request that waits holds no task slots.

The lock value names the node (hostname, PID and start time), and the unlock is a script that deletes the key only if it still holds that value, so a slow node never releases a lock another node has taken since. Correctness never depends on the lock: if it fails or expires, the step runs twice and both nodes store the same layer.

When something fails

Section titled “When something fails”
FailureWhat happens
Redis is down or slowEach operation gets 2 seconds. A failure reads as a miss, is counted in faber_cache_shared_errors_total and logged once, and Redis is left alone for 5 seconds. Locks are skipped and the step runs
A layer cannot be parsed, or holds unclean resultsIt is deleted from both tiers and the lookup stops there
A blob does not match its SHA-256It is deleted and counts as a miss; the lookup falls back to an earlier step
Restoring a workspace fails in the sandboxThe request gets 500 sandbox_setup_failed, and the layer that was being restored is deleted so the next request runs the step
A snapshot is incomplete or a step was not cleanNothing from that step on is stored; the request’s results are unaffected
The node holding a lock diesThe lock expires; waiters stop polling by then and run the step
The computing request goes away before storingIts lock is released in the background, and local waiters wake up

Where this lives in the code

Section titled “Where this lives in the code”