Skip to content
Faber Faber

Isolation layers

No single kernel feature makes a sandbox. Faber stacks eleven of them around every task, and most of them cover a gap that another leaves: the read-only root stops writes, Landlock stops the writes a read-only mount does not cover, seccomp stops the system calls that could undo either. The sandbox page describes the result from inside a task; this page takes it apart.

The layers, outermost first

Section titled “The layers, outermost first”

The numbers are the order in which they are applied. The first two are set up once per request by the jailer; the rest are set up for every task, by its supervisor or by the task process itself between fork and exec.

  1. Mount namespace and read-only root per request, by the jailer unshare(CLONE_NEWNS), pivot_root, root remounted ro,nosuid,nodev,noexec
    stops
    Seeing the image beyond its read-only toolchain paths. Creating or changing files anywhere but /faber, /tmp and /dev/mqueue (EROFS). Setuid bits everywhere, and device files other than the five in /dev.
    leaves open
    Reading the toolchains and the few files under /etc. Tasks of one request share the workspace and /tmp.
  2. Network, UTS and IPC namespaces per request, by the jailer CLONE_NEWNET, CLONE_NEWUTS, CLONE_NEWIPC
    stops
    Every connection off the sandbox: the only interface is loopback. Renaming the host. Reaching the host's or other requests' System V IPC and message queues.
    leaves open
    Tasks of one request share these namespaces, so they can talk through Unix sockets and IPC.
  3. PID namespace per task, by the supervisor unshare(CLONE_NEWPID), an init as PID 1
    stops
    Seeing or signalling Faber, host processes and other tasks, including the parallel tasks of the same request. Reaching through /proc/1/root.
    leaves open
    The processes the task starts itself.
  4. Task cgroup per task, joined before anything else memory.max, memory.swap.max 0, pids.max, cpu.max
    stops
    Using more memory or processes than the limit, for the whole process tree. Swapping. More CPU per period than the quota.
    leaves open
    Disk I/O bandwidth: there is no io controller (the task writes only to tmpfs, which is memory). Wall time is the supervisor's job.
  5. Its own /proc and /sys per task unshare(CLONE_NEWNS), proc with subset=pid, empty read-only tmpfs on /sys
    stops
    Reading host-wide kernel state: /proc/meminfo, /proc/cmdline, /proc/sys, sysfs, the cgroup filesystem.
    leaves open
    The per-process files of its own PID namespace. Tools that need host-wide files, such as ps, do not work.
  6. Resource limits per task process setrlimit: CPU, FSIZE, NOFILE, STACK, CORE, MEMLOCK, MSGQUEUE, SIGPENDING, NICE, RTPRIO
    stops
    CPU time past the limit, files over 64 MiB, more than 256 descriptors, stacks over 8 MiB, core dumps, raising priority, real-time scheduling.
    leaves open
    Most are per process. Locked memory, message-queue bytes and pending signals are charged to the request's identity, so they hold for the request as a whole.
  7. User namespace per task, created by the request's identity CLONE_NEWUSER, uid_map and gid_map 65534 → leased ID, setgroups deny
    stops
    Being root anywhere. Inside the task is 65534; outside it is a host UID no other running request has. Per-user kernel limits are charged to that UID.
    leaves open
    Tasks of one request share the identity, and so its per-user limits.
  8. Capabilities per task ambient, bounding, effective, permitted, inheritable: all cleared
    stops
    Privileged operations, even those a user namespace would grant its owner: mounting, chroot, setting the hostname, raw sockets, rewriting the ID maps.
    leaves open
    What any unprivileged process may do.
  9. No new privileges per task PR_SET_NO_NEW_PRIVS
    stops
    Gaining privileges through exec of a setuid or file-capability binary.
    leaves open
    Nothing on its own; it is what lets an unprivileged process install Landlock and seccomp.
  10. Landlock per task, Linux 5.19 and later landlock_restrict_self over every write right
    stops
    Creating, writing, removing, renaming, linking or truncating outside /faber, /tmp and /dev/mqueue. Opening anything else for writing, /proc/self/mem included.
    leaves open
    Reading and executing. On kernels without Landlock ABI 2 it is skipped, and the service says so at startup.
  11. Seccomp per task, last before exec x32 guard, ENOSYS filter, denylist, allowlist
    stops
    Kernel interfaces a task has no use for: namespaces, mounts, modules, BPF, ptrace, keyrings, perf. Under native profiles also processes, threads and sockets.
    leaves open
    Bugs in the system calls it allows.

Your command exec, with every layer above in place

  • Faber and the kernel (trusted)
  • Submitted code (untrusted)
Seccomp comes last because the layers before it need system calls it refuses: mounting and creating namespaces, which every profile kills, and changing identity, which the native profiles do not allow.

The filesystem a task sees covers layers 1 and 5 path by path, cgroups and resource accounting layer 4, and seccomp profiles layer 11.

Entering the user namespace

Section titled “Entering the user namespace”

Layer 7 needs two processes. A process cannot write its own UID map after dropping to an unprivileged ID, and Faber wants the namespace created by the request’s identity rather than by root, so that per-user kernel counters (pending signals, inotify instances, nested namespaces) are charged to the request. The task and its supervisor do it together over two pipes:

  1. Task process: setgroups([]); keep capabilities across the ID change
  2. Task process: setresgid, setresuid to the leased host ID
  3. Task process: unshare(CLONE_NEWUSER): the namespace is owned by the leased ID
  4. Task process to Supervisor: ready, with its PID
  5. Supervisor: Write deny to setgroups, then 65534 <leased ID> 1 to uid_map and gid_map
  6. Supervisor to Task process: mapped
  7. Task process: setgid(65534), setuid(65534), and check them
  8. Task process to Supervisor: done
  9. Supervisor: Read /proc/<pid>/status: all four Uid and Gid values must be the leased ID
  10. Task process: Clear capabilities, no new privileges, Landlock, seccomp, exec
  • Faber and the kernel (trusted)
  • Submitted code (untrusted)
  • Starts or calls
The supervisor kills the task if any step fails or if the IDs it reads back are not exactly the leased identity.

The map has one line, so the task’s 65534 is the only ID that exists for it. Calls such as setuid are allowed under compile_v2, but they can only keep the ID it already has.

What tasks share

Section titled “What tasks share”
  • Faber and the kernel (trusted)
  • Submitted code (untrusted)
The request is the unit of sharing. Do not put work from two parties in one request.

Identities are leased per Faber process. Two Faber services on one kernel hand out the same UIDs unless each has its own SANDBOX_IDENTITY_BASE range.

What no layer stops

Section titled “What no layer stops”
  • Kernel bugs. Every layer is enforced by the kernel the task talks to. A bug in a system call the profile allows can bypass all of them, and the Faber container is privileged, so that is a compromise of the host. See the isolation tier.
  • Shared hardware. CPU caches, memory bandwidth and speculative execution are shared with everything else on the host.
  • Observing the cache. All API key holders share one step cache; a hit tells a request that someone ran the same step before.
  • Memory deduplication. If the host runs KSM, tasks can mark memory mergeable and time the merges. Keep KSM off on Faber hosts.

Where this lives in the code

Section titled “Where this lives in the code”