Skip to content
Faber Faber

The filesystem a task sees

The jailer builds one root directory per request, under /tmp/faber/ in the Faber container, and makes it the root of the request’s mount namespace with pivot_root. Each task then swaps in its own /proc and /sys. The sandbox page lists the paths in short; this page has every mount and the reasons behind it.

The mount tree

Section titled “The mount tree”
  • / ronosuid,nodev,noexec The request's root, /tmp/faber/<12 random characters> bound onto itself and remounted read-only once setup is done. It holds only mount points, directories and links.
    • /bin /lib /lib64 /usr ro bindnosuid,nodev The image's toolchains, bound recursively.
    • /etc Only what the read-only paths put there: no resolver configuration, no hosts file.
      • alternatives ro bind The links behind cc, java and the like.
      • ld.so.cache ro bindfile
      • passwd group ro bindfile So that 65534 has a name.
    • /faber tmpfs128 MiB16384 inodesmode 0777nodev,nosuid The workspace. Every task starts here; every task of the request shares it; the step cache captures it.
    • /tmp tmpfs128 MiB16384 inodesmode 1777nodev,nosuid Shared by the request's tasks. Not captured by the step cache.
    • /dev
      • null zero full random urandom bind of the container's device
      • mqueue mqueuenosuid,nodev,noexec POSIX message queues of the request's IPC namespace.
      • fd stdin stdout stderr symlinks To /proc/self/fd. They work because each task owns its stdio pipes.
      • shm symlink to /tmp POSIX shared memory lands in /tmp and counts against its limits.
    • /proc procsubset=pidnosuid,nodev,noexec The task's own PID namespace, per-process directories only, over an empty read-only tmpfs.
    • /sys empty tmpfsrosize=4kmode 0555
  • Read-only, or Faber's
  • Writable by the task
  • A directory or link on the read-only root
With the default SANDBOX_READONLY_PATHS. Read-only paths that do not exist in the image are skipped. The task can create and change files only in the three ember mounts.

The hostname is faber. There is no /home: give toolchains that want a home or a cache directory HOME=/tmp or a path in the workspace through env.

Two views of /proc and /sys

Section titled “Two views of /proc and /sys”

The jailer and the task live in different mount namespaces, and they see different things at the same two paths:

  • Faber and the kernel (trusted)
  • Submitted code (untrusted)
The task joins its cgroup by writing to cgroup.procs while it can still see the cgroup filesystem, then hides it before anything else runs.

subset=pid (Linux 5.8) leaves only the per-process directories, so files such as /proc/cmdline, /proc/meminfo, /proc/stat, /proc/interrupts and everything under /proc/sys do not exist for the task.

Sizes and inode limits

Section titled “Sizes and inode limits”

/faber and /tmp are each a tmpfs of 128 MiB and 16384 inodes. Both numbers are fixed, and both are part of every step cache key.

  • Bytes. tmpfs keeps files in memory. A page is charged to the cgroup of the task that wrote it, and to the request cgroup once that task’s cgroup is gone, so the request’s memory allowance includes the workspace. Writing past the size fails with ENOSPC.
  • Inodes. Without a limit, tmpfs allows an inode per two pages of RAM, and empty files use no pages at all, so size= alone would not stop a task from creating millions of them. Each inode, with its directory entry, costs kernel memory, which Faber counts as 1 KiB. Past 16384 files, directories and links, creating one more fails with ENOSPC.
  • What it adds up to. 128 MiB + 128 MiB + 32768 × 1 KiB = 288 MiB per request, the workspace allowance in cgroup sizing.
  • One file may not exceed 64 MiB (RLIMIT_FSIZE), and uploaded files a request references may not add up to more than the workspace’s 128 MiB.

How submitted files are written

Section titled “How submitted files are written”

The supervisor writes a task’s files as root, after earlier steps and possibly parallel tasks have had the workspace to themselves. A path such as src/main.c could meet a symlink, a FIFO or a directory swapped in at the last moment. So every path is resolved one component at a time, relative to a directory that is already open and verified:

  • Faber and the kernel (trusted)
  • Starts or calls
openat2 with RESOLVE_BENEATH, RESOLVE_NO_SYMLINKS, RESOLVE_NO_MAGICLINKS and RESOLVE_NO_XDEV refuses any path that would leave the open directory, follow a link or cross into another mount.

O_NONBLOCK keeps a FIFO left in the workspace from blocking the open, and the regular-file check then refuses it. The content comes from the request for inline files, and from the job memfd, read by offset, for uploaded ones. The jailer restores a cached workspace before the first step with the same routine, so a restored file is as safe as a submitted one.

Files the task creates itself belong to its identity because the task is that identity. Faber does not change modes or owners afterwards.

Where this lives in the code

Section titled “Where this lives in the code”