Skip to content

Sizing

Faber admits at most MAX_CONCURRENCY tasks at a time and refuses the rest with 503 and Retry-After; it never queues. So its peak use is bounded and can be computed from the configuration.

Sandboxes can use at most:

(MEMORY_MAX + 64 MiB + 2 × REQUEST_OUTPUT_LIMIT_BYTES + 288 MiB) × MAX_CONCURRENCY

per task slot: the task’s own limit, the supervisors’ fixed allowance, room for the request’s output, and the 288 MiB the workspace and /tmp tmpfs can hold (128 MiB each, plus 1 KiB for each of their 16384 inodes). Faber caps its own cgroup at this figure.

The Faber process itself needs, on top of that:

ItemSize
The serviceabout 256 MiB
File store, memory backendup to FABER_STORE_MAX_TOTAL_BYTES (512 MiB)
File store, hybrid backendup to FABER_STORE_MAX_MEMORY_SIZE (100 MiB)
Step cache, local tierup to CACHE_MAX_BYTES (64 MiB), when enabled

With the defaults:

sandboxes: (256 + 64 + 32 + 288) MiB × 10 = 6.25 GiB
service: 256 MiB + 512 MiB store + 64 MiB cache ≈ 0.8 GiB
total: ≈ 7.1 GiB → container limit 8 GiB, host with 8 GiB free

Set the container’s memory limit (--memory, mem_limit, or the pod’s limit) above the total and equal to its swap limit, and make sure the host has that much free. The container limit is a backstop: if it is lower than the total, a busy instance hits it first, and tasks die with ancestor_out_of_memory instead of their own limits.

(PIDS_MAX + 3) × MAX_CONCURRENCY

(64 + 3) × 10 = 670 with the defaults. The container’s --pids-limit of 4096 leaves room for the service’s own threads.

Each task gets CPU_MAX of bandwidth: half a core by default. Fully loaded, an instance uses MAX_CONCURRENCY × quota / period cores: 5 with the defaults. Give the host at least that many, plus one for the service and the kernel. Tasks are throttled rather than killed when they exceed their bandwidth; cpu_throttled_usec in task stats shows it.

  1. Decide the per-task limits your workloads need (MEMORY_MAX, CPU_MAX, PIDS_MAX).
  2. Divide the host’s spare memory, after the service’s share, by the per-slot figure above.
  3. Check that the CPU figure fits the host’s cores.
  4. Keep MAX_PARALLEL_TASKS at or below the result.

Raise the container memory and pid limits together with MAX_CONCURRENCY.

Watch faber_execute_responses_total{status="503"}. Sustained growth means requests are being shed for lack of slots: raise MAX_CONCURRENCY (with memory) or add instances. Instances share cached steps through Redis; see running a cluster.