Sizing
Faber admits at most MAX_CONCURRENCY tasks at a time and refuses the rest
with 503 and Retry-After; it never queues. So its peak use is bounded and
can be computed from the configuration.
Memory
Section titled “Memory”Sandboxes can use at most:
(MEMORY_MAX + 64 MiB + 2 × REQUEST_OUTPUT_LIMIT_BYTES + 288 MiB) × MAX_CONCURRENCYper task slot: the task’s own limit, the supervisors’ fixed allowance, room
for the request’s output, and the 288 MiB the workspace and /tmp tmpfs can
hold (128 MiB each, plus 1 KiB for each of their 16384 inodes). Faber caps its
own cgroup at this figure.
The Faber process itself needs, on top of that:
| Item | Size |
|---|---|
| The service | about 256 MiB |
File store, memory backend | up to FABER_STORE_MAX_TOTAL_BYTES (512 MiB) |
File store, hybrid backend | up to FABER_STORE_MAX_MEMORY_SIZE (100 MiB) |
| Step cache, local tier | up to CACHE_MAX_BYTES (64 MiB), when enabled |
With the defaults:
sandboxes: (256 + 64 + 32 + 288) MiB × 10 = 6.25 GiBservice: 256 MiB + 512 MiB store + 64 MiB cache ≈ 0.8 GiBtotal: ≈ 7.1 GiB → container limit 8 GiB, host with 8 GiB freeSet the container’s memory limit (--memory, mem_limit, or the pod’s
limit) above the total and equal to its swap limit, and make sure the host
has that much free. The container limit is a backstop: if it is lower than
the total, a busy instance hits it first, and tasks die with
ancestor_out_of_memory instead of their own limits.
Processes
Section titled “Processes”(PIDS_MAX + 3) × MAX_CONCURRENCY(64 + 3) × 10 = 670 with the defaults. The container’s --pids-limit of 4096
leaves room for the service’s own threads.
Each task gets CPU_MAX of bandwidth: half a core by default. Fully loaded,
an instance uses MAX_CONCURRENCY × quota / period cores: 5 with the defaults.
Give the host at least that many, plus one for the service and the kernel.
Tasks are throttled rather than killed when they exceed their bandwidth;
cpu_throttled_usec in task stats shows it.
Choosing MAX_CONCURRENCY
Section titled “Choosing MAX_CONCURRENCY”- Decide the per-task limits your workloads need (
MEMORY_MAX,CPU_MAX,PIDS_MAX). - Divide the host’s spare memory, after the service’s share, by the per-slot figure above.
- Check that the CPU figure fits the host’s cores.
- Keep
MAX_PARALLEL_TASKSat or below the result.
Raise the container memory and pid limits together with MAX_CONCURRENCY.
When to add instances
Section titled “When to add instances”Watch faber_execute_responses_total{status="503"}. Sustained growth means
requests are being shed for lack of slots: raise MAX_CONCURRENCY (with
memory) or add instances. Instances share cached steps through Redis; see
running a cluster.