Resource limits
Limits are set by the operator through environment variables and apply to
every task. A request cannot raise them. GET /info reports the effective
values, so clients can check before sending work.
Per task
Section titled “Per task”| Resource | Default | Set with | Enforced by |
|---|---|---|---|
| Memory (whole process tree) | 256 MiB | MEMORY_MAX | cgroup memory.max; swap is disabled |
| Processes and threads | 64 | PIDS_MAX | cgroup pids.max |
| CPU bandwidth | half a core (50000 100000) | CPU_MAX | cgroup cpu.max; the task is throttled, not killed |
| CPU time | 5 s | CPU_TIME_LIMIT_SECS | RLIMIT_CPU per process |
| Wall time | 5 s | WALL_TIMEOUT_MS | Supervisor timer; the whole task cgroup is killed |
| Output | 1 MiB per stream | OUTPUT_LIMIT_BYTES | Supervisor; the task is killed when stdout or stderr exceeds it |
| File size | 64 MiB | fixed | RLIMIT_FSIZE |
| Open files | 256 | fixed | RLIMIT_NOFILE |
| Stack | 8 MiB | fixed | RLIMIT_STACK |
| Core dumps | none | fixed | RLIMIT_CORE |
Per request
Section titled “Per request”| Limit | Default | Set with |
|---|---|---|
| Steps | 64 | MAX_STEPS_PER_REQUEST |
| Tasks in one parallel step | 8 | MAX_PARALLEL_TASKS |
| Whole request deadline | 30 s | OVERALL_TIMEOUT_MS |
| Output kept across all tasks | 16 MiB | REQUEST_OUTPUT_LIMIT_BYTES |
| Request body | 1 MiB | EXECUTE_BODY_LIMIT_BYTES |
Workspace /faber | 128 MiB | fixed |
/tmp | 128 MiB | fixed |
Too many steps or too wide a step is rejected up front with 422
(limits_exceeded); a body over the size limit gets 413.
Deadline. Each step’s wall timeout is clipped to whatever is left of the
request deadline. When the deadline passes, the running tasks are killed and
the remaining steps are returned with outcome not_started, alongside the
results of the steps that did finish. If the request still overruns (a stuck
jailer), a hard backstop answers 504 (deadline_exceeded).
Output budget. Output is kept per stream up to OUTPUT_LIMIT_BYTES, and
across the request up to REQUEST_OUTPUT_LIMIT_BYTES: later tasks get what is
left, then nothing. Truncation is reported in stats.stdout_truncated and
stats.stderr_truncated.
Per service
Section titled “Per service”MAX_CONCURRENCY (default 10) is the number of task slots: sandboxed tasks
that may run at once across all requests. A request reserves one slot per task
of its widest step and holds them until its sandbox is gone, even if the client
disconnects. A request that does not fit is refused immediately with 503
(capacity_exhausted) and a Retry-After header; Faber does not queue.
Faber’s own cgroup is capped so that every admitted task can reach its own limit at the same time, plus the supervisors’ allowance. Sizing has the formula.
How limits show up in results
Section titled “How limits show up in results”A task that hits a limit is still a normal result with HTTP 200. Look at
stats.outcome. When several causes apply, the first row that matches wins:
| Outcome | Cause | Details |
|---|---|---|
timed_out | Wall time exceeded | The task cgroup was killed |
output_limit | stdout or stderr exceeded OUTPUT_LIMIT_BYTES | The task was killed; the truncation flag is set |
out_of_memory | The task’s own memory.max | oom_kill_count > 0 |
ancestor_out_of_memory | A memory limit above the task (request, service, container or host) | Usually means the deployment is undersized |
pids_limit | pids.max was reached at least once | pids_limit_hit_count > 0. The program may have handled the failed fork and exited normally; check exit_code |
policy_violation | A denied syscall | termination_signal is 31 (SIGSYS), exit_code 159 |
signaled | Killed by another signal: a crash, abort(), or the CPU time limit | termination_signal holds the signal number, exit_code is 128 + signal |
exited | The process exited by itself | exit_code is its status. A non-zero code is still exited |
not_started | The deadline passed, or an earlier step failed with stop_on_failure | The result has error instead of output |
infrastructure_failure | Faber could not set up the task (for example, working_dir does not exist) | The result has error |
A cmd that cannot be executed (not found, not executable) is reported as
exited with exit code 127 and a line starting with faber: on stderr.
Statistics
Section titled “Statistics”Every result carries stats, measured from the task’s cgroup:
| Field | Meaning |
|---|---|
memory_peak_bytes | Peak memory of the task’s process tree |
cpu_usage_usec | CPU time used, in microseconds |
cpu_nr_throttled, cpu_throttled_usec | How often and how long cpu.max throttled the task |
pids_peak | Most processes and threads alive at once |
execution_time_ms | Wall time from start to exit |
oom_kill_count, pids_limit_hit_count | Limit events seen by the cgroup |
cleanup_succeeded | Whether the task’s cgroup was removed cleanly |
Use cpu_usage_usec rather than execution_time_ms to compare the speed of
submissions: wall time depends on throttling and host load.
Go deeper
Section titled “Go deeper”- Cgroups and resource accounting: the cgroup tree, how slots and aggregate limits are sized, and how OOM kills are told apart.
- Outcomes and failure handling: how Faber decides which outcome a task gets.