Skip to content
Faber Faber

Outcomes and failure handling

Failures happen at two levels. A task that crashes, times out or breaks a rule gets an outcome in its result, and the request still answers 200. Only when Faber cannot accept a request, or cannot run it at all, does the request get an HTTP error. Resource limits lists the outcomes with their fields, and errors the error codes; this page shows how Faber arrives at each.

How a task ends

Section titled “How a task ends”

The supervisor decides a task’s outcome after the task and everything it started have exited, from what it saw itself and from the task cgroup’s event counters. When several causes apply, the first in this order wins:

  1. stop_on_failure is set and an earlier task did not exit with 0?
    yes
    not_started "Skipped because an earlier step failed"
  2. The request's deadline passed before the step could start?
    yes
    not_started "Not started: the overall execution deadline was reached"
  3. Setup failed before exec: the task cgroup, its files, the user namespace, the security setup, working_dir, a NUL byte?
    yes
    infrastructure_failure the result has error instead of output
  4. The wall timeout fired?
    yes
    timed_out cgroup.kill on the whole task
  5. stdout or stderr reached its limit?
    yes
    output_limit cgroup.kill; a truncation flag is set
  6. The OOM killer killed a process, and the task's own memory.max was reached?
    yes
    out_of_memory oom_kill_count > 0
  7. The OOM killer killed a process?
    yes
    ancestor_out_of_memory a limit above the task ran out
  8. pids.max refused a fork at least once?
    yes
    pids_limit the program may still have exited normally
  9. The task's process died of SIGSYS?
    yes
    policy_violation seccomp; exit_code 159
  10. It died of another signal?
    yes
    signaled exit_code is 128 + the signal
  11. exited exit_code is the process's status
  • Faber or the kernel killed the task
“The task’s process” is the one Faber started; how its children ended matters only through what it does about them. A command that cannot be executed at all (not found, not executable) is not a setup failure: the result is exited with exit_code 127 and a line starting faber: on stderr.

Some consequences of the order:

  • A task killed at its wall timeout reports timed_out even if it was also close to its memory limit; the timeout is what ended it.
  • pids_limit comes before the signals. A fork bomb whose processes then crash is reported as pids_limit, which is the more useful explanation.
  • A non-zero exit code alone never changes the outcome: a failing test is exited with exit_code 1.

Two shapes of result

Section titled “Two shapes of result”
OutcomeResult shapeWhat it holds
not_started, infrastructure_failurefailed: error and statsWhy the task did not run. Statistics are empty, except execution_time_ms for a setup failure
everything elsecompleted: stdout, stderr, exit_code, statsThe output kept within the limits, the exit status (128 + signal for a signal), and the cgroup’s statistics

A result replayed from the step cache is always a completed exited result with exit code 0, because nothing else is ever stored. It carries the statistics of the run that produced it and cached: true.

What a task’s failure does to the request

Section titled “What a task’s failure does to the request”

Nothing, by default. The request still answers 200, and later steps still run in the same sandbox. With stop_on_failure, the first step with a task that did not complete with exit code 0 is reported in full, and every later step is not_started. When the request’s deadline passes, the running step’s tasks reach their wall timeout (clipped to the time left) and every later step is not_started, so a slow request returns what it finished.

What the API returns

Section titled “What the API returns”
  • POST /api/v1/execute
    • Refused before anything runs
      • 400 invalid_request Malformed JSON, no steps, an empty parallel step, a bad file_id or inline content, a body slower than REQUEST_BODY_TIMEOUT_MS.
      • 401 unauthorized No key, or a key that is not configured.
      • 404 file_not_found A task references an upload this instance does not have.
      • 413 payload_too_large The body is over EXECUTE_BODY_LIMIT_BYTES.
      • 422 limits_exceeded, profile_not_allowed Too many steps, too wide a step, too many uploaded bytes, or a profile outside ALLOWED_SANDBOX_PROFILES.
      • 429 rate_limited The key's bucket is empty; Retry-After says when it refills.
    • Not run
      • 503 capacity_exhausted Not enough free task slots. Retry-After: 1.
      • 503 shutting_down The instance is draining. Retry-After: 1.
      • 503 cancelled The client went away; the sandbox was killed.
    • Started, but no results
      • 500 sandbox_setup_failed The sandbox could not be built, a cached workspace could not be restored, or the jailer could not read its job or join its cgroup.
      • 500 internal_error Anything else, such as a request cgroup that could not be created or jailer output that could not be read.
      • 504 deadline_exceeded The jailer was still running two seconds after OVERALL_TIMEOUT_MS and was killed.
    • 200 One result per task, in the shape of the steps, each with its outcome; plus the cache summary.
  • The request ran
  • No results
Branch on the code, not only on the status: 503 has three meanings and 500 two. Task outcomes never appear here.

The order of the checks is the order of the request lifecycle: authentication and rate limiting first, then the body, validation, uploaded files, the cache, and the slots. Everything up to the slots is decided before a sandbox exists, so a refused request costs Faber almost nothing.

Counting outcomes

Section titled “Counting outcomes”

GET /api/v1/metrics counts both levels: faber_execute_responses_total by HTTP status and faber_tasks_total by outcome. A rise in ancestor_out_of_memory or any infrastructure_failure points at the deployment rather than the workload; monitoring and the runbook say what to do about each.

Where this lives in the code

Section titled “Where this lives in the code”