Compile and test a submission

This guide builds the core of an automated grader: take a C++ submission, compile it, run it against test cases, and decide a verdict for each test. The same pattern works for any compiled language; interpreted languages skip the compile step.

The request

{
  "stop_on_failure": true,
  "steps": [
    {
      "cmd": "g++",
      "args": ["-O2", "-std=c++17", "-o", "solution", "solution.cpp"],
      "files": { "solution.cpp": "..." },
      "cache": true
    },
    [
      { "cmd": "./solution", "stdin": "3\n1 2 3\n", "sandbox_profile": "native_v2" },
      { "cmd": "./solution", "stdin": "1\n42\n", "sandbox_profile": "native_v2" },
      { "cmd": "./solution", "stdin": "0\n", "sandbox_profile": "native_v2" }
    ]
  ]
}

• stop_on_failure: a compile error skips the tests.
• cache: true on the compile step: if the step cache is enabled, the next request with the same source skips compilation and starts with solution already in the workspace. Rejudging a submission or running a second batch of tests becomes cheap.
• native_v2 for the tests: the submission cannot start processes, threads or sockets. If your language or the problem needs threads, use compile_v2; see sandbox profiles.
• No cache on the tests: they always run, so their stats are fresh.

Turning results into verdicts

Faber reports what happened; the verdict is your policy. A mapping that most judges use:

Result	Verdict
Compile step exit_code ≠ 0	Compilation error (show its stderr)
outcome: timed_out	Time limit exceeded
outcome: out_of_memory	Memory limit exceeded
outcome: output_limit	Output limit exceeded
outcome: policy_violation	Restricted function (or runtime error)
outcome: signaled, or exited with exit_code ≠ 0	Runtime error
outcome: exited, exit_code 0, stdout matches	Accepted
outcome: exited, exit_code 0, stdout differs	Wrong answer
ancestor_out_of_memory, infrastructure_failure, not_started from the deadline	Judge error: retry, do not blame the submission

Compare outputs leniently if your problems allow it (trailing whitespace, final newline). Use stats.cpu_usage_usec for reporting run time; wall time varies with throttling and host load.

In code

The pipeline needs nothing beyond fetch. A TypeScript sketch (the SDKs provide typed models and helpers for the same thing):

type TestCase = { input: string; expected: string };
type Verdict = "AC" | "WA" | "TLE" | "MLE" | "OLE" | "RE" | "CE" | "JUDGE_ERROR";

type Stats = { outcome: string; cpu_usage_usec: number; memory_peak_bytes: number };
type TaskResult =
  | { stdout: string; stderr: string; exit_code: number; cached?: boolean; stats: Stats }
  | { error: string; cached?: boolean; stats: Stats };

const FABER_URL = process.env.FABER_URL ?? "http://localhost:3000/api/v1";
const API_KEY = process.env.FABER_API_KEY ?? "";

export async function judge(source: string, tests: TestCase[]): Promise<Verdict[]> {
  const response = await fetch(`${FABER_URL}/execute`, {
    method: "POST",
    headers: { Authorization: `Bearer ${API_KEY}`, "Content-Type": "application/json" },
    body: JSON.stringify({
      stop_on_failure: true,
      steps: [
        {
          cmd: "g++",
          args: ["-O2", "-std=c++17", "-o", "solution", "solution.cpp"],
          files: { "solution.cpp": source },
          cache: true,
        },
        tests.map((test) => ({ cmd: "./solution", stdin: test.input, sandbox_profile: "native_v2" })),
      ],
    }),
  });
  if (!response.ok) {
    const { code, error } = await response.json();
    throw new Error(`Faber refused the request (${response.status} ${code}): ${error}`);
  }

  const { steps } = (await response.json()) as { steps: [TaskResult, TaskResult[]] };
  const [compile, runs] = steps;
  if (!("exit_code" in compile) || compile.exit_code !== 0) return tests.map(() => "CE");

  return runs.map((run, index) => verdict(run, tests[index].expected));
}

function verdict(run: TaskResult, expected: string): Verdict {
  if (!("exit_code" in run)) return "JUDGE_ERROR";
  switch (run.stats.outcome) {
    case "timed_out":
      return "TLE";
    case "out_of_memory":
      return "MLE";
    case "output_limit":
      return "OLE";
    case "exited":
      if (run.exit_code !== 0) return "RE";
      return run.stdout.trimEnd() === expected.trimEnd() ? "AC" : "WA";
    case "signaled":
    case "policy_violation":
      return "RE";
    default:
      return "JUDGE_ERROR";
  }
}

The parallel step can hold at most MAX_PARALLEL_TASKS tests (8 by default). For more, see multiple test cases.

Limits are per service

Time and memory limits are set by the operator for every task (WALL_TIMEOUT_MS, MEMORY_MAX, …); a request cannot choose them. If problems need different limits, run separate Faber deployments (for example, one per limit class) and route each problem to the right one. GET /info reports the limits of an instance.

Handling 503

When every task slot is busy, Faber answers 503 with capacity_exhausted and a Retry-After header instead of queueing. Retry after that many seconds, with backoff. The SDKs do this for you. Do not retry 4xx responses: the request itself is wrong.