Multiple test cases

One test is one task. How you lay out many tests depends on the limits of the instance. Read them from GET /info instead of hard-coding them:

curl -s http://localhost:3000/api/v1/info -H "Authorization: Bearer $API_KEY" | jq .limits

The ones that matter here:

Limit (/info field)	Default	Effect
max_parallel_tasks	8	Tasks in one parallel step
max_concurrency	10	Task slots on the instance; a request needs as many as its widest step
max_steps	64	Steps in one request
overall_timeout_ms	30000	Deadline for the whole request
request_output_limit_bytes	16 MiB	Output kept across all tasks of a request
execute_body_limit_bytes	1 MiB	Request body, including inline test inputs

Up to max_parallel_tasks tests: one parallel step

{
  "stop_on_failure": true,
  "steps": [
    { "cmd": "g++", "args": ["-O2", "-o", "main", "main.cpp"], "files": { "main.cpp": "..." }, "cache": true },
    [
      { "cmd": "./main", "stdin": "...test 1...", "sandbox_profile": "native_v2" },
      { "cmd": "./main", "stdin": "...test 2...", "sandbox_profile": "native_v2" }
    ]
  ]
}

Tests run at the same time, each with the full per-task limits. Results come back in the same order as the tasks.

More tests: several parallel steps

Add more parallel steps to the same request:

"steps": [
  { "cmd": "g++", "...": "..." },
  [ "...tests 1-8..." ],
  [ "...tests 9-16..." ],
  [ "...tests 17-20..." ]
]

Simple, but everything shares one request’s budget: all steps must finish within overall_timeout_ms, and all output counts against request_output_limit_bytes. Twenty tests that each take two seconds need three waves of about two seconds, which fits; sixty-four slow tests do not.

With stop_on_failure, a test that crashes, times out or exits non-zero skips every later wave; a wrong answer that exits 0 does not. If every test should run regardless, leave it off. A compile error then makes each test fail on its own, with exit code 127 because ./main does not exist.

Many tests: several requests sharing a cached compile

When tests do not fit in one request’s deadline, split them across requests, each with the same compile step marked cache: true:

request 1:  compile (runs, stored)  + tests 1-8
request 2:  compile (cache hit)     + tests 9-16
request 3:  compile (cache hit)     + tests 17-24

With the step cache enabled, only the first request compiles. The others start with the binary already in the workspace, and their responses report "cache": { "hits": 1, ... }. The requests can run concurrently, as long as the instance has slots for them.

Without the cache, each request compiles again, which still works.

A sketch in TypeScript:

type Limits = { max_parallel_tasks: number };

export function chunk<T>(items: T[], size: number): T[][] {
  const chunks: T[][] = [];
  for (let i = 0; i < items.length; i += size) chunks.push(items.slice(i, i + size));
  return chunks;
}

export function requestsFor(source: string, inputs: string[], limits: Limits) {
  const compile = {
    cmd: "g++",
    args: ["-O2", "-o", "main", "main.cpp"],
    files: { "main.cpp": source },
    cache: true,
  };
  return chunk(inputs, limits.max_parallel_tasks).map((batch) => ({
    stop_on_failure: true,
    steps: [compile, batch.map((stdin) => ({ cmd: "./main", stdin, sandbox_profile: "native_v2" }))],
  }));
}

Large inputs

Inline inputs count against execute_body_limit_bytes. For large or shared test data, upload it once and reference it by file_id, then read it from a file or redirect it in a shell:

{
  "cmd": "sh",
  "args": ["-c", "./main < input.txt"],
  "files": { "input.txt": { "file_id": "…" } },
  "sandbox_profile": "compile_v2"
}

The shell needs compile_v2 because it starts a process. Alternatively, have the program read the file directly and keep native_v2.

Large outputs

Each task keeps at most OUTPUT_LIMIT_BYTES per stream (1 MiB by default) and is killed with outcome output_limit beyond it. Across the request, output is capped at request_output_limit_bytes; later tasks get what is left. If the expected outputs are large, compare inside the sandbox instead of shipping the output back:

{
  "cmd": "sh",
  "args": ["-c", "./main < input.txt > out.txt && cmp -s out.txt expected.txt"],
  "files": {
    "input.txt": { "file_id": "…" },
    "expected.txt": { "file_id": "…" }
  }
}

Exit code 0 means the outputs match, 1 that they differ.