Monitoring

Health

GET /api/v1/health needs no API key. It answers 200 {"status":"ok"} while the instance accepts work and 503 {"status":"shutting_down"} once shutdown has begun, so load balancers and readiness probes stop routing to it before it exits.

Metrics

GET /api/v1/metrics (with the API key) serves Prometheus text format. A scrape configuration:

scrape_configs:
  - job_name: faber
    metrics_path: /api/v1/metrics
    authorization:
      type: Bearer
      credentials_file: /etc/prometheus/faber-api-key
    static_configs:
      - targets: ["faber-1:3000", "faber-2:3000"]

What to watch:

Signal	Meaning	Act when
faber_execute_responses_total{status="503"}	Requests shed for lack of slots	Sustained growth: raise MAX_CONCURRENCY (and memory) or add instances
faber_execute_responses_total{status="500"}	Sandbox or runtime failure	Any occurrence: find the log line with the same request ID
faber_execute_responses_total{status="504"}	A request hit the hard deadline backstop	Repeated: the host is overloaded or a jailer is stuck
faber_tasks_total{outcome="infrastructure_failure"}	A task could not be set up	Any occurrence
faber_tasks_total{outcome="ancestor_out_of_memory"}	A task was killed by a limit above its own	Growth: memory is undersized for MAX_CONCURRENCY (sizing)
faber_tasks_total{outcome="policy_violation"}	A task made a denied syscall	Spikes: someone is probing, or a toolchain needs a syscall the profile denies
faber_execution_slots_in_use vs faber_execution_slots	Slots held now, and configured	Stuck at the maximum with no traffic: see the runbook
faber_execute_duration_seconds	Request latency	Shifts after a deploy or a load change
Step cache metrics (faber_cache_*)	Lookups, hits, stores, shared-tier (Redis) errors, bytes in the local tier	Redis errors above zero; a hit rate that drops to zero after a configuration change

A falling cache hit rate right after a deploy is expected when the image or the limits changed: they are part of every cache key.

Logs

Faber logs to stdout. Set LOG_FORMAT=json for one JSON object per line, and RUST_LOG to change the level (faber=debug,faber_api=debug,faber_runtime=debug for troubleshooting).

Every request is logged with a request_id, the method, path, status and duration. The same ID is returned to the client in the x-request-id header, so a client can hand you the ID of a failed request. Request bodies are never logged.

Suggested alerts

• Any 500 or infrastructure_failure in the last 15 minutes.
• 503 rate above a few percent of requests for 10 minutes.
• ancestor_out_of_memory above zero.
• faber_execution_slots_in_use == faber_execution_slots for 10 minutes while request rate is low.
• /health failing on an instance that is not being deployed.