faultkitfaultkit

#Installation

faultkit ships as a single static Go binary. Install it through a package manager, the Go toolchain, or grab a release binary.

$ brew install faultkit/tap/faultkit          # macOS / Linux (Homebrew)
$ yay -S faultkit-bin                          # Arch (AUR)
$ go install github.com/faultkit/faultkit/cmd/faultkit@latest

The proxy mode runs on macOS and Linux with no special privileges. eBPF mode is Linux-only and has additional requirements (see below). Run faultkit check after install to see which modes are available on your host.

#faultkit run: usage & flags

The core command wraps a target process, injects a fault, and exits with a status code that reflects what happened. Everything after -- is the command faultkit runs.

$ faultkit run [flags] -- <command> [args…]

• --scenario <name>  Run a built-in scenario by name (see Scenarios).
• --config <file>  Load a scenario you authored in YAML instead of a built-in.
• --provider <id>  Limit a fixture-driven LLM scenario to one provider (e.g. openai, anthropic); the default fans out across all of them.
• --base-url  Point the SDK at faultkit via OPENAI_BASE_URL / ANTHROPIC_BASE_URL instead of HTTPS_PROXY — for clients (Node fetch, filtered subprocesses) that ignore proxy env.
• --mode <auto|proxy|ebpf>  Force an injection mechanism. Default auto picks by scenario.
• --report <file.json>  Write a JSON report artifact in addition to the terminal summary.
• --verbose  Log every matched request and fired fault for debugging.

By default faultkit prints a terminal summary: which faults fired, how the target behaved, and the final exit code.

#The proxy mechanism

For HTTP(S) scenarios (LLM providers, RAG / vector DBs, gateways), faultkit spins up a localhost proxy backed by a per-run ephemeral CA. It injects HTTPS_PROXY and the CA into the target's environment so the target trusts it through standard env vars; no system keychain changes, nothing left behind after the run.

Each request is matched by a host glob and path glob. For requests that match and fire, faultkit synthesizes a vendor-accurate response: correct status, body shape, and streaming behavior. Everything else passes straight through to the real upstream. The target sees a real HTTP 429 over a real TLS connection, because that is exactly what it receives. The proxy serves HTTP/1.1.

#The eBPF mechanism & requirements

For syscall-level faults, faultkit loads small BPF programs that hook __x64_sys_* kprobes (for example recvmsg, recvfrom, and openat) and use bpf_override_return to rewrite the syscall return value for processes in the target's PID tree. All decision logic (probability, matching) lives in Go; the BPF program is a dumb policy enforcer reading an LRU map populated by userspace. The target gets a real ECONNRESET or EACCES straight from the kernel.

Requirements

• Linux 5.8+, x86-64.
• CAP_BPF (or root).
• A kernel built with CONFIG_BPF_KPROBE_OVERRIDE, which most distro kernels have.

#Authoring scenarios in YAML

Built-ins cover common cases, but you can describe your own with host and path globs and a fault that fires with a given probability. Pass it with --config.

name: stale-vector-search
experiments:
  - name: shuffled-results
    match:
      host: '*.pinecone.io'
      path: /query*
    fault:
      http_status: 200
      response_body: '{"matches":[]}'
    probability: 0.5      # fire on half of matched requests

$ faultkit run --config scenario.yaml -- pytest tests/test_rag.py

The example above is the raw fault + match form for any host. Built-in LLM scenarios instead use a higher-level failure: mode with an optional provider:, which resolves to a per-provider response fixture — so one scenario covers OpenAI, Anthropic, and any provider added later.

#CI usage

Because faultkit is single-shot and communicates through exit codes, it drops straight into CI. Run it as a step and let the runner branch on the code: exit 1 means your code broke under the fault; exit 3 means no matched traffic ever hit the proxy, which usually means the test didn't call the API you targeted.

- name: Fault injection
  run: |
    faultkit run --scenario llm-api-degraded --report fault.json -- pytest tests/
    code=$?
    if [ $code -eq 3 ]; then
      echo '::warning::no matched traffic, fault never fired'
    fi
    exit $code

#Troubleshooting

Start with faultkit check, which reports which modes are available on the host, including whether the kernel supports eBPF kprobe override. If a run exits 3, the target never hit matched traffic: confirm your host/path globs against the real requests with --verbose, which logs every matched request and fired fault.

• Exit 3, expected a fault: globs didn't match; check hostnames and paths with --verbose.
• eBPF unavailable: run faultkit check; you likely lack CAP_BPF or CONFIG_BPF_KPROBE_OVERRIDE.
• TLS errors in the target: the target must inherit the env faultkit injects; don't strip HTTPS_PROXY in a wrapper script.