# faultkit

> Reliability testing for AI agents and AI applications. AI evals test quality; faultkit tests resilience.

faultkit is an open-source CLI for reliability testing of AI agents and AI applications. It is a single Go binary that wraps any command and injects realistic production failures into its traffic, so you can run your error-handling paths on purpose instead of discovering them in production. Your evals tell you whether the answer was correct; faultkit tells you whether your application survives failure. It wraps the target process, injects the fault, and captures the exit code. No daemon, no sidecar, no cluster. faultkit injects at the SDK/API boundary, where agent failures actually live, and is built for local development and CI. Licensed under Apache-2.0.

Why it matters now: AI apps depend on many external systems (LLM providers, vector DBs, search and embedding APIs, tool subprocesses), agent workflows are probabilistic rather than deterministic, and unit tests and mocks check the happy path rather than how your app behaves when a real dependency degrades.

Two execution modes: an HTTPS proxy (primary; macOS and Linux, no privileges) that synthesizes vendor-accurate responses for matched requests, and eBPF kprobes (advanced mode; Linux 5.8+, x86-64) that rewrite syscall return values for the target's PID tree.

## Documentation

- [faultkit homepage and docs](https://faultkit.dev/): Landing page with the positioning, why it matters now, the two execution modes, scenarios, quickstart, exit codes, and a full documentation section.
- [Full documentation (Markdown)](https://faultkit.dev/llms-full.txt): Complete, self-contained documentation as clean Markdown. What it is, why now, how it works, scenarios, exit codes, install, quickstart, YAML authoring, CI usage, troubleshooting.
- [GitHub repository](https://github.com/faultkit/faultkit): Source code, releases, and issues (Apache-2.0).

## Scenarios

Twelve built-ins. The LLM scenarios fire against every provider faultkit knows (OpenAI, Anthropic); narrow to one with `--provider`, or point SDK clients that ignore HTTPS_PROXY at faultkit with `--base-url`.

- llm-api-degraded (proxy): 429/503 from OpenAI + Anthropic with realistic Retry-After headers.
- malformed-json-response (proxy): LLM response body replaced with syntactically invalid JSON.
- malformed-tool-use (proxy): tool call with malformed or schema-violating arguments.
- max-tokens-truncation (proxy): a truncated 200 (finish_reason length / stop_reason max_tokens) treated as complete.
- llm-streaming-cutoff (proxy): SSE chat completion drops mid-token without a [DONE].
- anthropic-overloaded (proxy, Anthropic): HTTP 529 overloaded_error under load (no OpenAI equivalent).
- anthropic-stream-error (proxy, Anthropic): SSE stream emits an error event mid-stream, no message_stop.
- anthropic-tool-use-cutoff (proxy, Anthropic): a tool_use block truncated by max_tokens.
- anthropic-refusal (proxy, Anthropic): 200 with stop_reason refusal — the model declined.
- anthropic-request-too-large (proxy, Anthropic): HTTP 413 request_too_large for an oversized request.
- flaky-network (eBPF, Linux 5.8+): ECONNRESET on TCP recvmsg / recvfrom.
- tool-permission-denied (eBPF, Linux 5.8+): EACCES on openat.
- Custom scenarios: author your own in YAML and pass them with `--config scenario.yaml`.

## Install

- Homebrew (macOS / Linux): `brew install faultkit/tap/faultkit`
- Arch (AUR): `yay -S faultkit-bin`
- Go: `go install github.com/faultkit/faultkit/cmd/faultkit@latest`
- Or download a release binary from [GitHub Releases](https://github.com/faultkit/faultkit/releases).

## Links

- [Website](https://faultkit.dev/): Project homepage.
- [GitHub](https://github.com/faultkit/faultkit): Source and releases.
- [License: Apache-2.0](https://www.apache.org/licenses/LICENSE-2.0): Open source, perpetual.