AI DevOps Incident Investigator

When production breaks, on-call engineers spend the first hour gathering evidence across logs, metrics, databases, deploys and Git history. AI agents promise to speed this up, but most of them produce confident answers with no way to tell whether they are right, and some will happily take actions on a guess.

This project set out to build an agent an engineer could actually trust: one that shows its reasoning, proves every claim with real tool output, can conclude "this isn't our bug" when a third party is at fault, never writes anything without human approval, and is measured against known answers rather than demos.

The agent is a LangGraph graph with triage, hypothesis generation, evidence gathering, evaluation and report nodes. One model call judges the last tool result and chooses the next step, keeping a full investigation to 6–9 calls. Tools are exposed through an MCP server and are read-only: log search with error-signature grouping, metrics with change-point detection, SQL guarded by sqlglot (SELECT-only, row-capped, statement timeout, read-only connection), Git history via GitPython and deploy events.

Every claim in the report must cite a source reference a tool actually returned; a validator removes invented citations and downgrades unsupported "confirmed" hypotheses. A cause is confirmed only with evidence from both the symptom side and the change side, and if the model tries to stop early, the loop fetches the missing side itself. Tool output is wrapped as untrusted data with secret and PII redaction. Before its only write action, opening a GitHub issue, the graph interrupts; a signed-in reviewer can edit, approve or reject, and the checkpointed state lets a run resume hours later in another process.

Around the agent is a multi-tenant SaaS on FastAPI and SQLAlchemy 2.0: organisations as the data boundary (cross-tenant requests return 404), per-organisation roles, plan limits enforced with 402, usage metering, hashed API keys for alert ingestion and Stripe Checkout in test mode. A Next.js 15 UI streams each step over SSE and resumes from a sequence number after a refresh. Every model call and tool result is recorded to a cassette, so investigations, the demo and CI evals replay at zero cost. A purpose-built FastAPI checkout service with seeded Git history and a fault injector provides six scenarios with ground truth: bad deploy, config change, rotated secret, N+1 query, feature flag and third-party outage. OpenTelemetry spans carry tokens, cost and latency for every node and call.