Raglemetry – Audit‑First RAG for Incident Response

Deterministic retrieval + audit trace before LLM synthesis, designed for reliable incident response.
Tech Stack: Python Async I/O Docker Git LLM Observability
Source Code

Summary

I built Raglemetry, an audit‑first RAG system for incident responses. Instead of asking an LLM to “figure out what happened,” it first runs deterministic retrievers across Git commits, logs, and database execution telemetry, then writes a clean output/audit_trace.json with raw evidence. Only after that does the LLM act as a context compiler, condensing the audit trail into an actionable diagnostic report with citations.

The goal: reliable, low‑hallucination IR tooling for those 2 a.m. calls, where you need the truth fast, not a confident invented story.

Impact / Why It Matters

  • Reduces hallucination risk – LLM never synthesizes before seeing an uncorrupted audit trace.
  • Faster incident triage – Parallel retrieval bounded by the slowest endpoint, not the sum of all scans.
  • Clear auditability – output/audit_trace.json provides a stable, reviewable record independent of any model.
  • Privacy‑aware – Command‑line parameters and private payloads are scrubbed at the worker layer before any state is recorded or sent to disk/LLM.

What I Built

  • Deterministic Retrievers – Hash‑based lookups for commit SHAs and DDL migrations; Boyer–Moore sublinear pattern scanning for error codes and text in logs.
  • Parallel Worker Architecture – Concurrent workers running with async I/O; total latency is bounded by the slowest individual worker.
  • Audit Trace Layer – Structured output/audit_trace.json capturing raw findings, timestamps, and sources before any synthesis.
  • LLM as Context Compiler – Prompt framing that treats the model as a compressor/summarizer of the audit trail, enforcing explicit source stage citations.
  • Telemetry & Graceful Shutdown – Real‑time audit logging and safe flush on interruption (Ctrl+C), ensuring partial runs still produce usable traces.
  • Dockerized & Local Execution – Flexible setup supporting both containerized deployment (docker-compose) and air-gapped local CLI workflows.

Results & Evidence

Design properties:

  • Retrieval latency: Latency_total = max(T_Git, T_Logs, T_DB) instead of summing all scan times.
  • No vector embeddings over fast‑changing logs; retrieval is index‑free and pattern‑based for instant incident telemetry.
  • Audit trace is written even if execution is interrupted mid‑scan.

Current status:

  • Core retrieval workers and telemetry implemented.
  • Mock incident data (mock/app.log, mock/db_telemetry.json, mock/git_history.json) and validation in place.
  • LLM synthesis layer designed as a “context compiler” with citation support.

Tech & Methods

  • Language: Python, async I/O
  • Retrieval: Hash tables, Boyer–Moore string search, Git/DB/log parsers
  • LLM Integration: Prompt framing, citation‑backed synthesis, pluggable local runtime (e.g., Ollama)
  • Deployment: Docker, docker‑compose, standalone CLI
  • Testing: Unit and integration tests for pattern matching, retrieval, and trace generation

Limits & Possible Next Steps

Validated:

  • Parallel worker design and audit trace generation.
  • Deterministic retrieval patterns on mock incident data.
  • Graceful shutdown and partial trace writes.

Not (yet) validated:

  • Performance and correctness on large, real production codebases and log volumes.
  • Integration with real cloud platforms (e.g., AWS) and diverse observability stacks.
  • Systematic evaluation of hallucination reduction vs. baseline RAG approaches.

Next steps:

  • Connect to real repos, log stores, and DB audit logs.
  • Add structured output modes (e.g., SARIF‑like) for integration with incident tooling.
  • Evaluate different LLMs strictly as compilers over identical audit traces.
incident-response rag llm observability devops data-engineering