Trust console · Evals
Answer quality is a number, checked nightly.
A 20-question corpus runs against the real /api/answer endpoint — not the libraries underneath it — and every answer is scored on keywords, citations, substance, and latency. Below 50% overall, the run fails loudly.
The corpus
data/evals/test-questions.jsonl — 20 questions across concept, technical, architecture, security, and operations categories, each with expected keywords. The questions probe the system about itself: retrieval mechanics, the privacy model, rate limits, schema choices.
The harness
scripts/evals-runner.ts posts each question to /api/answer over HTTP, so a passing run exercises validation, the prompt guard, retrieval, generation, and citation construction end to end. Results are written to eval-results.json and a human-readable eval-summary.txt.
Latest scores
Recent eval results
No scores are published right now, and that is the honest state: the nightly pipeline has no successful run in its retained history, and production recovery is an open incident tracked in docs/OPERATIONS.md. This panel only ever shows real eval-summary.txt numbers — once the pipeline is healthy again, the latest overall quality, per-category scores, and run date get published here.
Continuous verification
How the nightly run works
- GitHub Actions cron at 2:00 UTC (plus manual dispatch) builds and boots the app.
- The runner scores all 20 questions against the live server.
- Results upload as artifacts with 30-day retention.
- Below the 50% gate, the workflow fails and comments the summary on the latest commit — regressions are loud, not silent.
Run it locally
Run it locally
npm run dev # or point API_URL at any deployment npm run evals # scores all 20 questions, writes eval-results.json
Source of truth: scripts/evals-runner.ts, data/evals/test-questions.jsonl, .github/workflows/nightly-evals.yml