Accuracy Evals
Build a test suite of questions with expected answers, run it against live Orion, and track a pass rate to measure and improve answer accuracy.
When to use it — Before rolling Orion out to end users, and any time you change skills or corrections and want to confirm answers actually improved.
Where to find it — Context Studio → Accuracy Evals tab.
Who can use it — Luminary and Weaver.
Orion can answer in many ways, and "it sounded right" is not a measurement. Accuracy Evals turns that into a number: you write the questions Orion must get right, record what a correct answer looks like, and run the suite to see how many it passes today.
The result is a single pass rate plus a per-question breakdown of what failed. Failures are the point — they tell you exactly where Orion needs more context, so you can fix it and re-run to prove the fix worked.
Overview
The Accuracy Evals tab has three parts, top to bottom:
- Stat cards and Run evals — the headline numbers from your most recent run (pass rate, passed, failed, last run) and the Run evals button.
- Run history — a list of recent runs, each with a date, optional domain badge, a colored bar, and the pass rate. Up to 8 runs are shown.
- Eval questions — your test suite: every active question, its optional domain badge, and its expected answer.
When you click Run evals, Celiq scores all active questions against live Orion and records the result as a new run. The pass rate is color-coded so you can read health at a glance: green at 80% or above, amber at 60–79%, red below 60%.
When to use it
- Before launch — confirm Orion clears your accuracy bar before end users rely on it.
- After changing context — whenever you apply a correction or update a domain skill, re-run to verify the change helped and did not regress other answers.
- As a regression guard — re-run periodically so you catch accuracy drift over time, not after users complain.
- To triage a complaint — when someone flags a bad answer, capture it as an eval question (see below) so the failure is reproducible and tracked.
Concepts
| Term | What it means |
|---|---|
| Eval question | A question Orion should answer correctly, optionally paired with an expected answer (the correct answer, or the key facts the answer must include) and a domain. |
| Domain | An optional tag that scopes a question to one business area. Domain-tagged questions can be run and diagnosed on their own. |
| Run | One execution of the suite against live Orion. Each run records totals, passed/failed counts, and a pass rate. |
| Pass rate | The percentage of questions in a run that Orion answered correctly. Your objective accuracy measure. |
| The improvement loop | Failed evals → corrections → skill updates → re-run. The repeatable cycle for raising your pass rate. |
Getting started
Prerequisites- You are a Luminary or Weaver in the workspace.
- At least one eval question exists. The Run evals button stays disabled while the suite is empty.
- Domains in the question form come from your configured skills, so set up domain skills first if you want to tag questions by domain.
Go to Context Studio, then select the Accuracy Evals tab. The header shows how many questions and runs you have recorded, for example "12 questions · 5 runs recorded".
Step-by-step
Add a question manually
In the Eval questions section, click + Add question. Fill in the Question field with something Orion should answer correctly, add the Expected answer (the correct answer, or the key facts it must include), optionally pick a domain from the dropdown, and click Save. Leave the domain as All domains if it is not domain-specific.
Or capture a bad answer as a question
When Orion gives a wrong answer in a chat, give it a thumbs-down and choose Create eval. The question and answer are carried into the suite as a new eval question, so a real failure becomes a tracked test.
Run the suite
Click Run evals in the top-right. The button reads "Running…" while the run is in progress. Every active question is scored against live Orion and the run is recorded.
Read the result
When the run completes, the stat cards update with the new Pass rate, Passed, Failed, and Last run date, and a new bar appears at the top of Run history.
Find the failures
Review which questions failed — they tell you exactly where Orion needs help. Each failed question points you at the context that is missing or wrong.
Fix and re-run
Apply a correction or update a domain skill to close the gap, then click Run evals again. Compare the new bar against the previous one in Run history to confirm the pass rate went up.
Examples
Suppose finance users complain that Orion gets the fiscal calendar wrong. You add a targeted eval question so the problem is reproducible and measurable.
Question: When does our fiscal year start?
Expected answer: The fiscal year starts in February. Q1 is Feb–Apr.
Domain: finance
You run the suite and the question fails — Orion still assumes a January start. You record the correct behavior as a correction (a fact: "fiscal year starts February"), then re-run.
Jun 27 finance ████████████████████░░ 91% 10/11
Jun 27 finance ██████████████░░░░░░░░ 64% 7/11
The pass rate jumps from 64% to 91% and the finance question now passes — confirmation that the correction worked, not a guess.
Best practices
- Aim for above 80% before rolling Orion out to end users. The pass rate turns green at 80%, which is a useful target line.
- Write a real expected answer for each question. Listing the key facts the answer must include makes a pass unambiguous.
- Group questions by domain. Domain-specific runs are faster to diagnose because a failure is already scoped to one area.
- Re-run after every context change. Treat the suite as a regression guard, not a one-time check.
- Grow the suite from real failures. Thumbs-down → Create eval is the fastest way to turn complaints into coverage.
Tips
The fastest way to grow a meaningful suite is to capture failures as they happen: thumbs-down any wrong Orion answer and choose Create eval. Over time your suite becomes a record of exactly the questions your users care about — and the improvement loop (failed evals → corrections → skill updates → re-run) keeps the pass rate honest.
Common mistakes
- Empty suite. With no questions, Run evals stays disabled — add at least one question first.
- No expected answer. A question with no expected answer is hard to score consistently; record the correct answer or its key facts.
- Running once and stopping. A single pass rate is a snapshot. Re-run after each change so you can see whether accuracy is improving or drifting.
- Skipping the fix step. Failed evals are only useful if you act on them — apply a correction or update a skill, then re-run to verify.
Troubleshooting
| Symptom | Cause | Fix |
|---|---|---|
| Run evals button is greyed out | The suite has no questions (questions.length === 0) | Add at least one question with + Add question, or capture one via thumbs-down → Create eval. |
| Saving a question does nothing | The Question field is empty — a blank question is ignored | Type a question before clicking Save. |
| No stat cards or run history shown | No runs recorded yet for this workspace | Click Run evals once; the cards and history bars appear after the first run completes. |
| The domain dropdown is empty | No domain skills are configured, so there are no domains to choose | Set up domain skills, or leave the question as All domains. |
| Results do not update right after running | The run is asynchronous; the view refreshes a moment after you click | Wait for the button to leave the "Running…" state — the cards and a new history bar update on the refresh. |