Evals: prompt sets and the judge runner
Regression-test that Orion answers questions correctly against your semantic model using prompt sets and an LLM judge.
When to use it — After you change CML, add a metric or reveal, or want to confirm Orion still answers your key business questions correctly.
Where to find it — Open Forge and switch to the Evals mode tab.
Who can use it — Weaver and Luminary roles (the same builders who edit the semantic model).
Evals turns "does Orion still answer this right?" into a repeatable, scored check. You write a list of real questions a stakeholder might ask, attach what good behavior looks like, and run them. Each answer is graded by an LLM judge that returns a pass/fail verdict, a confidence level, and — when it fails — a failure type. The result is an accuracy score you can watch over time as your model evolves.
Overview
Evals mode has three views that you move between in order:
- Prompt-set list — every eval set in your workspace, with its prompt count, last score, and last run date. This is where you create and delete sets.
- Set detail — the prompts inside one set (add, edit, delete), plus that set's past runs. This is where you press Run eval.
- Run results — the live or completed grading of a run. Each prompt row shows a verdict, confidence, failure type, and timing. The page polls until the run finishes.
A run sends each prompt in the set to Orion, captures Orion's answer (and the SQL or reveal it used), then hands the question, the expected behavior, and Orion's answer to an LLM judge. The judge decides pass or fail, and the run's overall accuracy score is the share of prompts that passed.
When to use it
- You edited CML — a metric definition, a dimension, a reveal — and want to confirm Orion still resolves the right node and returns the right answer.
- You added a new domain or topic and want a baseline of how well Orion handles questions about it.
- You are about to ship a model change and want a before/after accuracy comparison rather than spot-checking by hand.
- You keep a "golden set" of the questions your business actually asks, and you re-run it on a schedule of model changes to catch regressions early.
Evals tests the semantic model through Orion, not the warehouse directly. A failing prompt usually means a modeling gap (a missing synonym, an ambiguous metric, a wrong reveal mapping), not a broken database.
Concepts
| Term | What it means |
|---|---|
| Prompt set | A named collection of eval prompts, scoped to a domain and optionally a reveal. Holds up to 25 prompts. |
| Prompt | A single natural-language question you want Orion to answer, with an optional expected behavior describing what a correct answer looks like. |
| Run | One execution of a prompt set against Orion. Produces a result row per prompt and an overall accuracy score. |
| Verdict | The judge's pass/fail decision for one prompt. |
| Confidence | How sure the judge is — high, medium, or low. |
| Failure type | A short label the judge attaches to a failed answer (for example wrong_metric, missing_filter), shown with underscores turned into spaces. |
| Accuracy score | The percentage of prompts in the run that the judge marked pass. |
Getting started
Prerequisites:
- A Weaver or Luminary role. The Evals API is guarded so only these roles can list, create, or run.
- A working semantic model in the domain you want to test, so Orion has something to answer against.
- Your workspace Anthropic API key configured, since both Orion and the judge are Claude calls.
To open Evals, go to Forge and select the Evals mode. You land on the prompt-set list.
Step-by-step
Create a prompt set
On the prompt-set list, click New prompt set. Enter a Name (for example Order KPIs) and a Domain (for example ecommerce) — both are required. Optionally enter a Reveal (for example order_perf) to scope the set to one reveal. Click Create. You are taken straight into the new set's detail view.
Add prompts
In the Prompts card, type a question into the Add a new eval prompt box. Optionally fill in Expected behavior to tell the judge what a correct answer should contain. Click Add prompt. Repeat for each question — a set holds up to 25 prompts, and the counter (for example 8/25) turns red when you hit the limit.
Edit or delete a prompt
Hover a prompt row and use the pencil icon to edit the text and expected behavior, then Save, or use the trash icon to delete it. Deletes take effect immediately.
Run the eval
Click Run eval in the header. The button is disabled until the set has at least one prompt. You move to the run results view as soon as the run starts.
Watch the results
The results table fills in as prompts are graded — the page polls every two seconds while the run is Running. Each row shows the prompt, its verdict, confidence, failure type, and duration in milliseconds.
Inspect a result
Click any result row to expand it. You see the judge's Rationale, the full Orion answer, the SQL / reveal used, and an error message if the prompt errored.
Review the score
When the run completes, the header shows the overall accuracy score badge and a passed/total count (for example 7/8 passed). The run is also saved under Past runs on the set detail page.
Examples
Suppose you maintain an ecommerce model and want to lock in that Orion answers your core order questions. You create a set named Order KPIs on domain ecommerce, reveal order_perf, and add prompts with expected behavior:
Prompt: What were total orders last month?
Expected behavior: Returns a single number for completed orders in the prior calendar month.
Prompt: What is average order value this quarter?
Expected behavior: Uses the AOV metric (revenue / orders), not raw revenue.
Prompt: Show me returns by category.
Expected behavior: Groups by product category and uses the returns metric, filtered to returned orders.You press Run eval. When it finishes, the run shows 7/8 passed with an accuracy score of 88% (a green badge). You click the one failing row and read the judge's rationale:
Verdict: fail
Confidence: high
Failure type: wrong metric
Rationale: The question asked for average order value, but Orion answered with
total revenue. The AOV metric was not used.That tells you exactly where to look in your CML — the AOV question is resolving to the wrong metric. You fix the model in Forge, re-run the set, and confirm the score recovers.
Best practices
- Write expected behavior for every prompt. The judge grades against it; without it, the verdict is looser and harder to act on.
- Keep one set per concern. Group prompts by domain (and reveal when relevant) so a low score points at one part of the model.
- Re-run after every model change. Treat the set as a regression suite — run it before you consider a CML change done.
- Use real questions. The closest signal to production quality is the questions your stakeholders actually ask.
- Watch the trend, not one run. A single low score can be noise; a sustained drop across runs is a real regression.
Tips
Expand the failing rows first. The Failure type label plus the judge Rationale usually name the modeling fix directly — a missing synonym, the wrong metric, or a filter Orion did not apply. The SQL / reveal used field shows whether Orion even reached the right node.
Common mistakes
- Running with no prompts. The Run eval button stays disabled until the set has at least one prompt.
- Trying to add a 26th prompt. Sets cap at 25; the add box is replaced by a "Maximum 25 prompts reached" message.
- Starting too many runs at once. A workspace allows at most two concurrent runs; a third is rejected.
- Deleting a set to clean up. Deleting a prompt set removes all of its runs too — you are warned before it happens, but the history is gone.
- Treating a fail as a database bug. Evals grades Orion's understanding of your model. A fail is almost always a modeling gap, not a warehouse problem.
Troubleshooting
| Symptom | Cause | Fix |
|---|---|---|
| Run eval button is greyed out | The set has zero prompts | Add at least one prompt, then run |
| "Maximum 25 prompts per set" when adding | The set already holds 25 prompts | Delete or edit existing prompts, or create a second set |
| "Maximum 2 concurrent eval runs per workspace" | Two runs are already in progress in your workspace | Wait for a run to finish, then start the new one |
| "Name and domain are required" on create | Name or domain was left blank | Fill in both fields — they are mandatory |
| A row shows error with an error message | Orion or the judge failed on that prompt | Expand the row to read the error; check that your workspace API key is set and the model is valid |
| Result row stuck on pending… or judging… | The prompt is still running or awaiting the judge | The page polls every two seconds; let it finish — these are normal in-progress states |
| "Run not found" on the results page | The run id is invalid or belongs to another workspace | Go Back and reopen the run from Past runs |
| Accuracy score badge is red | Under 60% of prompts passed | Expand the failing rows, read the rationales, and fix the model gaps they name |