Evaluation · AI explanations · added in 2026
Does the AI explain a schedule correctly?
The Dispatch Lab can explain a schedule in plain language with an AI model and your own key. Here that feature faces a fixed test: the same 24 schedules every time, the same prompt as the Lab, and automatic checks against the optimiser's own numbers, with 95% intervals.
It runs only when you click, with your key. This site has no key, so it publishes no results of its own: the results table fills in for you.
Evaluation design
How a reply is judged
Evaluation design
- Unit
- One reply to one schedule. A schedule passes if none of the checks below fails.
- Schedules
- 24 fixed schedules: 6 edge cases chosen by rule and 18 market days drawn at random (seed 20261010), 6 from each sample on /evaluation. The strategies take turns: perfect foresight, time of day, rolling persistence. The 18 random days are reported on their own as the estimate for a typical day; the edge cases are counted.
- Prompt
- Exactly what the Dispatch Lab sends: the same system prompt, the same snapshot built from the solver's output, the same JSON schema. What is evaluated is what ships.
- Intervals
- Shares of schedules: 95% Wilson intervals. Paired differences between runs: 95% Newcombe hybrid score intervals. The mean number of ungrounded numbers per reply: a 95% bootstrap percentile interval over schedules (B = 4,000, seed 20261010).
- Comparisons
- Two runs (two models, or one model twice) are paired by schedule: the difference in pass rates with its interval, and an exact McNemar test on the schedules where they disagree. The exact test decides which run is ahead, at the 5% level.
- Human review
- You can mark each automatic verdict right or wrong. The page reports how often the scorer was upheld, separately for automatic passes and fails, with intervals, and the pass rate after review. You pick what to review, so these describe the reviewed replies only.
Set and scorer version explain-eval-v1, set hash a5447527ab6c (SHA-256 of the snapshots, the prompt and the settings: runs are only compared within one hash). The 24 snapshots, the system prompt and the settings are public at /api/explain-eval. Rationale and trade-offs: DR-006.
01 · The checks
Six things a reply must get right
Each check compares the reply with the snapshot the model was sent, which comes from the optimiser. The grounding check alone cannot tell a true time in a wrong sentence (“it buys at 18:00” when 18:00 is a selling block); the time checks can. Times are read on either clock (“18:00” and “6 pm” are the same), since the prompt does not ask for one.
| Check | Passes when | Cannot see |
|---|---|---|
| Valid reply | A reply arrived, was not a refusal or cut off, and matched the JSON schema. | Says nothing about the content. Network, key and rate-limit failures are not scored. |
| Grounded | Every number and clock time it checks (24-hour or am/pm) can be found in the snapshot that was sent (the Lab's grounding check). | Lexical: a true number in a wrong sentence passes. Whole numbers from 0 to 10 are not checked. |
| Charging times | The charging section cites at least one clock time (14:30, 2:30 pm, noon or midnight all count), and every time it cites falls inside a charging block. | A time mentioned for context ("it waits past the 04:00 dip") fails it, and in "from 1 to 3 pm" only 3 pm is read; the reviewer can overrule. |
| Selling times | The selling section cites at least one clock time, and every time it cites falls inside a selling block. | As for charging times. |
| Revenue stated | The headline or the economics section states the day's revenue (gross or net of fixed O&M) within 1%, or to the precision it is written with up to 5% ("about $6.3 million"), and with the right sign: a loss must read as a loss, and no revenue figure may be reported with the wrong sign. | Any number that close counts, so a coincidental match passes. The sign is read from a minus sign and the words next to the figure ("lost", "a loss of", "earned"), so an unusual sentence can fool it. Not applicable when the revenue is under $1. |
| Length | 90 to 270 words in total (the prompt asks for about 180). | Instruction-following only. |
02 · Testing the scorer
Before it scores a model, the scorer is scored
A template explanation is written by rule from each snapshot, with no AI, so every number and time in it is right: the checks must pass it, with its times on the 24-hour clock and again on the 12-hour clock. Then 7 faults are planted in copies of it, one at a time, on every schedule where they can apply: the check aimed at each fault must catch it, and no other check should fire. Computed when the site is built, on all 24 schedules.
| Reply | Aimed check | Caught | Other checks fired |
|---|---|---|---|
| Template (correct)Written from the snapshot by rule; must pass every check. | all must pass | passed 24 of 24 | none |
| Template, 12-hour times (correct)The same reply with every time written as “1:30 pm”, “noon” or “midnight”. | all must pass | passed 24 of 24 | none |
| Off-schema replyNo usable reply (as after a refusal or a cut-off answer). | Valid reply | 24 of 24 | none |
| Invented priceAdds a sentence quoting a peak price $137.21/MWh above the real one. | Grounded | 24 of 24 | none |
| Buying and selling swappedThe charging and selling sections trade places: right times, wrong sentences. | Charging times or Selling times | 24 of 24 | none |
| One wrong timeAdds to the charging section a time that is in the snapshot but outside every charging block. | Charging times | 24 of 24 | none |
| Revenue left outDrops the day's revenue from the headline and the economics. | Revenue stated | 24 of 24 | none |
| Loss reported as earningsOn a day that loses money (before or after fixed O&M), says it earned the amount it lost. | Revenue stated | 2 of 2applies to 2 of 24 | none |
| Far too longPads the comparison with repeated sentences to over 270 words. | Length | 24 of 24 | none |
This shows the checks do what they say on faults of a known kind. It does not show they agree with a careful reader on real replies, whose mistakes are subtler: that is what your review of each verdict measures.
03 · The schedules
24 schedules, fixed in advance
The brief's battery (300 MW, 580 MWh, 81% round trip), every strategy solved as in the Lab, rolling persistence planning to midnight. The edge cases are where a careless explanation is most likely to go wrong.
| Item | Strategy | Revenue | Blocks (buy / sell) | Why it is in the set |
|---|---|---|---|---|
| E1 17 Jul 2020Edge case | Perfect foresight | $51,350 | 4 / 4 | Deliverable 1's day: the brief's own worked example, no cycle limit. |
| E2 17 Jul 2020Edge case | Perfect foresight1-cycle limit | $43,360 | 3 / 3 | The same day with the brief's one-cycle limit, which binds. |
| E3 10 Oct 2025Edge case | Perfect foresight | $63,877 | 14 / 13 | Most negative half-hours in 2025 (48 of 48): the battery is paid to charge. |
| E4 12 Jun 2025Edge case | Perfect foresight | $6,321,342 | 4 / 4 | Highest half-hourly price in 2025 ($14,649/MWh): one spike dominates the day. |
| E5 6 Jul 2021Edge case | Rolling persistence | $51,067 | 4 / 5 | The test window's most valuable day for perfect foresight, explained for a policy that cannot see it coming. |
| E6 17 Nov 2025Edge case | Time of day | $1,529 | 2 / 3 | The least valuable day of 2025 for perfect foresight: a fixed schedule on a day with little worth trading. |
| T1 8 Jul 2021Test window 2021 | Perfect foresight | $148,225 | 3 / 4 | Drawn at random (seeded) from the sample. |
| T2 18 Jul 2021Test window 2021 | Time of day | $70,785 | 2 / 3 | Drawn at random (seeded) from the sample. |
| T3 26 Jul 2021Test window 2021 | Rolling persistence | $89,678 | 7 / 5 | Drawn at random (seeded) from the sample. |
| T4 29 Jul 2021Test window 2021 | Perfect foresight | $194,858 | 6 / 6 | Drawn at random (seeded) from the sample. |
| T5 6 Aug 2021Test window 2021 | Time of day | $64,621 | 2 / 3 | Drawn at random (seeded) from the sample. |
| T6 7 Aug 2021Test window 2021 | Rolling persistence | $88,231 | 2 / 2 | Drawn at random (seeded) from the sample. |
| C1 6 Jan 2021Cross-validation 2021 | Perfect foresight | $68,308 | 8 / 9 | Drawn at random (seeded) from the sample. |
| C2 26 Jan 2021Cross-validation 2021 | Time of day | $11,539 | 2 / 3 | Drawn at random (seeded) from the sample. |
| C3 15 Feb 2021Cross-validation 2021 | Rolling persistence | $10,340 | 1 / 1 | Drawn at random (seeded) from the sample. |
| C4 21 Feb 2021Cross-validation 2021 | Perfect foresight | $25,174 | 5 / 5 | Drawn at random (seeded) from the sample. |
| C5 14 Apr 2021Cross-validation 2021 | Time of day | $80,935 | 2 / 3 | Drawn at random (seeded) from the sample. |
| C6 17 Apr 2021Cross-validation 2021 | Rolling persistence | $100,079 | 2 / 2 | Drawn at random (seeded) from the sample. |
| Y1 21 Jan 2025Calendar 2025 | Perfect foresight | $106,407 | 5 / 6 | Drawn at random (seeded) from the sample. |
| Y2 7 Feb 2025Calendar 2025 | Time of day | −$52,360 | 2 / 3 | Drawn at random (seeded) from the sample. |
| Y3 26 Apr 2025Calendar 2025 | Rolling persistence | $36,975 | 3 / 2 | Drawn at random (seeded) from the sample. |
| Y4 16 Jul 2025Calendar 2025 | Perfect foresight | $169,707 | 7 / 5 | Drawn at random (seeded) from the sample. |
| Y5 23 Aug 2025Calendar 2025 | Time of day | $82,883 | 2 / 3 | Drawn at random (seeded) from the sample. |
| Y6 13 Sep 2025Calendar 2025 | Rolling persistence | $91,011 | 4 / 4 | Drawn at random (seeded) from the sample. |
04 · Run it
Your key, your model, your results
Anthropic (default Claude Haiku 4.5, or Claude Sonnet 5.5) or OpenAI (default gpt-5-mini, editable), set in AI settings. Each of the 24 calls (up to 48 if calls are retried once after a rate limit, overload, server or network error) sends about as much as one explanation in the Lab and is written to the AI audit log with its score, failures included; runs are read back from there, so they survive a reload and can be exported.
Run the evaluationoptional · AI · your own key
Needs your own Anthropic or OpenAI key. The results, the score of every reply and your reviews stay in this browser's AI audit log.
Reading earlier runs from this browser…
05 · Reading the results
What a score here does and does not mean
- 24 schedules separate a model that often gets the facts wrong from one that rarely does. They cannot separate 90% from 95%: with every schedule passing, the 95% Wilson interval still reaches down to 86%.
- Replies vary from call to call (no sampling settings are fixed, as in the Lab). Run the same model twice and compare the runs to see how much.
- The checks are about facts and instructions, not about whether an explanation is helpful or well reasoned. A reply can pass every check and still be a poor read.
- The edge cases are chosen to be hard, so the share over all 24 is not an estimate for any population. For a typical day in the Lab, read the 18 random days (with their own, wider, interval); the 6 edge cases are reported as counts. The pooled share is the headline for comparing two runs, where the pairing makes the mix irrelevant.
- The checks are lexical, so they can be wrong both ways; the self-check above covers planted faults, and your reviews measure agreement on real replies. You choose which replies to review, so agreement describes those replies, not the whole run.
- Providers update models behind the same name. A run is a measurement of a model on the day it ran; the audit log records which model answered.