Each record states the decision first, then the context, the options, the reasons, what actually happened (weak numbers included) and what I would change. Records are never edited after the fact; a new record supersedes an old one. The sources are in docs/decisions.
- DR-001A mixed-integer LP for the perfect battery, with an exact solver in the browser
- DR-002Public AEMO prices instead of the course's staff spreadsheet
- DR-003Degradation as an optional linear cost per MWh delivered
- DR-004“Explain this schedule”: optional, bring your own key, grounded and audited
- DR-005Evaluate per market day, daily reset, with a seeded block bootstrap
- DR-006Evaluate the AI explanations against the solver on a fixed set of schedules
DR-001 · Accepted · 2026-10-09
A mixed-integer LP for the perfect battery, with an exact solver in the browser
Applies to: scripts/battery_model.py, web/src/lib/battery, /lab, /benchmark, /sizing, /methods#model
Decision
Model the brief's "perfect but impractical" battery as a linear program with one binary per half-hour that forbids charging and discharging at once, solve it with HiGHS in Python as the reference, and solve the same model exactly in the browser with a dynamic programme (plus simplex and branch-and-bound when a cycle cap applies), checked against HiGHS to the cent.
Context
The brief's mandatory task is to find the revenue-maximising charge and discharge schedule for a 300 MW / 580 MWh battery that knows every future Victorian spot price, under power, capacity, efficiency and loss-factor limits. It hints at a linear program. The revived site also needed the same optimiser to run in a visitor's browser (the Dispatch Lab re-solves on every slider move) and to answer Deliverable 1, which caps the day at a single cycle.
Decision
- Variables per half-hour: grid-side charging power c_t, storage-side discharging power r_t, state of charge s_t, and a binary u_t for "charging mode". Revenue is MLF × price × (η_d r_t − c_t) × 0.5 h, less an optional degradation cost (DR-003).
- Constraints: the storage balance s_t = s_{t−1} + η_c c_t Δ − r_t Δ, 0 ≤ s_t ≤ E, c_t ≤ P u_t and r_t ≤ P (1 − u_t), the battery starts empty, and an optional cap on energy discharged for the cycle limit.
- Reference solver: SciPy's
milp (HiGHS). It solves the LP relaxation first and adds the binaries only when the relaxation overlaps charging and discharging.
- Browser solver: an exact dynamic programme over a piecewise-linear value of stored energy when there is no cap; a bounded-variable simplex, a Lagrangian bound and a small branch-and-bound when there is one.
- Ties: a tiny time-weighted preference (at most $0.0002/MWh) makes the optimum unique, identically in both languages, so forecast-driven policies cannot depend on which solver happened to break a tie.
- "Single cycle" is read as the brief words it: at most 580 MWh discharged in the day.
Options considered
- A plain LP. Exact and fast whenever prices are positive, but at negative prices it charges and discharges in the same half-hour to burn energy through the round-trip loss and get paid for it. A real battery cannot do that.
- An LP with a post-hoc fix (zero out the overlap). Simple, but the fixed schedule is no longer optimal and the size of the error is unknown.
- A mixed-integer LP (chosen for the reference). Exact, and HiGHS solves a 48-period day in milliseconds.
- Dynamic programming only. Exact and very fast, but a cycle cap couples every period, so it needs a Lagrangian outer loop that does not always close the gap on its own.
- HiGHS compiled to WebAssembly in the browser. Exact everywhere, but a large download and a build step for a page that otherwise has no dependencies.
Why
The binary is the smallest change that makes the model physically honest, and solving the LP first keeps it as fast as a plain LP on ordinary days. Two independent implementations checked against each other are stronger evidence than either alone: a bug would have to appear in both, in the same way.
What happened
- The TypeScript port matches HiGHS to the cent on 10 days × 3 battery designs for perfect foresight, a part-charged start, a one-cycle cap and all four practical policies, and every perfect-foresight schedule matches half-hour by half-hour to within 0.001 MW (
web/src/lib/battery/*.test.ts).
- In 2026 the same check was extended to the evaluation artefact: the TypeScript optimiser reproduces every one of the 42 test-window days at all 25 sensitivity designs, and the policies on a spread of 2025 days, to within a few cents (
web/src/lib/evaluation/evaluation.test.ts).
- The weak spot is branch-and-bound with a cycle cap on days with many negative prices. It can stop at its node budget a fraction of a percent short of HiGHS (under 0.5% in every case found, Christmas 2025 being the worst). The Lab says so on screen when it happens.
- During the 2026 evaluation build HiGHS printed an internal diagnostic (
transformNewIntegerFeasibleSolution) on a handful of 2025 rolling-persistence solves. The solves completed, no policy beat perfect foresight on any day, and the TypeScript solver reproduces the sampled days, so I kept the results, but I have not traced the message to its cause.
- Under the brief's reading the best single-cycle revenue on 17 Jul 2020 is $43,360. Under the stricter reading (one charging phase, then one selling phase) it is $34,902. Both are on the site; I chose the literal one.
What I'd change
- Ship HiGHS as WebAssembly for the capped case and delete the hand-written branch-and-bound and its caveat.
- Model five-minute settlement from October 2021 instead of averaging to half-hours.
- Co-optimise frequency-control (FCAS) income, which real batteries depend on and this model ignores.
DR-002 · Accepted · 2026-10-09
Public AEMO prices instead of the course's staff spreadsheet
Applies to: scripts/fetch_aemo.py, scripts/aemo_data.py, scripts/check_staff_parity.py, web/data/analytics.db, /methods#data
Decision
Build every number on the site from AEMO's public Aggregated Price and Demand files for region VIC1 (2018 to 2025), check them against the course's staff-supplied spreadsheet where the two overlap, and never redistribute the spreadsheet.
Context
In 2021 the subject supplied students with a spreadsheet extract of Victorian prices. It is course material: I may use it to check my work but not publish it, and it stops in August 2021. The revival is public, needs to be reproducible by anyone, and is more interesting with the years after the brief (negative midday prices, five-minute settlement).
Decision
- Download AEMO's monthly VIC1 "price and demand" CSVs with
scripts/fetch_aemo.py into an ignored cache.
- Map each interval to the half-hour that contains it (AEMO labels an interval by its end), and from October 2021 average the six five-minute prices in each half-hour, which is how the old 30-minute trading price was defined.
- Keep only complete days (48 half-hours) and store the prices, rounded to four decimals, in the read-only
web/data/analytics.db. Every later script, including the 2026 evaluation, reads its prices from that file, so all pages use identical numbers.
- Compare against the staff extract with
scripts/check_staff_parity.py and commit only the result (scripts/staff_parity.json), never the data.
Options considered
- The staff spreadsheet. The brief's own data, but not mine to redistribute, and it ends in August 2021.
- AEMO's dispatch-level archives (MMSDM / NEMweb). The most detailed source, but large, awkward to parse and more than a half-hourly model needs.
- AEMO's public aggregated price and demand files (chosen). Small, stable, public, and the same series the course extract was drawn from.
Why
Anyone can rebuild the site from public data with two commands, the extra years make the analysis current, and the parity check means nothing is lost by not using the course file.
What happened
- All 63,456 overlapping half-hours from 1 Jan 2018 to 14 Aug 2021 match the staff extract exactly (largest difference $0.00).
- Deliverable 1's price question checks out independently: $20.70/MWh for the half-hour ending 15:00 on 1 Jul 2020.
- The dataset covers 2,922 complete market days. The five-minute averaging is an assumption I have not validated against a settlement-weighted alternative; it matters only after October 2021, so it affects the 2025 results on /evaluation but not the brief's windows.
What I'd change
- Record a checksum of every downloaded file, so a later AEMO revision of a month is detected rather than silently absorbed.
- Use dispatch-level five-minute prices from October 2021 and let the optimiser act every five minutes.
DR-003 · Accepted · 2026-10-09
Degradation as an optional linear cost per MWh delivered
Applies to: web/src/lib/battery/params.ts, scripts/battery_model.py, /lab, /sizing, /evaluation#sensitivity
Decision
Keep the brief's $0/MWh variable cost for every brief result, and model battery wear as an optional linear cost per MWh delivered to the grid, so it can be swept and set in the Lab without leaving the linear program.
Context
The brief sets variable O&M to $0/MWh, so the optimiser will happily cycle for a spread of a few dollars. Real lithium-ion batteries wear with every cycle, and that wear is the main reason operators skip marginal trades. A revival that only ever showed the $0 case would overstate how much a battery should cycle; one that changed the default would no longer answer the brief.
Decision
- Add a variable cost v ($ per MWh delivered to the grid) to the objective: revenue falls by v × η_d × r_t × Δ in every discharging half-hour.
- Default v = $0 everywhere the brief's numbers are reported (Overview, Benchmark, Deliverable 1).
- Expose v as a slider in the Dispatch Lab ($0 to $100/MWh), sweep it on /sizing (six-week joint horizon) and include it in the sensitivity analysis on /evaluation, with intervals over days.
Options considered
- Ignore wear (the brief). Faithful, but it hides the most important practical trade-off.
- A linear throughput cost (chosen). One extra coefficient; the model stays a linear program and the solvers stay exact.
- A cycle limit only, like Deliverable 1's cap. Easy to explain, but it treats a shallow and a deep cycle the same and needs integer reasoning in the browser.
- A depth-of-discharge model (rainflow counting, a non-linear wear curve). Closest to the chemistry, but non-linear and non-convex, and the brief gives no data to calibrate it.
Why
It keeps the exactness of DR-001, it answers the practical question (how much revenue and cycling does wear take away), and it leaves every brief number untouched.
What happened
- On the brief's six-week test window (joint horizon), a $10/MWh cost trims revenue by 9.9% but cuts cycling by 18%: the optimiser drops its marginal round trips first.
- With intervals over days (daily reset, /evaluation), +$10/MWh changes revenue by −9.8% (95% CI −11.3% to −8.3%) in the test window and by −8.0% (95% CI −11.2% to −5.6%) in 2025. The 2025 interval is wider because a few extreme days dominate that year's revenue.
- The weak part is the number itself: I have no calibrated cost for a specific cell chemistry, so the slider shows a range rather than an answer.
What I'd change
- Calibrate v to a published degradation curve and report revenue net of wear alongside gross revenue.
- Approximate depth-dependent wear with a piecewise-linear cost on state-of-charge bands, which keeps the model linear.
DR-004 · Accepted · 2026-10-09
“Explain this schedule”: optional, bring your own key, grounded and audited
Applies to: /lab, /ai-log, /methods#ai-use, web/src/lib/ai, web/src/components/ai
Decision
An optional AI explanation of the schedule on screen in the Dispatch Lab, called from the visitor's browser with their own key, fed only that schedule's numbers from the optimiser, labelled AI-generated, checked for numbers and times it did not receive, and recorded in a local audit log with the visitor's accept, edit or reject decision.
Context
A schedule chart answers "what" (charge here, sell there) but not "why", and the why is the interesting part: which price gaps beat the round-trip loss, why the optimiser charges twice, why a no-foresight policy missed the evening spike. A short plain-language reading is useful, and it is also a small, realistic test of governing generative AI, because the explanation is about numbers and language models are prone to inventing numbers. The project has no budget for an API key, no server for AI, and nothing on the site may depend on AI.
Decision
- Bring your own key, browser only. The visitor pastes an Anthropic (default Claude Haiku 4.5, the cheapest tier; Claude Sonnet 5.5 optional) or OpenAI key (model id editable, default gpt-5-mini). It stays in
sessionStorage unless they tick "remember on this device" (localStorage); "Forget key" removes it. Calls go straight from the browser to the provider (anthropic-dangerous-direct-browser-access: true for Anthropic). The key never reaches this site, a log or the repository.
- Grounded input. The request holds only a JSON snapshot built from the solver's output: the day's price summary, the battery settings and cycle limit, the selected strategy's schedule as charging and selling blocks (times, average prices, energy, money), its revenue arithmetic, and every strategy's revenue and share of perfect foresight that day. No other data, and nothing about the visitor.
- Structured output. The reply must follow a JSON schema (headline, charging, selling, economics, comparison, caveats) through each provider's structured-output mode, and is validated again with zod against the same contract. Long replies are shortened on screen only.
- Refusals. Requests to Claude Sonnet 5.5 opt into Anthropic's server-side refusal fallback (
fallbacks: "default", beta server-side-fallback-2026-07-01): if Sonnet declines, Anthropic may re-run the request on another Claude model, billed to the visitor's key at that model's rates. The settings dialog says so when Sonnet is chosen, and the audit log records the requested model, the model that answered and whether a fallback ran (a fallback content block or a fallback_message entry in usage.iterations). Haiku 4.5 does not take the fallback. A refusal that still comes back is shown as a plain message and logged.
- Transparency. Every explanation is labelled "AI-generated" and shows the model, latency and token usage. A grounding check lists every clock time and every number in the reply that is not in the snapshot (dollar amounts with thousands separators, "k" and "M" suffixes and percentages of shares are understood; whole numbers from 0 to 10 are skipped as counts).
- Human in the loop and audit. The visitor accepts, edits or rejects each explanation. Every call, failures included, is written without the key to an IndexedDB audit log, viewable and exportable (JSON, CSV) at /ai-log. A reply that arrived but was discarded (refused, cut off or off-schema) is kept in the log and labelled AI-generated too.
- Scope of the AI use statement. This feature is the only AI that runs on the site. The site's code, analysis scripts and text were developed with an AI coding assistant (Claude Code) under my direction; I reviewed the results, and every number is computed by the deterministic solvers and statistics code and checked by the test suite, not by an AI. The statement on /methods says both.
Options considered
- No AI. Safe, but misses a useful explanation and a governance showcase.
- A server proxy with my key. Costs money, invites abuse, and puts a secret on a server.
- The Anthropic and OpenAI SDKs in the browser. Typed and maintained, but two extra client bundles for one request shape each, and harder to mock uniformly in tests. Plain
fetch against the documented HTTP APIs keeps the bundle small and every request visible in one place.
- Bring your own key, plain
fetch from the browser (chosen).
Why
The site stays free, static and fully functional without AI, and the governance controls become concrete: what is sent, what comes back, who decided what, all visible to the visitor and exportable. The design is informed by the Australian Government's policy for the responsible use of AI in government, the EU AI Act's transparency principles and the NIST AI Risk Management Framework; it is not a claim of compliance with any of them.
What happened
- The adapters, error mapping, key storage, redaction, audit log, snapshot and grounding check are unit-tested with the network mocked (
web/src/lib/ai/ai.test.ts): the key only ever travels in a request header, a key echoed in a provider error is redacted before it is stored or shown, and the snapshot never contains anything key-like.
- No live call was made during development or CI, because the project has no key. The request shapes follow the providers' documented APIs; a real call is the first thing to check after deployment.
- A review before publication found that the first draft of the AI use statement said the site's text was produced without AI, which the commit history contradicts: every commit is co-authored by an AI coding assistant. The statement now says how AI was used to build the site as well as what runs on it.
- The grounding check is lexical. It catches an invented price or time, but not a correct number in a wrong sentence ("charged at the dearest price"), so it prompts the reviewer rather than deciding.
- The snapshot only describes the schedule; it does not include the optimiser's dual values, so the model cannot say what one more MWh of storage would have been worth that day. It is told to say a claim is not available rather than guess.
What I'd change
- Build a small evaluation set of days with known facts (which blocks charge, which policy wins, whether the cycle cap binds) and score explanations from each model against it, with intervals, as the regret analysis on /evaluation does for the policies.
- Add the marginal value of stored energy from the dynamic programme to the snapshot, so explanations can say why a block was worth trading.
DR-005 · Accepted · 2026-10-09
Evaluate per market day, daily reset, with a seeded block bootstrap
Applies to: scripts/build_evaluation.py, web/data/evaluation.json, web/src/lib/stats, web/src/lib/evaluation, /evaluation
Decision
Add uncertainty to the results by treating the market day as the unit of analysis, solving every day from empty with every plan ending at midnight (daily reset), and putting 95% intervals on every estimate with a seeded circular block bootstrap over days (blocks of seven days), with paired comparisons whenever two methods or two designs are compared; the original six-week totals stay as they were.
Context
The original revival reports single numbers: the six-week revenue of each strategy and a sizing sweep on one window. A single six-week total cannot say whether time of day beat rolling persistence because it is better or because of a few lucky days, or whether a 10% larger battery reliably earns more. The brief's test window is only 42 days, and days are not independent: weather systems and outages run for several days.
Decision
- Unit: the market day.
scripts/build_evaluation.py writes per-day revenues for three samples: the brief's test window (42 days), its cross-validation half-year (181 days) and calendar 2025 (365 days).
- Protocol: daily reset. Every strategy and the optimiser start each day empty and are solved on that day alone, so a day's regret (perfect foresight minus policy) is never negative and days are comparable units. Prices and calibrated policy parameters are read from
analytics.db, so the numbers agree with every other page.
- Horizon: energy still stored at midnight is worth nothing under daily reset, so every policy that plans ahead plans only to midnight: perfect foresight and day-ahead persistence by construction, rolling persistence by planning 48 − k half-hours ahead at half-hour k. The original rolling policy, which plans 24 hours ahead, is kept beside it as a variant, and the energy each strategy still holds at midnight is reported.
- Intervals: circular block bootstrap (Politis and Romano) with blocks of 7 consecutive days, B = 4,000 resamples, seed 20210701, 95% percentile intervals. Shares of days use Wilson intervals. The Wilson intervals and the Wilcoxon tests below treat days as independent, so they are optimistic; the block-bootstrap intervals are the primary evidence.
- Comparisons: paired by day. Each is reported as a mean daily difference with its interval first, then d_z, the share of days won (Wilson), and a Wilcoxon signed-rank p-value, Holm-adjusted across the three questions asked of each sample.
- Sensitivity: one parameter at a time around the brief's design, with changes as paired ratios of totals and arc elasticities from the designs 10% either side.
- Code: helpers in
web/src/lib/stats, each checked against SciPy, statsmodels and base R (scripts/stats_reference.py, scripts/stats_reference.R).
Options considered
- Keep the six-week totals only. Faithful, but no uncertainty at all.
- Carry charge across days, as the benchmark does. Faithful to the benchmark, but a policy can buy on one day and sell on the next, so a day's result is no longer a property of that day and regret can be negative.
- An ordinary (iid) bootstrap over days. Simple, but too narrow when neighbouring days are correlated; on an AR(1) test series with φ = 0.7 the block interval is more than 1.3 times wider (
stats.test.ts).
- Daily reset with a block bootstrap (chosen).
- A parametric model of daily revenue (for example a gamma regression with autocorrelated errors). More efficient if right, but daily revenue is heavy-tailed and spiky, and the model would be another assumption to defend.
Why
Daily reset makes the day a clean experimental unit, which turns every comparison into a paired one on the same days. The block bootstrap respects the dependence between days without a parametric model. Printing the seed, B and block length makes every interval reproducible.
What happened
- The daily-reset perfect-foresight total for the test window is $5,921,103, the same as the original "perfect foresight, daily reset" row; time of day is identical under both protocols ($3,748,713) because its plan ends each day empty. The threshold rule earns less under daily reset than with charge carried over; rolling persistence, planning only to midnight, earns a little more ($2,564,692 against the benchmark's $2,443,858).
- Time of day captured 63.3% (95% CI 52.7% to 73.0%) of perfect foresight in the test window and beat rolling persistence by $28,191 (95% CI $13,441 to $46,854) a day, winning 31 of 42 days.
- Re-planning every half-hour was not clearly different from planning once at midnight in the test window, $7,212 (95% CI −$4,517 to $20,118) a day. In 2025 it was worse: −$26,116 (95% CI −$59,224 to −$7,323) a day, and rolling persistence lost money on 39 of 365 days. Why is not established. Its regret tracks the persistence forecast's error more closely than day-ahead's (Spearman ρ 0.69 (95% CI 0.62 to 0.74) against 0.46 (95% CI 0.36 to 0.56)), which fits re-planning on a noisy forecast amplifying its errors, but I have not tested that mechanism directly.
- A review before publication found that the first version of this evaluation let rolling persistence plan 24 hours ahead, past midnight, while daily reset values energy left at midnight at nothing. It ended 21 of 42 test-window days and 176 of 365 days in 2025 holding energy it had paid for. That version made re-planning look worse than planning at midnight in the test window, −$4,136 (95% CI −$15,813 to $8,150) a day, and inflated time of day's lead over rolling persistence to $39,539 (95% CI $22,197 to $64,514) a day; I had attributed the 2025 result to the forecast chasing spikes without testing it. Planning to midnight is worth $11,535 (95% CI $9,478 to $13,750) a day in 2025. Both versions are on /evaluation.
- The threshold rule does not plan at all and still ends 34 of 42 test-window days holding energy, so daily reset is harsh on it; the carried-charge totals on Benchmark are kinder (the reconciliation is on /evaluation).
- The intervals are wide. Six weeks is not much data, and in the test window the share captured by rolling persistence, 43.3% (95% CI 33.1% to 54.7%), is compatible with anything from a poor to a middling policy.
- The block length of seven days is a judgement, not an estimate; I did not tune it.
What I'd change
- Choose the block length from the data (for example Politis and White's automatic selection) and show how the intervals move with it.
- Add a carried-charge version of the regret analysis with weekly units, so the benchmark's own protocol gets intervals too.
- Report the false discovery rate across the whole page, not just Holm within each sample's three questions.
- Put shares of days and the paired tests on the same footing as the rest: block-bootstrap intervals for shares and a block sign-flip test in place of the Wilcoxon test, so no number assumes independent days.
- Test the noisy-forecast explanation directly, for example by re-running both persistence policies on the real prices plus noise of known size.
DR-006 · Accepted · 2026-10-10
Evaluate the AI explanations against the solver on a fixed set of schedules
Applies to: /evaluation/explanations, /api/explain-eval, /ai-log, web/src/lib/ai/explain-eval*.ts, web/src/lib/ai/grounding.ts, web/src/lib/stats/paired.ts
Decision
Add an evaluation harness for "Explain this schedule": 24 fixed schedules (six edge cases chosen by rule, 18 days drawn with a printed seed), the Lab's prompt unchanged, six automatic checks against the optimiser's own output, a scorer that is itself tested on a template reply and on planted faults, Wilson intervals, paired comparisons between runs decided by an exact test, the visitor's own key, every call in the audit log, and no published results.
Context
DR-004 made the explanation optional, grounded and audited, and its grounding check flags any number the model was not sent. That check is lexical and works one reply at a time: it cannot say how often a model gets the facts right, and it cannot catch a true number or time in the wrong sentence ("it buys at 18:00" when 18:00 is a selling block). DR-004 named the fix under "What I'd change": a small evaluation set of days with known facts, scored with intervals. The project still has no budget for an API key, so whatever is built has to run with a visitor's key, and the site cannot quote results it never measured. This record extends DR-004 without replacing it: the feature, its prompt and its safeguards are unchanged.
Decision
- Item set: 24 schedules on the brief's battery. Six edge cases are chosen by rule: Deliverable 1's day with and without the one-cycle limit, the 2025 day with the most negative half-hours, the 2025 day with the highest price, the test window's most valuable day (explained for rolling persistence) and the least valuable day of 2025 (for the time-of-day schedule). Then six days are drawn at random, without replacement, from each sample on /evaluation (seed 20261010), with perfect foresight, time of day and rolling persistence taking turns. Every schedule is solved exactly as in the Lab, and the 24 snapshots are public at /api/explain-eval.
- Prompt: the Lab's system prompt, snapshot and JSON schema, unchanged, so the evaluation measures what ships.
- Checks: a valid reply; grounded (the Lab's check); every clock time in the charging section inside a charging block and every time in the selling section inside a selling block, with at least one cited in each (times are read on either clock, "18:00" or "6 pm", since the prompt asks for neither); the day's revenue stated in the headline or the economics (gross or net of fixed O&M, within 1%, or to the precision it is written with up to 5%) with the right sign, so a loss must read as a loss and no revenue figure may be reported with the wrong sign; 90 to 270 words. A schedule passes if no check fails, and a check that cannot apply (no selling block, revenue under $1) is skipped. Refusals and cut-off or off-schema replies fail "valid"; network, key and rate-limit failures are not scored.
- Testing the scorer: a template reply written from each snapshot by rule, with no AI, must pass every check, with its times on the 24-hour clock and again on the 12-hour clock. Seven planted faults (an off-schema reply, an invented price, buying and selling swapped, one wrong time, the revenue left out, a loss reported as earnings, padding past the limit) must each be caught by the check aimed at it and by no other, on every schedule where the fault can apply. This runs when the site is built and is shown on the page.
- Statistics: shares of schedules with 95% Wilson intervals. The 18 random days are reported on their own, as the estimate for a typical day in the Lab, and the six edge cases as counts: they are hand-picked to be hard, so a share pooled over all 24 estimates no population and is kept only as the headline for comparing two runs, where the pairing makes the mix irrelevant. The mean number of ungrounded numbers per reply has a 95% bootstrap percentile interval over schedules (B = 4,000, seed 20261010). Two runs are compared by the paired difference in pass rates, with Newcombe's hybrid score interval, and an exact McNemar test on the schedules where they disagree; the exact test decides which run is ahead, at the 5% level. Both are checked against R (base R and the contingencytables package) and against statsmodels and SciPy.
- Human review: the visitor can mark each automatic verdict right or wrong. Agreement with the scorer is reported separately for automatic passes and automatic fails, and the pass rate after review, all with Wilson intervals. The visitor chooses what to review, so these figures describe the reviewed replies only, and the page says so. The audit entry's decision stays "not applicable", because the review judges the scorer, not the reply.
- Records: every call, retries and failures included, goes to the AI audit log with the feature
explain-schedule-eval, the run, the schedule, its score and the set's hash (SHA-256 of the snapshots, the prompt and the settings, computed when the site is built). Runs are read back from the log, so they survive a reload and export as CSV or JSON, and runs are only compared within one version and one hash, so a rebuilt set is never paired with the old one. A run sends 24 requests, two at a time, retries a call once after a rate limit, overload, server or network error (so up to 48 requests), and stops on a bad key or an unknown model, or as soon as the visitor forgets the key.
Options considered
- Keep only the per-reply grounding check (DR-004). No measure of how often a model is right, and blind to a true time in the wrong sentence.
- An LLM judge that grades each reply. It could catch subtler errors, but it needs a second paid call per reply, has an error rate of its own to measure, and grades with the same kind of system it is judging.
- Publish results from my own runs. The page would have numbers, but that needs a key and a budget I do not have, and a published score goes stale as providers update models behind the same name.
- Fixed schedules, rule-based checks against the solver, a tested scorer and the visitor's key (chosen).
Why
The facts that matter most in an explanation of a schedule (when it buys, when it sells, what it earned, which numbers it quotes) are already known exactly from the optimiser, so they can be checked without another model. Testing the scorer on planted faults shows what each check can catch before it is trusted with a model's replies. Pairing two runs on the same schedules makes "is model B better?" and "how much do replies vary between runs?" the same paired question the regret analysis asks of the policies.
What happened
- The template reply passes every check on all 24 schedules, with its times on either clock, and each of the 7 planted faults is caught wherever it applies by the check aimed at it, with no other check firing: on all 24 schedules for six of them, and for a loss reported as earnings, on the 2 schedules that lose money (
explain-eval.test.ts, and on the page).
- A whole run was tested against a mocked provider: replies scored, a refusal scored as not valid, a rate-limited call retried once, a bad key stopping the run, the key forgotten mid-run (no further request sent), every call in the log and the key in none of them.
- A review before release found four flaws in the first version, all fixed before it shipped. The comparison called a winner whenever a percentile bootstrap interval excluded 0, which is too narrow with few disagreements: a 4 to 0 split of 24 schedules gave an interval above 0 while the exact McNemar p was 0.125. The revenue check ignored the sign, so "earned $52,360" on a day that lost $52,360 passed. Times written as "1:00 pm" were read as 01:00, so a correct reply on the 12-hour clock failed two checks. And a rounded revenue such as "about $0.1 million" passed at 33% off, against the stated 1%.
- With a handful of disagreements, Newcombe's interval can just exclude 0 while the exact test does not reach 5% (a 5 to 0 split: interval above 0, p = 0.0625). The page then says the runs cannot be told apart, and why.
- No live run was made, because the project has no key, so the page shows no results until a visitor runs it. Whether the checks agree with a careful reader on real replies is unknown until someone reviews a run; the planted faults are cruder than a model's mistakes.
- With 24 schedules the intervals are wide: a run that passes all 24 still has a Wilson interval reaching down to 86%, and one that passes all 18 random days down to 82%. The set can tell a model that often gets the facts wrong from one that rarely does, not 90% from 95%.
- The time checks will fail some fair replies, such as "it waits past the 04:00 dip" in the charging section. That is why the reviewer can overrule a verdict, and why agreement is reported.
What I'd change
- Grow the set, or draw a fresh one for each run, once real runs show how wide the intervals are in practice.
- Check claims beyond times, numbers and revenue: which strategy did best, whether a no-foresight policy is said to have known a price, whether the cycle limit binds.
- Add a helpfulness rubric scored by people, so the evaluation covers whether an explanation is useful and not only whether it is right.
- Compare effort settings and prompt variants on the same schedules, paired, before changing the Lab's defaults.
- Offer a review mode that presents replies in a seeded random order, so agreement with the scorer estimates the whole run rather than the replies a visitor chose to open.