Evaluation · added in 2026
How sure are these numbers?
The other pages report one total for one six-week window. Here every number comes with a 95% interval over market days, comparisons are paired day by day, and every seed is printed. The original results on the other pages are unchanged.
Evaluation design
How every number below is made
Evaluation design
- Unit of analysis
- The market day. Every number is built from per-day results, so every interval is over days.
- Protocol
- Daily reset: each day starts empty and is solved on its own. Energy left at midnight is worth nothing, so every policy that plans ahead plans only to midnight (DR-005).
- Samples
- Brief test window (n = 42), Cross-validation window (n = 181), Calendar 2025 (n = 365)
- Intervals
- 95% percentile intervals from a circular block bootstrap: blocks of 7 consecutive days, B = 4,000 resamples, seed 20210701.
- Proportions
- Wilson score intervals (shares of days a policy wins, or loses money). They treat days as independent, so they are a little optimistic.
- Comparisons
- Paired by day: mean difference with its interval, the standardised effect d_z, the share of days won, and a Wilcoxon signed-rank test, Holm-adjusted across the three questions.
Per-day revenues: web/data/evaluation.json, written by scripts/build_evaluation.py with the Python reference optimiser (scipy.optimize.milp (HiGHS), SciPy 1.18.1) on the same prices as every other page. In the test suite the TypeScript optimiser reproduces every sensitivity design on every day (to $0.02), and the policies 2,350 of 2,352 strategy-days (to $0.05; the other two are ties between equally good plans). Intervals are computed at build time by the tested helpers in web/src/lib/stats, checked against SciPy, statsmodels and R.
01 · Sensitivity
Which knob matters most?
Perfect-foresight revenue for every day of a sample, re-solved with one battery setting changed at a time around the brief's 300 MW / 580 MWh / 81% design. Changes are paired: the same days are resampled for both designs, so an interval says how consistently the change pays, not how much daily revenue varies.
Mean per day
$140,979
95% CI $121k to $168k
Market days
42
Brief test window
Largest elasticity
0.99
Round-trip efficiency
+$10/MWh wear
−9.8%
95% CI −11.3% to −8.3%
Tornado
Change in revenue from the brief's design
- Power−10% (270 MW)−4.9% [−5.8%, −4.2%]
- Power+10% (330 MW)+4.5% [+3.7%, +5.4%]
- Energy capacity−10% (522 MWh)−5.5% [−6.3%, −4.6%]
- Energy capacity+10% (638 MWh)+5.1% [+4.2%, +5.8%]
- Round-trip efficiency−10% (72.9%)−9.6% [−12.7%, −6.2%]
- Round-trip efficiency+10% (89.1%)+10.2% [+6.5%, +13.4%]
- Degradation cost+$5/MWh ($5/MWh)−5.1% [−5.9%, −4.3%]
- Degradation cost+$10/MWh ($10/MWh)−9.8% [−11.3%, −8.3%]
Dots are estimates, bars 95% intervals. The line marks no change.
Elasticities
Revenue gained per 1% more
| Parameter | Elasticity |
|---|---|
| Round-trip efficiency72.9% to 89.1% | 0.990.63 to 1.30 |
| Energy capacity522 MWh to 638 MWh | 0.530.44 to 0.61 |
| Power270 MW to 330 MW | 0.470.40 to 0.56 |
An elasticity of 0.5 means a 1% larger value earns about 0.5% more. It comes from the designs 10% either side of the brief's.
Round-trip efficiency against energy capacity: the difference in elasticity is 0.46 (95% CI 0.04 to 0.85), so round-trip efficiency is the more valuable knob in this sample.
Power
Mean revenue per day
Power (MW); filled dot: the brief's design
Energy capacity
Mean revenue per day
Energy capacity (MWh); filled dot: the brief's design
Round-trip efficiency
Mean revenue per day
Round-trip efficiency; filled dot: the brief's design
Degradation cost
Mean revenue per day
Degradation cost ($/MWh); filled dot: the brief's design
Every designFull table: 27 designs × 42 daysshow table
| Design | Mean per day | Change vs brief | Cycles / day |
|---|---|---|---|
| Power 75 MW | $56,070$49k to $63k | −60.2%−63.4% to −56.9% | 1.20 |
| Power 150 MW | $97,129$85k to $111k | −31.1%−34.6% to −28.0% | 2.01 |
| Power 270 MW | $134,043$115k to $158k | −4.9%−5.8% to −4.2% | 2.77 |
| Power 300 MWbrief | $140,979$121k to $168k | – | 2.93 |
| Power 330 MW | $147,302$126k to $177k | +4.5%+3.7% to +5.4% | 3.11 |
| Power 450 MW | $168,542$141k to $208k | +19.6%+15.8% to +24.1% | 3.74 |
| Power 600 MW | $190,160$155k to $242k | +34.9%+27.4% to +44.0% | 4.52 |
| Energy capacity 145 MWh | $61,838$48k to $83k | −56.1%−61.0% to −50.3% | 7.08 |
| Energy capacity 290 MWh | $95,080$78k to $121k | −32.6%−36.3% to −28.0% | 4.52 |
| Energy capacity 522 MWh | $133,169$113k to $160k | −5.5%−6.3% to −4.6% | 3.12 |
| Energy capacity 580 MWhbrief | $140,979$121k to $168k | – | 2.93 |
| Energy capacity 638 MWh | $148,171$127k to $175k | +5.1%+4.2% to +5.8% | 2.78 |
| Energy capacity 870 MWh | $172,630$150k to $200k | +22.5%+18.5% to +25.7% | 2.37 |
| Energy capacity 1,160 MWh | $194,259$171k to $221k | +37.8%+30.8% to +43.9% | 2.01 |
| Round-trip efficiency 60% | $107,445$95k to $121k | −23.8%−31.0% to −15.5% | 2.28 |
| Round-trip efficiency 70% | $122,871$108k to $141k | −12.8%−16.9% to −8.3% | 2.51 |
| Round-trip efficiency 72.9% | $127,458$111k to $148k | −9.6%−12.7% to −6.2% | 2.60 |
| Round-trip efficiency 81%brief | $140,979$121k to $168k | – | 2.93 |
| Round-trip efficiency 89.1% | $155,337$131k to $190k | +10.2%+6.5% to +13.4% | 3.29 |
| Round-trip efficiency 95% | $166,710$139k to $207k | +18.3%+11.7% to +24.0% | 3.64 |
| Round-trip efficiency 100% | $177,307$145k to $223k | +25.8%+16.4% to +34.1% | 4.13 |
| Degradation cost $0/MWhbrief | $140,979$121k to $168k | – | 2.93 |
| Degradation cost $5/MWh | $133,744$114k to $160k | −5.1%−5.9% to −4.3% | 2.63 |
| Degradation cost $10/MWh | $127,127$107k to $154k | −9.8%−11.3% to −8.3% | 2.41 |
| Degradation cost $20/MWh | $115,560$96k to $142k | −18.0%−20.4% to −15.4% | 2.09 |
| Degradation cost $30/MWh | $105,109$86k to $131k | −25.4%−28.6% to −21.9% | 1.92 |
| Degradation cost $50/MWh | $86,417$68k to $111k | −38.7%−43.4% to −33.5% | 1.68 |
Daily reset, so these differ slightly from the six-week joint optimisation on Sizing, which lets energy carry over midnight. Round-trip efficiency is split evenly between charging and discharging. On 1 of 42 days the battery 10% more efficient earned less than the brief's: at negative prices the battery is paid to take energy, and conversion losses let a less efficient battery take more of it.
Mean per day
$155,607
95% CI $111k to $220k
Market days
365
Calendar 2025
Largest elasticity
0.71
Round-trip efficiency
+$10/MWh wear
−8.0%
95% CI −11.2% to −5.6%
Tornado
Change in revenue from the brief's design
- Power−10% (270 MW)−3.5% [−3.8%, −3.2%]
- Power+10% (330 MW)+3.1% [+2.8%, +3.4%]
- Energy capacity−10% (522 MWh)−6.9% [−7.2%, −6.6%]
- Energy capacity+10% (638 MWh)+6.5% [+6.2%, +6.8%]
- Round-trip efficiency−10% (72.9%)−6.8% [−7.6%, −6.1%]
- Round-trip efficiency+10% (89.1%)+7.3% [+6.6%, +8.2%]
- Degradation cost+$5/MWh ($5/MWh)−4.4% [−6.2%, −3.0%]
- Degradation cost+$10/MWh ($10/MWh)−8.0% [−11.2%, −5.6%]
Dots are estimates, bars 95% intervals. The line marks no change.
Elasticities
Revenue gained per 1% more
| Parameter | Elasticity |
|---|---|
| Round-trip efficiency72.9% to 89.1% | 0.710.64 to 0.79 |
| Energy capacity522 MWh to 638 MWh | 0.670.64 to 0.70 |
| Power270 MW to 330 MW | 0.330.30 to 0.36 |
An elasticity of 0.5 means a 1% larger value earns about 0.5% more. It comes from the designs 10% either side of the brief's.
Round-trip efficiency against energy capacity: the difference in elasticity is 0.03 (95% CI −0.04 to 0.13), so this sample cannot separate them.
Power
Mean revenue per day
Power (MW); filled dot: the brief's design
Energy capacity
Mean revenue per day
Energy capacity (MWh); filled dot: the brief's design
Round-trip efficiency
Mean revenue per day
Round-trip efficiency; filled dot: the brief's design
Degradation cost
Mean revenue per day
Degradation cost ($/MWh); filled dot: the brief's design
Every designFull table: 27 designs × 365 daysshow table
| Design | Mean per day | Change vs brief | Cycles / day |
|---|---|---|---|
| Power 75 MW | $73,849$56k to $99k | −52.5%−55.3% to −49.0% | 1.13 |
| Power 150 MW | $118,897$86k to $167k | −23.6%−25.5% to −21.6% | 1.85 |
| Power 270 MW | $150,193$107k to $212k | −3.5%−3.8% to −3.2% | 2.70 |
| Power 300 MWbrief | $155,607$111k to $220k | – | 2.90 |
| Power 330 MW | $160,457$115k to $227k | +3.1%+2.8% to +3.4% | 3.09 |
| Power 450 MW | $176,319$126k to $249k | +13.3%+12.1% to +14.9% | 3.76 |
| Power 600 MW | $191,807$137k to $271k | +23.3%+20.9% to +26.5% | 4.57 |
| Energy capacity 145 MWh | $56,904$41k to $80k | −63.4%−65.2% to −60.9% | 7.13 |
| Energy capacity 290 MWh | $95,904$68k to $135k | −38.4%−39.6% to −36.7% | 4.57 |
| Energy capacity 522 MWh | $144,872$103k to $205k | −6.9%−7.2% to −6.6% | 3.11 |
| Energy capacity 580 MWhbrief | $155,607$111k to $220k | – | 2.90 |
| Energy capacity 638 MWh | $165,781$118k to $234k | +6.5%+6.2% to +6.8% | 2.72 |
| Energy capacity 870 MWh | $202,544$144k to $287k | +30.2%+28.5% to +31.7% | 2.23 |
| Energy capacity 1,160 MWh | $237,794$171k to $333k | +52.8%+49.0% to +56.9% | 1.85 |
| Round-trip efficiency 60% | $128,804$92k to $182k | −17.2%−19.2% to −15.3% | 2.38 |
| Round-trip efficiency 70% | $141,265$101k to $200k | −9.2%−10.3% to −8.2% | 2.62 |
| Round-trip efficiency 72.9% | $144,958$104k to $205k | −6.8%−7.6% to −6.1% | 2.69 |
| Round-trip efficiency 81%brief | $155,607$111k to $220k | – | 2.90 |
| Round-trip efficiency 89.1% | $166,922$120k to $235k | +7.3%+6.6% to +8.2% | 3.16 |
| Round-trip efficiency 95% | $176,260$127k to $248k | +13.3%+12.0% to +15.0% | 3.47 |
| Round-trip efficiency 100% | $185,289$134k to $260k | +19.1%+17.1% to +21.7% | 3.84 |
| Degradation cost $0/MWhbrief | $155,607$111k to $220k | – | 2.90 |
| Degradation cost $5/MWh | $148,795$104k to $213k | −4.4%−6.2% to −3.0% | 2.33 |
| Degradation cost $10/MWh | $143,174$99k to $208k | −8.0%−11.2% to −5.6% | 2.01 |
| Degradation cost $20/MWh | $133,647$89k to $198k | −14.1%−19.7% to −9.9% | 1.68 |
| Degradation cost $30/MWh | $125,408$81k to $190k | −19.4%−27.0% to −13.7% | 1.49 |
| Degradation cost $50/MWh | $111,245$67k to $176k | −28.5%−39.6% to −20.2% | 1.25 |
Daily reset, so these differ slightly from the six-week joint optimisation on Sizing, which lets energy carry over midnight. Round-trip efficiency is split evenly between charging and discharging. On 35 of 365 days the battery 10% more efficient earned less than the brief's: at negative prices the battery is paid to take energy, and conversion losses let a less efficient battery take more of it.
02 · Regret
What does not seeing the future cost?
Each day, regret is what perfect foresight earned minus what a policy that sees only past prices earned. The policies were tuned on 2018–2020, so all three samples are out of sample. The two persistence policies are the forecast-based ones: they plan with the optimiser on the forecast “today repeats yesterday”.
Time-of-day schedule captured 63.3% of the perfect-foresight revenue (95% CI 52.7% to 73.0%). Re-planning every half-hour on the persistence forecast was not clearly different from planning once at midnight: $7,212 a day (95% CI −$4,517 to $20,118).
Per strategy
Revenue, regret and share captured
| Strategy | Total | Per day | Regret per day | Share captured | Days losing money | Holding energy at midnight |
|---|---|---|---|---|---|---|
| Perfect foresightforesight | $5.92M | $140,979$121k to $168k | – | 100% | 0 of 420% to 8% | 2 of 42mean 20 MWh |
| Rolling persistence (MPC)forecast | $2.56M | $61,064$52k to $71k | $79,915$56k to $111k | 43.3%33.1% to 54.7% | 2 of 421% to 16% | 3 of 42mean 27 MWh |
| Day-ahead persistenceforecast | $2.26M | $53,852$34k to $72k | $87,127$56k to $123k | 38.2%23.0% to 55.3% | 3 of 422% to 19% | 2 of 42mean 20 MWh |
| Fixed price thresholdsrule | $1.17M | $27,886$18k to $38k | $113,093$97k to $141k | 19.8%12.9% to 26.9% | 10 of 4213% to 39% | 34 of 42mean 407 MWh |
| Time-of-day schedulerule | $3.75M | $89,255$72k to $107k | $51,724$35k to $71k | 63.3%52.7% to 73.0% | 1 of 420% to 12% | 0 of 42mean 0 MWh |
Share captured is total policy revenue ÷ total perfect-foresight revenue, resampled by day. Regret per day is never negative under daily reset. Days losing money carry Wilson intervals. Holding energy at midnight: days ending with more than 1 MWh stored, energy bought and never sold, which daily reset values at nothing.
Share captured
Out of the perfect-foresight revenue
- Rolling43% [33%, 55%]
- Day-ahead38% [23%, 55%]
- Thresholds20% [13%, 27%]
- Time of day63% [53%, 73%]
Dots are estimates, bars 95% intervals. 100% would match perfect foresight.
Paired by day
Three questions
Does a fixed daily schedule beat re-planning on yesterday's prices?
Time of day minus Rolling: $28,191 a day (95% CI $13,441 to $46,854). Time of day earned more on 31 of 42 days (74%, 95% CI 59% to 85%).
Time of day higherd_z 0.41r_rb 0.58Wilcoxon p (Holm) 0.002
Does re-planning every half hour beat planning once at midnight?
Rolling minus Day-ahead: $7,212 a day (95% CI −$4,517 to $20,118). Rolling earned more on 21 of 42 days (50%, 95% CI 36% to 64%).
no clear differenced_z 0.17r_rb 0.14Wilcoxon p (Holm) 0.43
Is trading on the clock better than trading on price levels?
Time of day minus Thresholds: $61,369 a day (95% CI $38,368 to $83,446). Time of day earned more on 38 of 42 days (90%, 95% CI 78% to 96%).
Time of day higherd_z 0.89r_rb 0.90Wilcoxon p (Holm) < 0.001
d_z is the mean daily difference divided by its standard deviation; r_rb is the matched-pairs rank-biserial correlation (−1 to 1). Questions: 3, Holm-adjusted together.
Horizon check
Planning to midnight, or past it
Under daily reset, energy still stored at midnight earns nothing, so the rolling policy above plans only to midnight. The original rolling policy (Benchmark, Dispatch Lab) plans 24 hours ahead from every half-hour, so from the afternoon on it buys energy to sell tomorrow morning, which this protocol never pays for. Same days, both versions:
| Rolling plans | Share captured | Days losing money | Holding energy at midnight |
|---|---|---|---|
| To midnightevaluated above | 43.3%33.1% to 54.7% | 2 of 42 | 3 of 42mean 27 MWh |
| 24 hours aheadoriginal policy | 35.3%20.9% to 50.4% | 4 of 42 | 21 of 42mean 176 MWh |
Planning to midnight earned $11,348 a day more (95% CI $4,625 to $20,853). The questions that involve rolling persistence, both ways:
- Time of day minus Rolling a day: $28,191 (95% CI $13,441 to $46,854) planning to midnight; $39,539 (95% CI $22,197 to $64,514) with the 24-hour plan.
- Rolling minus Day-ahead a day: $7,212 (95% CI −$4,517 to $20,118) planning to midnight; −$4,136 (95% CI −$15,813 to $8,150) with the 24-hour plan.
Forecast error
Does a worse forecast cost more?
Regret on the day ($, symmetric-log scale)
| Policy | Spearman ρ |
|---|---|
| Rollingplans on the forecast | 0.730.56 to 0.83 |
| Day-aheadplans on the forecast | 0.740.40 to 0.83 |
| Thresholdsignores it | 0.20−0.18 to 0.55 |
| Time of dayignores it | 0.550.20 to 0.71 |
Volatile days are both hard to forecast and lucrative to trade, so regret can rise with forecast error even for a policy that never looks at the forecast. The rule-based policies are the control: the correlation for rolling persistence minus the one for time of day is 0.18 (95% CI 0.02 to 0.45), so forecast error hurts the policy that plans on the forecast more. Median forecast error: $42/MWh.
Reconciliation
Daily reset against the original six-week benchmark
| Strategy | Daily reset (this page) | Carried (Benchmark) |
|---|---|---|
| Perfect foresight | $5,921,103 | $5,926,534 |
| Rolling persistence (MPC) | $2,564,692 | $2,443,858 |
| Day-ahead persistence | $2,261,777 | $2,254,116 |
| Fixed price thresholds | $1,171,197 | $1,594,771 |
| Time-of-day schedule | $3,748,713 | $3,748,713 |
The benchmark lets each strategy carry stored energy past midnight, which helps the threshold rule most. The time-of-day plan ends every day empty, so its total is the same under both protocols. Rolling persistence plans only to midnight here and 24 hours ahead on the Benchmark. Perfect foresight's carried figure is the joint six-week optimum.
Time-of-day schedule captured 63.4% of the perfect-foresight revenue (95% CI 50.9% to 72.5%). Re-planning every half-hour on the persistence forecast was not clearly different from planning once at midnight: $7,389 a day (95% CI −$2,537 to $19,197).
Per strategy
Revenue, regret and share captured
| Strategy | Total | Per day | Regret per day | Share captured | Days losing money | Holding energy at midnight |
|---|---|---|---|---|---|---|
| Perfect foresightforesight | $17.8M | $98,546$63k to $147k | – | 100% | 0 of 1810% to 2% | 1 of 181mean 1 MWh |
| Rolling persistence (MPC)forecast | $10.3M | $56,830$32k to $91k | $41,716$29k to $58k | 57.7%49.0% to 64.4% | 8 of 1812% to 8% | 4 of 181mean 7 MWh |
| Day-ahead persistenceforecast | $8.95M | $49,441$28k to $82k | $49,105$32k to $69k | 50.2%39.8% to 59.0% | 14 of 1815% to 13% | 1 of 181mean 1 MWh |
| Fixed price thresholdsrule | $4.92M | $27,187$2.6k to $59k | $71,358$58k to $89k | 27.6%3.9% to 41.4% | 114 of 18156% to 70% | 170 of 181mean 527 MWh |
| Time-of-day schedulerule | $11.3M | $62,453$33k to $104k | $36,092$27k to $47k | 63.4%50.9% to 72.5% | 9 of 1813% to 9% | 0 of 181mean 0 MWh |
Share captured is total policy revenue ÷ total perfect-foresight revenue, resampled by day. Regret per day is never negative under daily reset. Days losing money carry Wilson intervals. Holding energy at midnight: days ending with more than 1 MWh stored, energy bought and never sold, which daily reset values at nothing.
Share captured
Out of the perfect-foresight revenue
- Rolling58% [49%, 64%]
- Day-ahead50% [40%, 59%]
- Thresholds28% [4%, 41%]
- Time of day63% [51%, 73%]
Dots are estimates, bars 95% intervals. 100% would match perfect foresight.
Paired by day
Three questions
Does a fixed daily schedule beat re-planning on yesterday's prices?
Time of day minus Rolling: $5,624 a day (95% CI −$3,379 to $17,803). Time of day earned more on 96 of 181 days (53%, 95% CI 46% to 60%).
no clear differenced_z 0.06r_rb 0.13Wilcoxon p (Holm) 0.23
Does re-planning every half hour beat planning once at midnight?
Rolling minus Day-ahead: $7,389 a day (95% CI −$2,537 to $19,197). Rolling earned more on 104 of 181 days (57%, 95% CI 50% to 64%).
no clear differenced_z 0.07r_rb 0.13Wilcoxon p (Holm) 0.23
Is trading on the clock better than trading on price levels?
Time of day minus Thresholds: $35,266 a day (95% CI $25,595 to $46,751). Time of day earned more on 163 of 181 days (90%, 95% CI 85% to 94%).
Time of day higherd_z 0.52r_rb 0.82Wilcoxon p (Holm) < 0.001
d_z is the mean daily difference divided by its standard deviation; r_rb is the matched-pairs rank-biserial correlation (−1 to 1). Questions: 3, Holm-adjusted together.
Horizon check
Planning to midnight, or past it
Under daily reset, energy still stored at midnight earns nothing, so the rolling policy above plans only to midnight. The original rolling policy (Benchmark, Dispatch Lab) plans 24 hours ahead from every half-hour, so from the afternoon on it buys energy to sell tomorrow morning, which this protocol never pays for. Same days, both versions:
| Rolling plans | Share captured | Days losing money | Holding energy at midnight |
|---|---|---|---|
| To midnightevaluated above | 57.7%49.0% to 64.4% | 8 of 181 | 4 of 181mean 7 MWh |
| 24 hours aheadoriginal policy | 52.1%41.3% to 59.8% | 31 of 181 | 81 of 181mean 165 MWh |
Planning to midnight earned $5,485 a day more (95% CI $3,966 to $7,178). The questions that involve rolling persistence, both ways:
- Time of day minus Rolling a day: $5,624 (95% CI −$3,379 to $17,803) planning to midnight; $11,108 (95% CI $2,388 to $23,091) with the 24-hour plan.
- Rolling minus Day-ahead a day: $7,389 (95% CI −$2,537 to $19,197) planning to midnight; $1,904 (95% CI −$7,703 to $13,106) with the 24-hour plan.
Forecast error
Does a worse forecast cost more?
Regret on the day ($, symmetric-log scale)
| Policy | Spearman ρ |
|---|---|
| Rollingplans on the forecast | 0.810.73 to 0.87 |
| Day-aheadplans on the forecast | 0.780.66 to 0.84 |
| Thresholdsignores it | 0.630.50 to 0.72 |
| Time of dayignores it | 0.540.35 to 0.67 |
Volatile days are both hard to forecast and lucrative to trade, so regret can rise with forecast error even for a policy that never looks at the forecast. The rule-based policies are the control: the correlation for rolling persistence minus the one for time of day is 0.27 (95% CI 0.16 to 0.42), so forecast error hurts the policy that plans on the forecast more. Median forecast error: $21/MWh.
Day-ahead persistence captured 55.2% of the perfect-foresight revenue (95% CI 48.3% to 60.7%). Re-planning every half-hour on the persistence forecast earned less than planning once at midnight: −$26,116 a day (95% CI −$59,224 to −$7,323).
Per strategy
Revenue, regret and share captured
| Strategy | Total | Per day | Regret per day | Share captured | Days losing money | Holding energy at midnight |
|---|---|---|---|---|---|---|
| Perfect foresightforesight | $56.8M | $155,607$111k to $220k | – | 100% | 0 of 3650% to 1% | 33 of 365mean 36 MWh |
| Rolling persistence (MPC)forecast | $21.8M | $59,777$42k to $89k | $95,830$65k to $143k | 38.4%27.3% to 48.1% | 39 of 3658% to 14% | 35 of 365mean 44 MWh |
| Day-ahead persistenceforecast | $31.4M | $85,893$55k to $131k | $69,714$54k to $91k | 55.2%48.3% to 60.7% | 34 of 3657% to 13% | 32 of 365mean 36 MWh |
| Fixed price thresholdsrule | $10.2M | $27,984$22k to $34k | $127,623$84k to $192k | 18.0%12.0% to 26.2% | 102 of 36524% to 33% | 208 of 365mean 294 MWh |
| Time-of-day schedulerule | $29.3M | $80,368$47k to $127k | $75,239$60k to $95k | 51.6%40.9% to 58.7% | 54 of 36512% to 19% | 0 of 365mean 0 MWh |
Share captured is total policy revenue ÷ total perfect-foresight revenue, resampled by day. Regret per day is never negative under daily reset. Days losing money carry Wilson intervals. Holding energy at midnight: days ending with more than 1 MWh stored, energy bought and never sold, which daily reset values at nothing.
Share captured
Out of the perfect-foresight revenue
- Rolling38% [27%, 48%]
- Day-ahead55% [48%, 61%]
- Thresholds18% [12%, 26%]
- Time of day52% [41%, 59%]
Dots are estimates, bars 95% intervals. 100% would match perfect foresight.
Paired by day
Three questions
Does a fixed daily schedule beat re-planning on yesterday's prices?
Time of day minus Rolling: $20,591 a day (95% CI $637 to $53,336). Time of day earned more on 186 of 365 days (51%, 95% CI 46% to 56%).
Time of day higherd_z 0.08r_rb 0.07Wilcoxon p (Holm) 0.23
Does re-planning every half hour beat planning once at midnight?
Rolling minus Day-ahead: −$26,116 a day (95% CI −$59,224 to −$7,323). Rolling earned more on 146 of 365 days (40%, 95% CI 35% to 45%).
Rolling lowerd_z −0.09r_rb −0.28Wilcoxon p (Holm) < 0.001
Is trading on the clock better than trading on price levels?
Time of day minus Thresholds: $52,384 a day (95% CI $19,835 to $98,294). Time of day earned more on 274 of 365 days (75%, 95% CI 70% to 79%).
Time of day higherd_z 0.16r_rb 0.57Wilcoxon p (Holm) < 0.001
d_z is the mean daily difference divided by its standard deviation; r_rb is the matched-pairs rank-biserial correlation (−1 to 1). Questions: 3, Holm-adjusted together.
Horizon check
Planning to midnight, or past it
Under daily reset, energy still stored at midnight earns nothing, so the rolling policy above plans only to midnight. The original rolling policy (Benchmark, Dispatch Lab) plans 24 hours ahead from every half-hour, so from the afternoon on it buys energy to sell tomorrow morning, which this protocol never pays for. Same days, both versions:
| Rolling plans | Share captured | Days losing money | Holding energy at midnight |
|---|---|---|---|
| To midnightevaluated above | 38.4%27.3% to 48.1% | 39 of 365 | 35 of 365mean 44 MWh |
| 24 hours aheadoriginal policy | 31.0%20.2% to 41.4% | 73 of 365 | 176 of 365mean 218 MWh |
Planning to midnight earned $11,535 a day more (95% CI $9,478 to $13,750). The questions that involve rolling persistence, both ways:
- Time of day minus Rolling a day: $20,591 (95% CI $637 to $53,336) planning to midnight; $32,126 (95% CI $11,598 to $65,411) with the 24-hour plan.
- Rolling minus Day-ahead a day: −$26,116 (95% CI −$59,224 to −$7,323) planning to midnight; −$37,651 (95% CI −$71,424 to −$18,068) with the 24-hour plan.
Forecast error
Does a worse forecast cost more?
Regret on the day ($, symmetric-log scale)
| Policy | Spearman ρ |
|---|---|
| Rollingplans on the forecast | 0.690.62 to 0.74 |
| Day-aheadplans on the forecast | 0.460.36 to 0.56 |
| Thresholdsignores it | 0.130.00 to 0.25 |
| Time of dayignores it | 0.130.01 to 0.25 |
Volatile days are both hard to forecast and lucrative to trade, so regret can rise with forecast error even for a policy that never looks at the forecast. The rule-based policies are the control: the correlation for rolling persistence minus the one for time of day is 0.55 (95% CI 0.44 to 0.66), so forecast error hurts the policy that plans on the forecast more. Median forecast error: $45/MWh.
03 · AI explanations
Does the optional AI get the facts right?
The Dispatch Lab's “Explain this schedule” is tested on its own page: 24 fixed schedules, the Lab's prompt, six automatic checks against the optimiser's numbers (tested first on planted faults), Wilson intervals, and paired comparisons between runs decided by an exact test. It runs with your own key; this site publishes no AI results of its own.
Evaluate the explanations04 · Reading these numbers
What the intervals do and do not cover
- They describe sampling variability across days: how much the result could move with a different run of similar days. They do not cover the model being wrong (a price taker with perfect availability and no FCAS income, see limitations).
- Neighbouring days are not independent (heat waves, outages and windless weeks last several days). Resampling whole weeks keeps that structure; an ordinary bootstrap would give intervals that are too narrow. The Wilson intervals on shares of days and the Wilcoxon p-values do treat days as independent, so they are optimistic: the block-bootstrap intervals are the primary evidence.
- Each sample is one stretch of history. The 2021 test window is only six winter weeks, and 2025 is a different market, with negative midday prices most spring days.
- The three paired comparisons in each sample are adjusted together (Holm); the comparisons are not adjusted across samples or across the many other numbers on this page. Read the p-values as a guide alongside the effect sizes and intervals, not as verdicts.
- The daily-reset protocol starts every day empty and values energy left at midnight at nothing, so it does not credit a policy for carrying cheap energy into the next morning. Every policy that plans ahead therefore plans only to midnight here (see the horizon check under Regret), and the threshold rule, which plans nothing, still ends many days holding energy. The six-week totals on Benchmark carry charge over and remain the brief's official comparison.