Accurate, and Still Wrong

Two optimization models, linked through a cheap stand-in for the first. The stand-in fits well and still sends the second model the wrong way.

This is a teaching toy, not research. Both models are tiny, every number is made up for this page, and none of it describes a real grid, plant or supply chain. No number it produces should be cited. It exists to make one mechanism visible.

The setup

A planning model picks the cheapest way to meet a fixed demand for power from four candidate plans, from solar with storage to all gas. Solar panels and batteries need far more metal than gas plants, so as the price of materials rises the cheapest plan jumps from one to the next. The metal the chosen plan needs is handed to a supply-chain model, which picks which of three processing plants to open to supply it.

In real studies the planning model takes hours to run, so it is run a handful of times and a surrogate (here a Gaussian process, fitted by maximum likelihood) is trained on those runs to predict its output at any price. The supply-chain model then plans on the surrogate's prediction instead of the real thing.

What this is showing

The black line is a staircase because a planning model's answer does not slide; it jumps between a few corner solutions as the price moves. The surrogate is smooth. With eight runs it explains most of the variation and looks like a good fit. But each step's flat top sits just above or below a point where the supply-chain model needs another plant, so a prediction off by a few percent lands on the wrong side and opens the wrong plants. The red bands are those prices. Where the plan comes up short, the metal has to be bought in a hurry at a premium.

Try it: add runs and watch the fit improve faster than the decisions do. Resample and see how much depends on where the runs happened to land. Then move the price into a red band and read the cost of acting on the surrogate.

Why it matters

Linking models is how one study's output becomes another's input: a grid plan becomes a materials forecast, which becomes a supply chain. When the first model is too slow to run everywhere, a surrogate stands in for it. This toy's lesson is that the usual test of a surrogate, how well it predicts, is the wrong test when its prediction feeds a decision. What matters is whether the next model makes the same choice it would have made with the real thing: the same plants, the same binding limits, a plan that can actually be carried out. Judging a surrogate on the decision, and choosing where to run the expensive model by what it does to the decision, is open research.

What it leaves out

Nearly everything that makes real linked models hard. One input instead of dozens; four candidate plans instead of a continuous build-out; three plants and no geography, transport or time. Real planning models are linear or mixed-integer programs with thousands of variables, and real supply-chain models have many tiers. The surrogate here is a textbook Gaussian process on a one-dimensional Latin hypercube; the "add a run" button uses the textbook rule of sampling where the surrogate is least certain.

The approach builds on the design and analysis of computer experiments (DACE) and surrogate-optimization work of UT Arlington's COSMOS center (V. Chen, J. Rosenberger and colleagues). Model: src/scripts/surrogateLinkModel.js, with its tests beside it, no dependencies. Companion toy: What a Grid Is Made Of, where you choose the mix instead of a model.