Run many scenarios at once — haql-engine scenario guide

Feature: parallel scenario execution · Available in run_uncertainty.py · Status: shipped

The idea, in one picture

Imagine a toll plaza with four booths and a queue of twenty cars. Each car still takes the same time to pass through a booth. But with four booths open, the queue clears sooner than with one.

A scenario study works the same way. Each simulation is a car; the workers are the booths. The engine does not make an individual simulation faster. It runs independent simulations at the same time.

Before One at a time

A 20-scenario study waits for each scenario to finish before starting the next.

Now Up to N at once

With four workers, four independent scenarios can be in progress together.

This matters because uncertainty studies are not one run. They are many plausible versions of the field, and the useful answer is the range they produce.

What changed

The scenario runner now accepts a worker count. The default is still one, so an existing study behaves exactly as before. Choosing a higher number opens more “booths” on the same machine.

Before

Sequential

One scenario runs, writes its result, then the next begins. Simple, but the machine is mostly waiting.

Now

Process-level parallel

Several independent scenario processes run together. Each one has its own output and receipt.

Still true

Same individual physics

The solver is unchanged. Parallelism does not alter a single scenario's equations or timestep decisions.

What you get — and what you don't

You get	You do not get
Shorter wall-clock time for a set of independent scenarios, when the machine has spare cores and memory	A faster single simulation; the solver remains single-threaded
The same inputs, physics and checks for every scenario	Different numerical results just because the worker count changed
Every failed or incomplete scenario kept visible in the study record	Automatic deletion of difficult cases to make the study look cleaner
A stable result order and a separate digital receipt for each run	A cluster scheduler, MPI, GPU execution or distributed computing

In one sentence: parallel running changes how quickly the set is completed, not what an individual simulation means.

A simple example

Suppose a study has 20 scenarios and each takes 10 minutes.

Run one at a time: about 200 minutes of waiting.
Run four at a time: about 50 minutes in an ideal world.

The real number will be different. Disk speed, memory, processor contention and the size of each scenario all matter. The example shows the shape of the benefit, not a guarantee.

For a client: the useful question is not “how many workers?” but “what machine will run the study, and how long does one scenario take?” Those two facts determine the realistic completion time.

What stays safe

A scenario study is only useful if the evidence survives the speed-up. Four things remain true:

Each scenario is isolated. A separate process writes its own result files, log and digital receipt. One scenario cannot corrupt another's output.
The final order is deterministic. Scenarios are listed by their planned index, not by whichever worker happens to finish first.
The result is traceable. Inputs and settings are fingerprinted before execution, and the output is recorded. Changing one byte changes the identity.
Failures stay visible. A scenario that cannot produce a valid result is recorded as failed, not quietly dropped from the summary.

Speed is useful only when the result can still be defended. The audit trail is part of the feature, not an extra report added afterwards.

If a scenario fails

A failed scenario is kept in the study record with its status and an explanatory artifact. The study is marked partial rather than complete. That distinction matters: a clean average over only the easy cases is not an honest uncertainty study.

A later resume run can reuse the scenarios that completed and retry the ones that did not. It does not reuse a result merely because a file exists; the input and settings must still match.

Do not judge a study by its average alone. The failed or water-filled case may be the one that changes the decision. Keeping it visible is the point.

What this feature is not

Not a faster solver. A single simulation still takes the same time.
Not a cluster. Workers run on one machine. Distributed execution and job-scheduler integration are separate work.
Not infinite. Each worker needs memory and disk space. Oversubscribing the machine can make the study slower.
Not a replacement for engineering judgement. More scenarios do not make a weak model trustworthy.

For operators

If you run the engine yourself, the command is:

python3 run_uncertainty.py \
  --input uncertainty_input.example.json \
  --out uncertainty_out \
  --T 365 \
  --workers 4

--workers defaults to 1; values below 1 are rejected. The run writes a separate process, log and digital receipt per scenario. A failed scenario produces a parseable failure artifact and leaves the experiment partial.

Choosing a worker count

Start with two workers on a laptop and increase only while the machine remains responsive. A practical ceiling is the number of physical processor cores available to the engine; memory usually becomes the real limit first.

Use a dedicated output directory and receipt store for a large study. Do not delete evidence while a study is running.

Full command options and the technical failure contract are in the technical reference.

Evidence

The regression suite includes a two-scenario study run with two workers. It verifies that:

both scenarios complete;
no failure is reported;
the experiment lists results in planned order 0, 1;
both scenario records are marked complete;
the forecast input is created.

That is a small test, but it checks the properties that matter: isolation, ordering, completion reporting and a usable downstream study.

Current limits

Area	Limit
Execution	One host. No MPI, GPU, Kubernetes or Slurm integration.
Solver	The individual simulation remains single-threaded.
Speed-up	Depends on cores, memory, disk and scenario size; it is not automatically proportional to the worker count.
Operations	No live dashboard or automatic resource scheduler; it is a command-line study runner.

Next infrastructure step: a scheduler adapter and per-scenario resource accounting. That is a separate feature, not part of this release.

Run many scenarios at once. Keep every result.