Computational Steering

Interacting with a running simulation instead of waiting for it has a name and a literature. It is called computational steering, it goes back to the HPC visualisation systems of the 1990s and 2000s — SCIRun, CUMULVS, the steering work around VisIt — and the motivation has not changed since:

A long solve is a hypothesis you cannot revise until it finishes.

What changed is that the tooling mostly went away. The systems that implemented steering were coupled to specific solvers, specific schedulers and specific visualisation stacks, and when those aged out the capability went with them. The modern default is a batch job, a log file, and finding out at 04:00 that a timestep was slightly too large.

OmniLoop is a steering system that does not know it is one. The primitives it built for robot control loops — bind a live variable, clamp it against an envelope, arm a predicate, freeze without dying, journal for replay — are the primitives steering needs, under different names.

Steering vocabulary	OmniLoop
Steerable parameter	tunable=[...], bound live to your config object
Steering envelope / admissible range	limits=, enforced in-process against every edit
Convergence monitor	loop.log(...) metrics, streamed at tick rate
Breakpoint on a physical criterion	Watchpoints
Pause / resume the solve	Halt & Step
Checkpoint on trouble	checkpoint= on freeze
Provenance of a steered run	.omni journal — every edit, with who and when

The last row is the one that matters for anything you intend to publish or certify. A steered run is a run somebody interfered with, and “what did you change, and when” stops being a curiosity the moment a result rests on it. Every mutation crossing the command ring is journaled with its value, its tick and its principal, and omniloop why run.omni --tick N walks it back.

The worked example

examples/simulation/steered_solver.py is a 1-D transient heat solve — no ML, no robot, no framework:

omniloop up                                     # terminal 1
python examples/simulation/steered_solver.py    # terminal 2

It integrates dT/dt = alpha * d²T/dx² + q(x) explicitly (FTCS), which is conditionally stable: the Fourier number

Fo = alpha * dt / dx²

must stay at or below 0.5. Above it the scheme does not converge slowly, it oscillates and runs away to inf — typically thousands of sweeps after the mistake, so the traceback names a line that is not the problem. That is precondition #4 from Is OmniLoop for me? in its purest form: a failure that is a bad number long before it is an exception.

The instrumentation is nine lines:

from omniloop import Loop

loop = Loop(
    cfg,
    tunable=["dt", "alpha", "source_strength", "source_center"],
    limits={"dt": (1e-5, DT_CEILING)},   # the stability envelope, enforced
    metrics=["sim_time", "residual", "fourier", "peak_temp"],
    journal="heat_solve.omni",
    checkpoint=lambda path: dump_field(path, field, cfg),
)
loop.watch("fourier", "greater_than", 0.5)   # trip when instability is certain
loop.watch_all_nonfinite(True)               # backstop for everything else

and the loop body is your existing sweep, wrapped:

while residual > CONVERGED:
    with loop.tick():
        for _ in range(SUBSTEPS):
            field, change = sweep(field, cfg, q)
        loop.log(residual=..., fourier=..., peak_temp=max(field))

The envelope holds across the whole slider space, not the current state

DT_CEILING in the example is computed as dx² / (2 * ALPHA_MAX) — the CFL bound for the most diffusive material the alpha slider will allow, not for the alpha currently set.

This is the general rule for limits and it is easy to get wrong: a bound computed against the present value of another tunable is a bound that turns illegal the moment someone drags that other slider. An envelope has to hold across the whole reachable space, because the two sliders move independently and nothing sequences them.

With the envelope armed, dragging dt to the top of its range clamps at the bound and the run continues; the dashboard logs the clamp. Run with --unsafe to drop the limit and see the other half:

python examples/simulation/steered_solver.py --unsafe --dt 1e-2

[solve] sweep     100 | t=1.000s | residual=6.6608e+19 | Fo=0.650 | dt=1.00e-02

[solve] FROZEN by watchpoint: {'name': 'fourier',
                               'condition': 'greater_than 0.5',
                               'value': '0.6502500000000001'}

The solve is now frozen with the field intact rather than dead with a OverflowError. Lower dt in the dashboard, resume, and it carries on from where it was — which on an eight-hour solve is the entire point.

A halted loop prints nothing on its own

While blocked at the barrier your process produces no output, which looks exactly like a hang. The example calls loop.watch_trip() after each tick and prints the trip itself; do the same in yours. omniloop status distinguishes halted from crashed, and reports a deliberately frozen loop as a note rather than a fault.

Your tick is your resolution

The example runs 100 sweeps inside one tick(), and the trip above fired after the field had already reached 6.6e19 — the watchpoint sees state at tick boundaries, so divergence that happens within a tick is caught at the end of it.

That is the trade every steered solve makes. One sweep per tick gives sweep-resolution tripwires and pays a barrier per sweep; a hundred sweeps per tick amortises the barrier and blurs detection to a hundred sweeps. Size it against how fast your instability actually develops, and measure the barrier cost before assuming it matters — see Performance.

Running on a cluster

The solve is usually not on your laptop. The topology is the same one a robot uses and is documented as such: bind the relay to loopback on the compute node, forward the port over SSH, open the dashboard locally. See Tethered Deployment → a training or compute node, including the ssh -J form for a node reachable only through a login host.

What this does not do yet

Stated plainly, because a steering page that implies a solved problem is worse than no page:

• No MPI rank fan-in. Each instrumented process publishes into its own session and omniloop up --watch a,b,c fans a handful onto one time axis. That works for a rank-0 monitor plus a few neighbours. It is not a design for steering 4,096 ranks, and pretending otherwise would waste your afternoon. Instrument the ranks you would actually watch.
• No field visualisation. The dashboard plots scalars and renders URDF. A temperature field, a mesh or a volume is not something it draws — export to Rerun or MCAP for that and keep OmniLoop for the control plane.
• Python-side overhead is per-tick, not per-sweep. If your inner sweep is already in C, Fortran or CUDA, keep it there and put the tick around the chunk; the C++ and Rust SDKs bind native double members zero-copy if you would rather not cross into Python at all.
• Determinism is your solver’s, not ours. Replay reproduces the inputs and the edits faithfully. Whether replaying them reproduces your numbers depends on your reductions, your thread count and your hardware — divergence detection tells you when it did not, which is the honest thing to offer.