Halt & Step

OmniLoop provides a robust execution control model that allows you to halt, step, and resume your training or control loops dynamically. This is particularly useful when you need to inspect loop state, apply mutations, or debug exceptions without restarting the entire process.

How it Works

The halt, step, and resume model is evaluated at a low level in the Rust core’s registry.rs.

The barrier for execution is either tracker.wait_if_halted() or loop.tick(). When the loop is halted, the training process actually blocks at this barrier. The Python thread is suspended at the OS/Rust level, rather than spinning and consuming CPU.

The OmniLoop Dashboard sends halt, resume, and step commands over a websocket to the server. These commands are placed into the command ring, where they are picked up by the Rust core to control the execution state.

Step Mode

In step mode, the loop is allowed to advance exactly one iteration. After that single iteration completes, the loop automatically re-halts. This is ideal for fine-grained debugging and observation.

Freeze-on-Exception

When an exception is raised inside a tick() block or a function decorated with @track_loop, OmniLoop prevents the loop from crashing immediately. Instead, it freezes the loop for inspection. The exception’s traceback is captured and sent to the dashboard, allowing you to review the error context.

Once you have investigated or mitigated the issue, you can resume execution directly from the dashboard.

Example Usage

Using the Tracker Barrier

If you are managing the tracker manually, insert wait_if_halted() at the top of your loop:

for it in range(num_iterations):
    tracker.wait_if_halted()      # <- blocks here while halted
    mutations = tracker.get_mutations()
    # ... training iteration ...
    tracker.publish_telemetry_raw(payload)

Using TrainingLoop

When using the higher-level TrainingLoop API, tick() automatically handles the barrier:

with loop.tick():                 # <- blocks here while halted
    train_one_iteration()