Watchpoints

Watchpoints are predicates over your loop state that are evaluated directly in the Rust core on every state sync. There is no Python code in the hot path and no round-trip to the dashboard before a halt occurs.

When a watchpoint condition fires (or “trips”), the loop freezes at its next wait_if_halted() or tick() barrier. The trip—containing the variable name, condition, and value—is recorded, outgoing telemetry is annotated, and a watch_trip control event is written to the journal.

Setting Watchpoints via SDK

You can define watchpoints directly in your code using the OmniLoop tracker. Available conditions include "non_finite", "less_than", and "greater_than".

from omniloop import tracker

# Halt if reward_mean drops below -50.0
tracker.add_watchpoint("reward_mean", "less_than", -50.0)

# Halt if joint_torque_3 becomes NaN or infinite
tracker.add_watchpoint("joint_torque_3", "non_finite")

# Global tripwire: halt on the FIRST NaN/inf in ANY tracked variable
tracker.set_watch_all_nonfinite(True)

Managing Trips

When the loop halts due to a watchpoint, you can inspect and clear the trip:

trip = tracker.peek_watch_trip()   # Returns dict: {"name", "condition", "value"} or None
tracker.take_watch_trip()          # Reads and clears the current trip
tracker.clear_watchpoints()        # Removes all configured watchpoints

Note

Resuming the loop from the dashboard automatically acknowledges and clears the trip state.

Setting Watchpoints from the Dashboard or Server

You can also dynamically set watchpoints from the dashboard side or via the server script:

tracker.publish_watchpoint("reward_mean", "less_than", -50.0)
tracker.publish_watch_clear()

Over the websocket API, this looks like:

{"watch": {"name": "reward_mean", "condition": "less_than", "threshold": -50.0}}
{"watch": "clear"}

Watchable State

You can set a watchpoint on anything the registry sees. This includes registered pointers (such as MuJoCo reflection data) AND every scalar included in a publish_telemetry_raw() payload.

Performance

Watchpoints are highly optimized. The evaluation cost is a single HashMap walk per state sync in Rust. If no watchpoints are armed, the evaluation short-circuits instantly, introducing negligible overhead to your loop.

For those interested in the Rust internals, this surface is implemented via WatchCondition, Watchpoint, WatchTrip, and StateRegistry::{add_watchpoint, clear_watchpoints, evaluate_watchpoints, take_last_trip} in omniloop-core/src/registry.rs.