Watchpoints

Watchpoints are predicates over your loop state that are evaluated directly in the Rust core on every state sync. There is no Python code in the hot path and no round-trip to the dashboard before a halt occurs.

When a watchpoint condition fires (or “trips”), the loop freezes at its next wait_if_halted() or tick() barrier. The trip—containing the variable name, condition, and value—is recorded, outgoing telemetry is annotated, and a watch_trip control event is written to the journal.

Setting Watchpoints via SDK

You can define watchpoints directly in your code using the OmniLoop tracker. Available conditions include "non_finite", "less_than", and "greater_than".

from omniloop import tracker

# Halt if reward_mean drops below -50.0
tracker.add_watchpoint("reward_mean", "less_than", -50.0)

# Halt if joint_torque_3 becomes NaN or infinite
tracker.add_watchpoint("joint_torque_3", "non_finite")

# Global tripwire: halt on the FIRST NaN/inf in ANY tracked variable
tracker.set_watch_all_nonfinite(True)

Managing Trips

When the loop halts due to a watchpoint, you can inspect and clear the trip:

trip = tracker.peek_watch_trip()      # {"name", "condition", "value"} or None
tracker.take_watch_trip()             # Reads and clears the current trip
tracker.clear_watchpoint("reward")    # Disarms one name, leaving the rest intact
tracker.clear_watchpoints()           # Removes all configured watchpoints

Watchpoints are edge-triggered

A watchpoint fires on the transition into a bad state, not continuously while the state is bad. Once it trips it stays latched, and it re-arms itself the moment its condition next evaluates false.

This matters the first time you arm a threshold on a slow-moving metric. A collapsed policy needs several rollouts to climb back above a reward floor; a poisoned tensor stays NaN. If watchpoints were level-triggered, resuming would re-trip at the very next barrier and Resume would be a no-op — the loop would appear to hang.

So the normal recovery is just: fix the cause, press Resume. You do not need to disarm anything, and the watchpoint remains armed to catch the next genuine excursion.

tracker.add_watchpoint("reward_mean", "less_than", 10.0)

# reward_mean falls to 9.0 -> trips, loop halts.
# You lower the learning rate and resume.
# reward_mean is still 9.4, then 9.8 -> no re-trip; the run keeps going.
# reward_mean climbs to 50 -> the watchpoint silently re-arms.
# reward_mean later collapses to 4.0 -> trips again. A new excursion, a new halt.

Each excursion produces exactly one watch_trip event in the journal, so the causal record stays one-event-per-incident rather than one-per-tick.

If a metric oscillates right on its threshold you will get one trip per crossing. Pick a threshold with some margin below normal operating range rather than at the edge of it.

Setting Watchpoints from the Dashboard or Server

You can also dynamically set watchpoints from the dashboard side or via the server script:

tracker.publish_watchpoint("reward_mean", "less_than", -50.0)
tracker.publish_watch_clear()

Over the websocket API, this looks like:

{"watch": {"name": "reward_mean", "condition": "less_than", "threshold": -50.0}}
{"watch": "clear"}

Watchable State

You can set a watchpoint on anything the registry sees. This includes registered pointers (such as MuJoCo reflection data) AND every scalar included in a publish_telemetry_raw() payload.

Performance

Watchpoints are highly optimized. The evaluation cost is a single HashMap walk per state sync in Rust. If no watchpoints are armed, the evaluation short-circuits instantly, introducing negligible overhead to your loop.

For those interested in the Rust internals, this surface is implemented via WatchCondition, Watchpoint, WatchTrip, and StateRegistry::{add_watchpoint, clear_watchpoints, evaluate_watchpoints, take_last_trip} in omniloop-core/src/registry.rs.

Group watchpoints (arrays)

Robot state is vectors. Arming a tripwire per element works, but it is n registrations to write, n to disarm, and n independent latches — so several joints leaving their envelope together produce a burst of trips that all describe one excursion.

A name ending in _* arms one group watchpoint over every prefix_<n> channel registered so far:

tracker.add_watchpoint("joint_pos_*", "greater_than", 2.0)

lp.watch_array_within("joint_pos", -2.0..=2.0)?;   // Rust SDK

loop.watch_array_within("joint_pos", -2.0, 2.0);   // C++ SDK

One registration, one shared latch — “the arm left its envelope” is a single event — and a trip that still names the individual element, so you know which joint. Disarm the whole group with clear_watchpoint("joint_pos_*").

Members are resolved when the watchpoint is armed, not matched on every tick: prefix-matching every key per iteration would put string work on the hot path to save a lookup that is already a hash probe. Bind the array before arming the group. A group that matches nothing is refused, because a tripwire covering zero variables reads as armed and protects nothing.

* on its own remains the global non-finite switch, not a group.

Reading back what is armed

"armed" and "currently firing" are different questions, and neither should be answered from memory:

armed = tracker.list_watchpoints()
# [{"name": "joint_pos_*", "condition": "greater_than", "threshold": 2.0,
#   "latched": False, "members": 6}, ...]

The same list rides out on every telemetry frame under __watchpoints__, is served by the list_watchpoints websocket action, and is exposed to agents as omniloop_list_watchpoints. latched means the condition is satisfied now and has already fired — the thing an operator deciding whether it is safe to resume actually needs.

This exists for the same reason limits have a __bounds__ readback: a safety condition nobody can read back is one nobody can verify is armed. On a machine that can move, “which tripwires are live right now” has a physical answer.