omniloop doctor

When the dashboard looks stale, there are usually three main suspects:

1. Loop not publishing: The training loop has stopped sending data.
2. Server can’t keep up: The server is polling too slowly, causing frames to be overwritten.
3. Session mismatch: The loop and server are communicating on different OMNILOOP_SESSION_IDs.

The OmniLoop IPC engine maintains strict counters for all traffic, allowing you to quickly diagnose these issues.

Diagnostic Counters

The following counters are tracked per engine handle:

Counter	Meaning
telemetry_published	Frames this handle wrote.
telemetry_last_bytes	Size of the last written frame.
telemetry_delivered	Frames this handle received (genuinely new frames).
telemetry_skipped	Frames that were overwritten before the consumer polled them.
telemetry_torn_reads	Number of seqlock retries due to concurrent writes.
commands_published	Commands written to the command ring.
commands_received	Commands read from the command ring.
command_ring_full	Rejected command publishes because the ring was full.
command_corruption_errors	Errors reading the command ring (corrupted cursor/length).

Checking Health

There are three ways to access these statistics depending on where you are debugging.

1. From Python (SDK)

You can inspect the counters for the current process’s handle:

stats = tracker.stats()
print(stats["telemetry_skipped"])

2. From the Server API

You can curl the stats endpoint of a running server:

curl http://127.0.0.1:8000/stats

3. Using the CLI

The fastest way to diagnose channel health is through the CLI diagnostic commands:

Quick Status Check (omniloop status)

Prints an instant one-line snapshot of shared-memory IPC and server reachability:

omniloop status

Full Diagnostic Audit (omniloop doctor)

Performs in-depth channel diagnostics, checks for multiple loop process collisions (multiple PIDs publishing into the same session), and queries server stats:

omniloop doctor

If it detects an issue (such as stagnant publishing, publisher PID collisions, or ring corruption), omniloop doctor exits with a non-zero status code, making it scriptable for CI health checks.

Shared Memory Cleanup (omniloop clean)

Purges stale shared-memory segments (/dev/shm/omniloop_*) left behind after abnormal process terminations:

omniloop clean