Troubleshooting

Start here when something looks wrong. Almost every symptom below is diagnosed by one command:

omniloop doctor

doctor exits non-zero on any fault, so it also works as a CI gate. Add --json for machine-readable output.

The dashboard looks stale or empty

The single most common report, and it has five distinct causes. doctor distinguishes them.

doctor says	Cause	Fix
holds a retained telemetry frame but NOTHING published a new one	The loop exited or crashed. Shared memory keeps the last frame written, so the dashboard shows it frozen forever	Restart the loop, or omniloop clean to drop the segment
no telemetry seen ... and no frame has ever been written	No loop is running, or it publishes under a different session id	Start the loop; check OMNILOOP_SESSION_ID matches on both sides
MULTIPLE loops are publishing	Two loops share one session and are overwriting each other	Stop all but one, or give each its own --session
server session 'x' != yours ('y')	Server and loop read different shared-memory segments	Start both with the same --session / OMNILOOP_SESSION_ID
no server listening	The dashboard has nothing to connect to	omniloop up

The first two look identical from the dashboard and are easy to confuse by hand — a segment whose publisher has died still reads back one valid frame. doctor separates them by ignoring the frame that was already sitting in the slot and watching for a new one.

If doctor reports the loop publishing at a healthy rate and the server session matches, but the dashboard still shows nothing, check the browser console — a websocket close code tells you the rest (see below).

doctor reports a lower rate than my loop runs at

It does not. The line to read is the publish rate:

[ok]  target loop is publishing at ~604 Hz (last frame 38 bytes)
      this probe sampled 19 of 614 frames over 1.0s (~19 Hz); sampling past
      frames is normal for a diagnostic and does not affect the loop.

doctor polls the telemetry slot 20 times a second, so on any loop faster than that it necessarily sees a fraction of the frames. The publish rate is reconstructed from the engine’s telemetry_skipped counter, which records exactly the frames that were overwritten between polls — so it stays accurate for a 1 kHz control loop. Only the “sampled” number is capped, and sampling past frames costs the target loop nothing.

Two experiments on one machine

Shared memory is keyed by session id. Two loops on the default session will silently fight over the same telemetry slot and command ring: mutations land on the wrong process, halts appear to do nothing, and telemetry flickers between two runs.

Give each stack its own session:

OMNILOOP_SESSION_ID=exp-a omniloop up --session exp-a

The dashboard header shows the session id it is attached to, so a mismatch is visible at a glance rather than diagnosed after the fact.

This applies to test suites too: a test that touches real shared memory while a training run is going will fail in confusing ways. OmniLoop’s own integration tests pin themselves to a unique session for exactly this reason.

The websocket closes immediately

Close code	Meaning	Fix
4401	Authentication failed or timed out	Check OMNILOOP_AUTH_TOKEN matches on server and client; the client must authenticate within OMNILOOP_WS_AUTH_TIMEOUT (default 5 s)
4403	Origin not allowed	Add your origin to OMNILOOP_ALLOWED_ORIGINS, or serve the dashboard from a loopback address

With no token configured, the server runs in zero-config local mode and every connection is the implicit local principal — you should not see 4401 at all.

Errors in the dashboard

Every error message carries a stable code alongside its human text:

Code	What to do
auth	Check the token; see the 4401 row above
capability	The server has the feature disabled (e.g. fork off a non-loopback bind) or your principal lacks the role
coercion	The value could not be coerced to the parameter’s declared type — check the schema’s type
path	The journal/trace file is missing, unreadable, or outside the allow-listed directories
quota	A limit was hit, e.g. too many concurrent forks
protocol	Malformed message or unknown action — usually a client/server version skew
internal	A server-side bug; the message carries the exception text

Stale shared memory after a crash

On Linux, a target process that died hard can leave segments behind in /dev/shm. Look before you purge:

omniloop clean --dry-run          # what would go
omniloop clean --session exp-a    # one session only
omniloop clean                    # everything stale

Segments written to in the last minute are skipped by default — removing one out from under a running loop corrupts its command ring, and on a shared machine “all sessions” otherwise includes somebody else’s run. --force overrides that. doctor prints this hint automatically when it runs on Linux.

On Windows the kernel reclaims shared memory with the last handle. On macOS POSIX shared-memory objects persist until reboot, but they are not exposed as files, so nothing can enumerate or unlink them — use a fresh --session id, or reboot.

A slider moves but nothing changes

Three possibilities, in order of likelihood:

1. The framework recomputes the value every step. An SB3 schedule or an adaptive-LR config overwrites your edit on the next update. The adapter pages document the exact config change needed to make edits stick — see ROS 2 and LeRobot.
2. A hard limit is clamping it. limits= is an enforced envelope, unlike bounds= which is only a slider hint. A clamp is journaled as a limit_clamp event and surfaced in the Event Console. See Limit Enforcement.
3. The edit is landing on a different process. See the session-collision row above.

A slider spans the wrong range

Sliders require explicit min, max, and step. Declaring one without them raises at declaration time, naming the parameter — a slider that silently spans 0..1 is a live-tuning hazard on real hardware. Either declare the bounds, or use kind="text" for an unbounded numeric entry box.

A journal from a crashed run

A process killed mid-write leaves a half-finished record on the end of the file, and no trailer. That journal is still the most valuable one you have, so every journal command reads as far as it can and reports the damage alongside the data rather than instead of it:

$ omniloop export-mcap crashed.omni
[ok]   exported 26 record(s) across 3 channel(s) -> 'crashed.mcap'
[fail] journal is damaged: stopped after 26 record(s) — Unknown record kind
       discriminator: 73. Everything above this line was recovered; the rest of
       the file is unreadable.
       the 26 recovered record(s) were still written to 'crashed.mcap'.
$ echo $?
2

Exit code 2 means “damaged, and here is the surviving prefix” — distinct from 1, which means the input could not be used at all. A salvaged MCAP is finished properly, so standard readers open it. omniloop inspect --json also carries intact and integrity fields for gating in CI.

Two conditions get their own message:

Message	Meaning
truncated: the trailer claims N record(s) but only M could be read	Bytes are missing. The trailer’s count is authoritative
no trailer — this journal was never closed	Still being written, or the run died before flushing. Counts are best-effort and the sparse tick index is unavailable

Replay plays at the wrong speed

Replay honours the journal’s recorded mono_ns timestamps, so a 200 Hz loop replays at 200 Hz. If the timeline shows N frames (no recorded timing), the journal is a legacy v1 file without per-record timestamps; playback falls back to a fixed interval. Re-record with a current build to get a real clock.

Exported .mcap won’t open in Foxglove

Real MCAP export needs the mcap extra:

pip install 'omniloop[mcap]'

Without it, omniloop export-mcap fails with an actionable message rather than writing a file Foxglove cannot read. Use --format jsonl if you want the dependency-free newline-delimited JSON dump instead.

Still stuck

omniloop inspect run.omni summarises any journal — format version, record counts, tick range, event histogram, and the final tuned values — without writing a line of code. It is the fastest way to confirm what a run actually recorded before digging further.