Flight Recorder

The crash you cared about is always in the run you weren’t journaling. To solve this, OmniLoop features a Flight Recorder — an always-on, in-memory ring buffer that acts as a black box for your loop.

The tracker retains the last $N$ telemetry frames (defaulting to 512) along with the mutation and control events received over IPC. If a crash happens, the causes leading up to it are already captured.

Memory budget

The flight recorder’s peak RSS contribution is bounded and predictable:

$$\text{memory} \approx \text{capacity} \times \overline{\text{frame_bytes}}$$

where $\overline{\text{frame_bytes}}$ is the average size of one serialised telemetry payload (the UTF-8 JSON string published via publish_telemetry_raw or loop.log). The ring reuses existing allocations as old frames are evicted, so steady-state memory stays flat.

As a rule of thumb:

Scenario	Payload size	Capacity	Peak RSS
Default (512 frames, typical RL loop ~500 B)	~500 B	512	~250 KB
30 Hz controls loop, 5 min retention	~200 B	9 000	~1.8 MB
100 Hz dense sensor log, 60 s retention	~2 KB	6 000	~12 MB

The payload size is whatever your loop publishes each tick. You can check it live:

from omniloop import tracker

# Bytes of the most recently published telemetry frame (publisher side)
print(tracker.stats().get("telemetry_last_bytes"))

To raise or lower the capacity — or disable the recorder entirely — call set_flight_recorder_capacity before the loop starts:

# ~1 minute at 30 Hz with a 200-byte payload ≈ 360 KB
tracker.set_flight_recorder_capacity(2048)

# Disable to reclaim memory (disables auto-dump on freeze too)
tracker.set_flight_recorder_capacity(0)

Automatic Crash Dumps

When a freeze-on-exception occurs (handled automatically by both @track_loop and TrainingLoop), the flight recorder ring is automatically dumped to disk in the working directory as omniloop_blackbox_<unix-ts>.omni.

This file is a complete, indexed journal. You can load it directly into the dashboard’s replay view to scrub through the final moments before the exception, or hand it over as a bug report.

The blackbox dump preserves telemetry, not your in-memory tensors. To also serialize model/optimizer state at the freeze point — so a multi-hour training run isn’t lost when it faults — pass checkpoint= to TrainingLoop. See State checkpoint on freeze.

SDK API

You can configure and interact with the flight recorder programmatically:

import json
from omniloop import tracker

# Set capacity to 2048 frames (~1 minute at 30Hz). Set to 0 to disable.
tracker.set_flight_recorder_capacity(2048)

# Check how many frames are currently recorded
length = tracker.flight_recorder_len()

# Dump the current buffer to disk on demand (non-destructive)
tracker.dump_flight_recorder("bug_report.omni")

# Record custom SDK-level causal events
tracker.record_flight_event("control", json.dumps({"reason": "manual_trigger"}))

For those looking at the Rust internals, this is managed by the FlightRecorder struct in omniloop-core/src/journal/recorder.rs, which provides the new, record, dump, and set_capacity methods.