Flight Recorder

The crash you cared about is always in the run you weren’t journaling. To solve this, OmniLoop features a Flight Recorder — an always-on, in-memory ring buffer that acts as a black box for your loop.

The tracker retains the last $N$ telemetry frames (defaulting to 512) along with the mutation and control events received over IPC. If a crash happens, the causes leading up to it are already captured.

Automatic Crash Dumps

When a freeze-on-exception occurs (handled automatically by both @track_loop and TrainingLoop), the flight recorder ring is automatically dumped to disk in the working directory as omniloop_blackbox_<unix-ts>.omni.

This file is a complete, indexed v2 journal. You can load it directly into the dashboard’s replay view to scrub through the final moments before the exception, or hand it over as a bug report.

SDK API

You can configure and interact with the flight recorder programmatically:

import json
from omniloop import tracker

# Set capacity to 2048 frames (~1 minute at 30Hz). Set to 0 to disable.
tracker.set_flight_recorder_capacity(2048)

# Check how many frames are currently recorded
length = tracker.flight_recorder_len()

# Dump the current buffer to disk on demand (non-destructive)
tracker.dump_flight_recorder("bug_report.omni")

# Record custom SDK-level causal events
tracker.record_flight_event("control", json.dumps({"reason": "manual_trigger"}))

For those looking at the Rust internals, this is managed by the FlightRecorder struct in omniloop-core/src/journal.rs, which provides the new, record, dump, and set_capacity methods.