Journals & Replay

OmniLoop journals record the entire lifecycle of a training session. Journals are not just a stack of telemetry frames; they interleave typed event records into the same stream, providing a unified timeline of data and control.

Journal Records

Every record in the journal carries the tick of the telemetry frame it belongs to.

Kind	Payload (JSON)	Written when
telemetry	the state frame	every published tick
mutation	{"name", "old", "new", "source"}	a parameter changes
control	`{“event”: “freeze_on_exception”	“watch_trip”
lifecycle	`{“event”: “session_start”	“session_end”, …}`
hash	8-byte LE FNV-1a 64	every hash_every ticks

TrainingLoop writes all of this automatically when initialized with a journal= path.

Reading Back Data

You can load and inspect journal data offline using the OmniLoop API:

from omniloop import load_events, JournalPlayer

# Iterate over events
for e in load_events("run.omni"):
    print(e["tick"], e["kind"], e["event"])

# Sequential playback
player = JournalPlayer("run.omni")
rec = player.read_next_record()
# Example output:
# {"kind": "telemetry", "tick": 0, "mono_ns": ..., "wall_ns": ..., "payload": b"..."}

Long Runs: Journal Rotation

By default a journal is one file that grows for as long as the run does. At 100 Hz that is roughly a gigabyte per hour of dense telemetry — and multi-hour runs are exactly the runs worth recording.

Pass a size or duration cap and the journal rolls over into segments instead:

loop = TrainingLoop.from_dataclass(
    cfg,
    tunable=["learning_rate"],
    journal="run.omni",
    journal_max_bytes=256 * 1024 * 1024,   # roll every 256 MB
    # journal_max_seconds=900,             # ...or every 15 minutes
)

which produces:

run.omni      the first segment — still the path you named
run.1.omni
run.2.omni

Each segment is finalized independently, with its own sparse index and trailer, so on-disk size is bounded by your cap rather than by run length. Nothing about reading changes: ticks keep counting across a rollover, and every offline API takes the base path and reads the whole set.

from omniloop import SegmentedPlayer, journal_segments

journal_segments("run.omni")      # ['run.omni', 'run.1.omni', 'run.2.omni']

player = SegmentedPlayer("run.omni")
player.record_count()             # total across all segments
player.read_next_record()         # one continuous stream

load_events(), cause_of(), effects_of(), find_divergence() and DeterministicReplay are already segment-aware and need no change.

A complete round trip, if you want to see it happen:

import json
from omniloop import SegmentedPlayer, StateJournal, journal_segments

journal = StateJournal("demo.omni", max_bytes=4096)
for tick in range(200):
    journal.write_frame(json.dumps({"tick": tick, "reward": tick * 0.5}).encode())
journal.close()

segments = journal_segments("demo.omni")
print(f"{len(segments)} segment(s): {segments}")

player = SegmentedPlayer("demo.omni")
print(f"{player.record_count()} records across the whole run")

first = player.read_next_record()
print(f"first tick: {first['tick']}")

Note

Rotation is opt-in. Without journal_max_bytes or journal_max_seconds a journal is written exactly as before — one file, flags == 0, byte-identical to every .omni file written to date.

Caution

Re-running into a path that already holds a rotated set clears the whole set first. A leftover run.3.omni from a longer previous run would otherwise be read back as a continuation of the new, shorter one.

Exporting Tuned Configurations

You can export the final mutated state of your parameters (a “tuned config”) either live or from a recorded journal:

# Live export during a run
loop.export_tuned_config("tuned.yaml")

# Offline export from a journal
from omniloop import export_tuned_config_from_journal
export_tuned_config_from_journal("run.omni", "tuned.yaml")

Two kinds of replay

OmniLoop offers both, and they answer different questions.

Scrub replay — load a journal in the dashboard and move through recorded telemetry frame by frame, with event markers on the timeline. Playback honours the journal’s recorded mono_ns timestamps, so a 200 Hz loop replays at 200 Hz. This is a view of what was recorded; it does not re-run your code.

Deterministic replay — TrainingLoop.replay() re-executes the loop, feeding back the recorded inputs instead of live IPC and verifying a state hash at each tick that has one. ReplayReport tells you how many ticks were verified and where the first divergence is. See Replay & Causality for the full API.

from omniloop import TrainingLoop

loop = TrainingLoop.replay("run.omni", cfg, tunable=["learning_rate"])
for _ in range(steps):
    with loop.tick():
        train_step(cfg)

print(loop.replay_report().first_divergence)

What determinism does and doesn’t guarantee

Deterministic replay feeds back recorded inputs — mutations, IPC commands, and any seeds the loop recorded — and verifies state hashes against the original run. It does not control nondeterminism outside the journal: unseeded RNG, wall-clock reads, thread scheduling, GPU kernel non-associativity, or filesystem state. A divergence means “this run did not reproduce”, not necessarily “OmniLoop replayed it wrong” — the first divergent tick is where to look.

fork_timeline is separate again: it relaunches the originating script with a frame’s state hot-patched, reusing the recorded argv. If the original script can’t be found or sits outside the trusted roots, the dashboard tells you why it fell back to a bundled mock rather than silently running a different program.

Making your loop forkable

Because a fork relaunches your script, your script has to tolerate the flags it is relaunched with — --init-state <json>, plus --replay-journal and --fork-tick when a journal is available. An argparse parser that has not declared them exits with code 2 before any of your code runs, which from the dashboard looks like a fork that quietly did nothing.

Two lines:

import argparse, omniloop

ap = argparse.ArgumentParser(parents=[omniloop.fork_arguments()])
args = ap.parse_args()

for name, value in omniloop.fork_state(args).items():   # {} on a normal run
    setattr(cfg, name, value)

fork_state() returns the forked frame with the __-prefixed provenance keys stripped, so a fork never claims to be the process it branched from. If you would rather not touch your parser, omniloop.fork_state_from_argv() reads the same flags out of sys.argv and ignores everything else.

The dashboard reports a fork that exits during startup, with the child’s own stderr, instead of logging “launched successfully” and leaving you to infer the failure from a journal that never appears.

Forks never overwrite the run they came from

A fork is handed the original command line, so it asks for the same journal= path — and TrainingLoop clears the segments at its journal path on startup. Left alone, forking a live run would truncate the recording you forked in order to keep.

The SDK redirects instead: a forked process journals to run.fork-<tick>.omni beside the original, and says so on stderr. Nothing is required of you, and the flag carrying the path can be named anything — the redirect happens where the journal is opened, not by pattern-matching the launch command. Read loop.journal_path if you need the path actually being written.