.omni Container Format

The OmniLoop journal format has been upgraded to a more robust v2 format, featuring mid-file corruption resistance, sparse indexing, and embedded metadata.

When to Use

You rarely need to parse this format yourself. Use the Python or Rust APIs unless you are building a custom language binding or third-party analyzer.

Container Format

All integers in the format are little-endian.

Binary Layout

File header (32 bytes)
  [0..8)   magic            b"OMNILOG1"
  [8..12)  version          u32 = 2
  [12..16) flags            u32 (bit 0 = SEGMENTED, bit 1 = CHAINED;
                                 bits 2..31 reserved)
  [16..24) created_wall_ns  u64 unix-epoch nanos
  [24..32) segment_index    u64 (ordinal within a rotated set when SEGMENTED
                                 is set; otherwise reserved, always 0)

Record (repeated; 36-byte header + payload)
  [0..4)   payload_len  u32
  [4..5)   kind         u8   0=telemetry 1=mutation 2=control 3=lifecycle
                             4=hash 5=input 6=chain 7=signature 8=guardrail
  [5..8)   padding
  [8..16)  tick         u64  monotonic frame counter, stamped by the Rust core
  [16..24) mono_ns      u64  nanos since journal creation (monotonic clock)
  [24..32) wall_ns      u64  unix-epoch nanos
  [32..36) crc32        u32  IEEE CRC-32 of the payload
  [36..)   payload

Index (written at close; one entry per 64 records)
  magic b"OMNIIDX1" | count u64 | count x { tick u64, offset u64 }

Trailer (last 24 bytes)
  magic b"OMNIEND1" | index_offset u64 | record_count u64

Header Flags

Bit	Name	Meaning
0	SEGMENTED	This file is one segment of a rotated journal set. segment_index at [24..32) holds its ordinal.
1	CHAINED	Records carry a SHA-256 hash chain, checkpointed by kind=6 records. See Hash Chain.
2–31	—	Reserved; a reader must ignore bits it does not recognise.

CHAINED is set on every journal this build writes. It exists so a verifier can distinguish “this file was written without a chain” from “this file’s checkpoints were removed” — otherwise those are the same bytes, and a verifier that cannot tell them apart is not a verifier.

A journal written without rotation has segment_index == 0 and SEGMENTED clear. It is no longer byte-identical to files written before the chain existed: flags now carries CHAINED.

Segmented Journals

A run recorded with a rotation policy is written as a set of files rather than one unbounded file. Segment 0 is the path the caller named; continuation segments insert the ordinal before the extension:

run.omni      segment 0
run.1.omni    segment 1
run.2.omni    segment 2

Each segment is a complete, independently-finalized v2 container — its own index and trailer — so any segment opens on its own and seek_to_tick() stays $O(\log n)$ within it. Ticks are assigned by the writer and keep counting across a rollover, so concatenating the segments in order reproduces exactly the record stream an unrotated run would have produced.

A third-party parser that only understands single files still reads segment 0 correctly; it simply sees a shorter run. To read the whole set, walk the naming convention from the base path and stop at the first missing ordinal.

Record Kinds

The kind field in the record header maps to the following enum:

• 0 = telemetry — state snapshot, one per tick
• 1 = mutation — parameter change, with old/new/source/correlation id
• 2 = control — system command (halt/resume/step), watch trips, clamps, checkpoints
• 3 = lifecycle — session start/stop, carrying schema, argv, and script path
• 4 = hash — state-hash checkpoint for divergence detection
• 5 = input — external inputs for deterministic replay (IPC commands, RNG seeds)
• 6 = chain — SHA-256 hash-chain checkpoint; payload is 32 raw bytes
• 7 = signature — Ed25519 over a chain checkpoint; payload is pubkey(32) || sig(64)
• 8 = guardrail — a limit downgrade: {name, requested, applied, min, max, cause, principal}

Kinds 6 and 7 describe the file, not the run. Consumers that analyse a run — replay, inspect, why, diff, MCAP export — should read through read_next_data_record(), which skips them. Verification uses read_next_record(), which returns everything.

Caution

The kind space is append-only, but this implementation rejects unknown kinds rather than skipping them, on the grounds that silently ignoring a record you cannot interpret is the wrong default for an evidence format. A third-party parser aiming for forward compatibility may choose to skip instead; a parser aiming to verify must not.

Hash Chain

Every record advances a running SHA-256 digest:

genesis  = SHA256("omniloop-chain-v1-genesis" || file_header[0..32])
chain[n] = SHA256("omniloop-chain-v1-step" || chain[n-1] || record_header[0..36] || payload)

A kind=6 checkpoint is written every 64 records and again when a segment is sealed, carrying the chain value immediately before itself; it then advances the chain in turn, so checkpoints are linked and cannot be individually rewritten. The closing checkpoint matters: a chain that stops 63 records short leaves the tail of the run editable.

The digest covers the record header as well as the payload, so tick and timestamps are immutable — hashing the payload alone would let a record be re-dated or re-ordered undetected.

What verification proves: no record has been altered, removed, re-timed or re-ordered since it was written.

What it does not prove: authorship. Nothing in the chain is secret, so an attacker who rewrites the entire file can recompute a consistent one. That is what kind=7 closes — an Ed25519 signature over a chain checkpoint cannot be reproduced without the private key, so a wholesale rewrite fails signature verification even though its chain is self-consistent.

Producing signatures requires the signing cargo feature and a key. Verifying them requires neither — an auditor must never have to trust a build flag to check a record.

Design Properties

• Untrusted Payloads: Ticks and timestamps are stamped in Rust, not trusted from the payload.
• Resilience: The IEEE CRC-32 per record means mid-file corruption raises an IOError, while truncation is handled gracefully.
• Fast Seeking: The sparse index and trailer make seek_to_tick() $O(\log n)$ on finalized files. If a run crashes, it falls back to a sequential scan.

API Usage

Python

The Python API provides tools for parsing and inspecting journal files:

player = JournalPlayer("run.omni")
player.version()          # 1
player.record_count()     # total records, from the trailer (None if crashed)
player.seek_to_tick(4000) # next read returns first record with tick >= 4000
player.is_segmented()     # True if this file is part of a rotated set
player.segment_index()    # its ordinal, or None for a standalone journal

player.read_next_record()      # every record, chain/signature checkpoints included
player.read_next_data_record() # skips them — what replay, inspect and why want

To read a rotated run end to end, use SegmentedPlayer. It chains the segments transparently and behaves identically on an unrotated journal, so you never have to know which kind you were handed:

from omniloop import SegmentedPlayer, journal_segments

journal_segments("run.omni")   # ['run.omni', 'run.1.omni', 'run.2.omni']

player = SegmentedPlayer("run.omni")
player.segment_count()         # 3
player.record_count()          # total across every finalized segment
while (rec := player.read_next_record()) is not None:
    ...                        # ticks stay monotonic across rollovers

cause_of(), effects_of(), find_divergence() and InputReplayer are all segment-aware already — they take the base path and read the whole set.

Chain verification is a plain function, not a method — it opens the file itself rather than working from an already-open player:

from omniloop import verify_chain

result = verify_chain("run.omni")
result["ok"]               # False if anything failed
result["declared_chained"] # whether the header claims a chain at all
result["checkpoints"]      # how many were checked
result["head"]             # final chain value, hex — what a signature covers
result["break_at"]         # None, or {"type": "DigestMismatch", ...}

break_at["type"] is one of DigestMismatch, CrcMismatch, MalformedCheckpoint, NotChained, ChainedButNoCheckpoints — see Hash Chain for what each means. The same check is one command away without writing any code:

omniloop verify run.omni
omniloop verify run.omni --json

Signature verification has no Python or CLI surface yet — see the Rust surface below.

Rust Surface

In the Rust core (omniloop-core/src/journal/), the main primitives are:

• StateJournal (create_rotating + RotationPolicy for segmented runs)
• JournalPlayer — one file
• SegmentedPlayer / segment_paths — a rotated set as one stream
• FrameKind
• FrameRecord
• verify_chain → ChainVerification / ChainBreak — always available
• verify_signed / JournalSigner — behind the signing feature

let v = omniloop_core::journal::verify_chain("run.omni")?;
if !v.ok {
    // ChainBreak::DigestMismatch | CrcMismatch | ChainedButNoCheckpoints | ...
    eprintln!("journal failed verification: {:?}", v.break_at);
}

Verification is sequential by construction: the sparse index is deliberately not used, because trusting an index to decide which bytes get hashed would let an attacker choose what gets verified.

Compatibility

Note

Legacy files (pre-v2 bare [u32 len][payload]) are auto-detected and remain readable forever. Their .version() will report 1.

Caution

New .omni files are not readable by pre-v2 builds — the format change is one-way.