Skip to content

Replay Divergence Detection

Snapshot scrubbing can’t inherently tell you whether a replayed run actually followed the recorded one. Hash checkpoints provide a mechanism to verify determinism — similar to the approach used by rr.

During recording, a deterministic FNV-1a 64-bit hash of the canonical state is journaled every few ticks.

The canonical state includes sorted name=value pairs, but excludes volatile keys like timestamps.

Comparing two journals pinpoints the first tick where their state disagrees. Checkpoints present in only one journal are ignored, so runs with different cadences compare cleanly.

from omniloop import find_divergence
div = find_divergence("recorded.omni", "replayed.omni")
# If div is None:
# runs agree at every shared checkpoint
# If divergent:
# {"tick": 4350, "hash_a": ..., "hash_b": ...}
# -> bisect from tick 4350

You can enable hash checkpoints and control their cadence in the TrainingLoop:

from omniloop import TrainingLoop
# Hash the canonical state every 50 ticks
# Set hash_every=0 to disable (the default)
loop = TrainingLoop(..., hash_every=50)

For raw-primitive users, you can manually write hash checkpoints:

import json
from omniloop import fnv1a64
# Exclude volatile keys like timestamps before hashing
canonical = json.dumps(state, sort_keys=True, separators=(",", ":"))
journal.write_hash(tick, fnv1a64(canonical.encode()))

Alternatively, if you’re using a tracker, tracker.state_hash(exclude=["timestamp"]) hashes the registry’s canonical state directly (handling pointer-reflection flows).