Tethered Deployment

OmniLoop’s security notice says: bind to loopback, never expose the port. That is correct, and it also describes a topology almost nobody has. The loop runs on some other machine; the dashboard runs on your laptop. Two machines, whether the other one is a robot, a GPU box or a batch node.

This page is the supported way to bridge them. The short version: forward the port over SSH, so the server keeps believing it is talking to localhost — because it is.

One pattern, three topologies

The mechanism is identical in all three. What differs is who else is on that network and what happens if the tunnel dies mid-halt, so the hardening below is graded rather than uniform.

Topology A: a robot

# On the robot: bring the loop and relay up, bound to loopback as always.
omniloop up --no-dashboard

# On your laptop: forward the relay's port over SSH, then open the dashboard.
ssh -N -L 8000:127.0.0.1:8000 operator@robot.local

Now http://127.0.0.1:8000 on your laptop is the robot’s relay. Open it and the dashboard works exactly as it does locally.

This is the case the hardware checklist below is written for, and the one where halt_policy="handoff" is not optional.

Topology B: a training or compute node

Most training does not happen on the machine you are sitting at, so this is the common case for anyone using OmniLoop on a long training run, a fine-tune or a steered solve rather than on hardware.

# On the training node, inside your job:
python train.py            # your loop, with TrainingLoop/Loop instrumented
omniloop up --no-dashboard # in the same allocation, same session id

# On your laptop:
ssh -N -L 8000:127.0.0.1:8000 you@gpu-node-04

Same command, different anxieties. What changes:

• A halt is genuinely just a pause. Nothing is holding a load, so the default halt_policy="block" is right and handoff would be ceremony. Most of the hardware checklist below does not apply to you.
• Freeze-on-exception is worth more here than anywhere else. The whole point is that a NaN at hour fourteen freezes a warm process with its weights, optimiser state and replay buffer intact instead of returning a traceback and an empty GPU. Pair it with checkpoint= so the freeze survives the process.
• The tunnel dying is not an emergency. Your laptop will sleep and the SSH connection will drop; the loop keeps running and you reconnect. Leave halt_timeout unset unless you have a reason — a deadman that resumes a run you deliberately froze is worse than no deadman.
• Journals rotate. A multi-day run writes a large .omni. Set journal_max_bytes or journal_max_seconds and let it segment.
• fork_timeline still works because the server is on loopback, and on a training node it is often what you actually want: fork the run at the tick before the collapse and try the other hyperparameter.

Topology C: a cluster node behind a login host

Batch nodes are usually not routable from your laptop. Jump through the login host in one command rather than nesting two tunnels:

# -J does the login-host hop; the -L target is resolved on the compute node.
ssh -N -J you@login.cluster.edu -L 8000:127.0.0.1:8000 you@node1423

Two things bite here and neither is OmniLoop’s doing:

• The node name is only known after the job starts. Have the job print its hostname (scontrol show hostname, $SLURMD_NODENAME, or just hostname) into the job log, and read it from there before opening the tunnel.

• Port 8000 is shared with every other user on that node. Move yours with OMNILOOP_PORT, or the relay will fail to bind — or worse, you will tunnel into somebody else’s:

  OMNILOOP_PORT=8137 omniloop up --no-dashboard
  ssh -N -J you@login.cluster.edu -L 8137:127.0.0.1:8137 you@node1423

  Note that omniloop up still prints http://127.0.0.1:8000 when you do this; the server binds where you told it to and the banner is stale. Trust the variable, not the banner.

Under any of the three, nothing about the server changes. It is still bound to 127.0.0.1, still refusing every non-loopback connection, still inside its origin allow-list. SSH carries the encryption, the authentication and the authorization, which are three things it does far better than this project would.

Why this and not OMNILOOP_HOST=0.0.0.0

Binding the relay to a routable interface hands full admin control — live memory mutation, halt, and subprocess execution via fork_timeline — to anything that can reach the port. On a robot’s network that is usually more machines than you think. The server warns loudly when you do it, and fork_timeline disables itself because spawning subprocesses for a remote caller is not a thing this project is willing to do quietly.

An SSH tunnel gives you the same reachability with none of that.

Defence in depth

The tunnel is the boundary. These narrow what is behind it.

Require a token

# On the robot, before `omniloop up`:
export OMNILOOP_AUTH_TOKEN="$(openssl rand -hex 32)"

Clients then authenticate with a first-message token, a ?token= query parameter, or a bearer header. Worth setting even behind a tunnel: it stops another process on the robot — or another user on a shared operator box — from attaching to a loop that can move an arm.

Tokens are redacted from server logs, so a ?token= in an access line does not end up in your journal archive.

Name the token holders

One shared secret answers “may this connection do anything”, and cannot answer “who commanded kp=400 at 14:32” — which is the question a bench with an operator, a developer and an agent on it actually asks.

export OMNILOOP_AUTH_TOKENS="alice:$(openssl rand -hex 32),\
agent:$(openssl rand -hex 32):observer"

Each entry is name:token or name:token:role, and each token authenticates as a distinct principal whose id is stamped onto every mutation crossing the command ring and into every journal event.

Role	May
observer	read only
operator	read, mutate, and control (halt / step / watchpoints / bounds)
admin (default)	all of the above, plus fork_timeline

fork_timeline is its own capability rather than part of control because a deployment that wants live tuning almost never wants arbitrary subprocess execution. A refused call comes back with the stable capability error code, so a client can grey the control out rather than offering a button that fails.

Coexists with OMNILOOP_AUTH_TOKEN; a deployment may set either or both. A malformed entry — no separator, an unknown role, a token already assigned to someone else — is dropped with a warning rather than failing startup, because a typo in an environment variable must not be a way to take the relay down.

Keep the origin allow-list tight

Browser handshakes are checked against an allow-list that covers the local dashboard ports by default. Extend it only if you serve the dashboard from somewhere else:

export OMNILOOP_ALLOWED_ORIGINS="http://127.0.0.1:5173"

Never set it to *. The server warns if you do; combined with credentialed requests it disables origin validation entirely.

Isolate the session

omniloop up --session arm_left

One publisher per session. On a robot with several instrumented processes, give each its own session id rather than letting them collide — omniloop doctor reports a collision as a fault precisely because two loops sharing a channel is never intentional.

The same applies for a different reason on a shared compute node, where the colliding process is not yours: derive the id from something unique to your allocation (--session "$SLURM_JOB_ID") rather than leaving the default.

Watch several nodes at once

Isolation is not separation. Give each node its own session, then have one relay watch all of them:

omniloop up --session arm_left --watch arm_right,estimator,locomotion

Each session keeps its own segment pair and its own single writer — that is what makes the telemetry seqlock cheap and tear-free, and it does not change. The relay opens one channel per session, tags every frame with its session, and the dashboard switches between them on one time axis, so the arm’s tick 8412 sits next to the estimator’s tick 8412.

A node that has not started yet is logged and skipped rather than fatal; on a robot some nodes start late and some never start at all.

Switching the view, not merging the control

A connection drives exactly one session at a time, chosen with select_session. Mutate, halt and step are per-loop and stay that way: a dashboard that quietly aimed a Halt at whichever node published most recently would be the worst possible affordance on a machine that can move. list_sessions reports selected so a client can show which controller a button would actually reach.

What turns itself off when you leave loopback

fork_timeline relaunches the originating script as a subprocess. That is a reasonable thing to offer a developer on their own machine and an unreasonable thing to offer the network, so it is enabled only when the server is bound to loopback. Force it off entirely with:

export OMNILOOP_ENABLE_FORK=0

Under the tunnel pattern the server is on loopback, so forking stays available. If you would rather it were not — and on anything attached to hardware, you probably would — set the variable.

Driving it from an agent

The MCP server reaches the relay over the same ws://127.0.0.1:8000/ws, so it works through the tunnel unchanged. Point it at the forwarded port and pass the token:

{
  "mcpServers": {
    "omniloop-robot": {
      "command": "python",
      "args": ["-m", "omniloop.mcp"],
      "env": {
        "OMNILOOP_MCP_SERVER_URL": "ws://127.0.0.1:8000/ws",
        "OMNILOOP_AUTH_TOKEN": "…",
        "OMNILOOP_SESSION_ID": "arm_left"
      }
    }
  }
}

Note the absence of --allow-control. Read-only is the default, and a tunnel to a machine that can move is the last place to change that without deciding to.

A checklist for hardware

Topology A only. On a training or compute node a halt is just a pause and most of this is ceremony — items 5 and 6 still apply, the rest do not. Before the tether goes anywhere near something that moves:

1. halt_policy="handoff" with a real fallback controller. Under the default block policy a dashboard Halt stops your loop from commanding, which on a legged or held-load machine is a fall. See Halt & Step.
2. halt_timeout armed. Your laptop will sleep, the wifi will drop, and the tunnel will die mid-halt. The deadman is what decides what happens next.
3. limit() on anything that can break something. A slider range is a UI hint; a limit is enforced inside the target process against every edit.
4. Tripwires armed and verified. Check omniloop_list_watchpoints — or the __watchpoints__ readback — rather than trusting that you armed them.
5. A release build. A debug core is ~100x slower and will miss deadlines for reasons unrelated to your controller.
6. A watchdog outside the process. OmniLoop’s deadman only advances while your loop is calling the barrier; it cannot notice a thread that stopped running. That failure needs a watchdog that outlives the process.

What this does not give you

Being explicit, because a security page that implies more than it delivers is worse than none:

• No identity provider. OMNILOOP_AUTH_TOKENS maps static tokens to named principals, which is enough to attribute an action in a journal and is not an identity system: there is no expiry, no rotation, no revocation list, and anyone holding the token is that principal.
• No TLS of its own. SSH provides the transport encryption. There is no supported way to expose the relay directly over TLS.
• Coarse authorization. Three roles, not a policy engine. A principal with no roles is treated as admin, so that upgrading does not silently demote a working deployment.

If you need real identities or per-user authorization, manage authentication upstream via reverse proxies or SSH tunnels.