ontime/docs/sync-engine/backend-plan.md

Ontime Sync Engine — Backend Plan

Status: draft / requirements. Scope: backend only. The client/UI integration is deliberately out of scope and will be specified separately.

1. Goal

Run two independent Ontime backends (typically one local and one in the cloud) and keep them in sync:

• The user, from the client of one backend, initiates a sync between the two.
• After the initial sync, both backends hold the same project data.
• Ongoing data changes (rundown edits, custom fields, settings, …) propagate to both.
• Ongoing playback actions (start / stop / pause / roll / load / add-time, messages, aux timers) propagate to both, so both report the same runtime state.

The brief suggests leaning on a CRDT library (Automerge / automerge-repo) to do the heavy lifting. This document validates that idea against the codebase and proposes a concrete architecture.

2. What the code actually looks like (validated)

There are three distinct kinds of state in the server, and they have very different sync requirements. This distinction drives the whole design.

2.1 Persistent project data — the document

DatabaseModel (packages/types/src/definitions/DataModel.type.ts):

type DatabaseModel = {
  rundowns: ProjectRundowns;   // Record<RundownId, Rundown>
  project: ProjectData;
  settings: Settings;
  viewSettings: ViewSettings;
  urlPresets: URLPreset[];
  customFields: CustomFields;
  automation: AutomationSettings;
};

• Persisted by DataProvider (apps/server/src/classes/data-provider/DataProvider.ts) via lowdb to a JSON file, with a 3 s trailing-edge debounced write (persist()).
• A Rundown carries { id, title, order, flatOrder, entries, revision }. entries is a Record<EntryId, OntimeEntry> — i.e. a map keyed by stable id. order/flatOrder are arrays of ids.
• This is collaborative-document-shaped data. It is the natural fit for a CRDT.

2.2 Rundown cache + transaction layer

apps/server/src/api-data/rundown/rundown.dao.ts:

• The currently loaded rundown lives in an in-memory cachedRundown plus derived rundownMetadata (computed schedule: gaps, delays, group times, ordered lists).
• All edits go through createTransaction({ rundownId, mutableRundown }) → rundownMutation.* (add/edit/remove/reorder/applyDelay/swap/clone/group/ungroup/renumber) → commit(). commit():
  1. bumps cachedRundown.revision,
  2. re-processes derived metadata (processRundown),
  3. persists through DataProvider.setRundown.
• Non-loaded ("background") rundowns bypass the cache: read from disk, mutate, persist.
• rundown.service.ts wraps every mutation and, in a setImmediate, fires side effects:
  • updateRuntimeOnChange() → pushes derived counts into runtime state,
  • notifyChanges() → runtimeService.notifyOfChangedEvents() (timer) and sendRefetch(RefetchKey.Rundown, revision, rundownId) (tells clients to re-pull).

Key takeaway: mutations are funnelled through a single, well-defined chokepoint with a post-commit side-effect hook. That hook is exactly where remote (synced) changes must also be injected, so that a change arriving from the peer triggers the same cache rebuild + client refetch as a local edit.

The other persistent slices (project, settings, viewSettings, urlPresets, customFields, automation) are written directly through DataProvider setters and do not go through the transaction/side-effect layer — they emit their own Refetch from their routers. Sync must cover these too.

2.3 Runtime / playback state — not document data

apps/server/src/stores/runtimeState.ts + EventTimer + runtime.service.ts:

• Live timer state is derived every tick from the local wall clock (timeCore.now() = Date.now()), recomputed at 30 fps and broadcast to clients at ~1 fps via eventStore over the websocket (MessageTag.RuntimeData).

• The entire playback state collapses to a tiny serialisable record — the existing RestorePoint (services/restore-service/restore.type.ts):

  type RestorePoint = {
    playback: Playback;
    selectedEventId: MaybeString;
    startedAt: MaybeNumber;     // TimeOfDay (ms since local midnight)
    addedTime: number;
    pausedAt: MaybeNumber;      // TimeOfDay
    firstStart: MaybeNumber;    // TimeOfDay
    startEpoch: Maybe<Instant>; // absolute epoch ms  ← timezone independent
    currentDay: MaybeNumber;
  };
  

• runtimeState.resume(restorePoint, event, rundown, metadata) already reconstructs a live playing timer from this record — this is the mechanism a follower backend will reuse to adopt remote playback state.

• Commands (start/startById/stop/pause/roll/load*/addTime/setOffsetMode) all live on the runtimeService singleton, decorated with @broadcastResult. External callers reach them through dispatchFromAdapter() (integration.controller.ts) from WS/OSC/HTTP.

• Messages (message.service.ts) and aux timers (AuxTimerService) are additional ephemeral runtime state held in eventStore, not in the DB.

Critical timing observation: timeCore.toTimeOfDay() uses the machine's local timezone offset (getTimezoneOffset). startEpoch is absolute and TZ-independent, but startedAt, pausedAt, firstStart, clock are all TimeOfDay in the originating machine's TZ. A local box and a cloud box in different timezones will not interpret a replicated TimeOfDay the same way. Playback sync must therefore anchor on absolute epoch + the project's configured timezone, and the follower must recompute TimeOfDay-derived fields locally rather than copying them verbatim.

3. Does a CRDT fit? — Verdict

Yes, but only for §2.1 (the project document). Automerge is a strong fit there:

• Edits are keyed by stable ids (entries[id], customFields[key], rundowns by id), so concurrent edits to different entries merge cleanly (Automerge maps merge per-key).
• It removes the need to hand-roll conflict resolution, op ordering, and incremental catch-up after disconnection.

No for §2.3 (playback/runtime). A CRDT is the wrong tool for real-time control:

• Timer ticks must not be streamed over the network — each backend already derives them locally from the wall clock. We only need to replicate intent transitions.
• Playback is a control-plane concern with a "last command wins" nature, not a mergeable document.

So the recommendation is a hybrid:

Domain	Mechanism	Library
Project document (DatabaseModel)	CRDT document, incremental sync	@automerge/automerge-repo + WS network adapter
Playback / messages / aux timers	Replicated intent (LWW register w/ logical clock), recomputed locally	small in-house module over the same socket
Live timer ticks	Not synced — derived locally on each node	existing EventTimer

4. Topology & connection model

• Initiation is directional, ongoing sync is bidirectional. "Sync now" from backend A must choose a baseline owner (whose project seeds the shared document). Merging two unrelated projects with a CRDT yields a union of both rundowns — almost never what the user wants. So:
  1. On "sync", A and B establish a connection.
  2. The chosen baseline (say A) exports its current project as the shared Automerge document; B adopts it (forks from A's document so they share lineage/history).
  3. From then on, both edit the same document and automerge-repo reconciles incrementally and bidirectionally.
• Who dials whom: the cloud instance (IS_CLOUD) has a reachable public endpoint; the local instance is usually behind NAT. The local node dials out to the cloud node, and the cloud node acts as the automerge-repo sync server / relay. This is exactly the automerge-repo WebSocket server/client split.
• Sync targets the currently loaded project only (one Automerge DocumentId ↔ one Ontime project). Switching projects detaches/attaches the sync session.
• Auth: the sync socket must authenticate. Reuse the existing auth (makeAuthenticateMiddleware / login flow + shared token). The "sync" action carries the peer URL + credentials.

5. Integration points (where code hooks in)

1. DataProvider becomes CRDT-backed (the document).

   • The shared Automerge doc holds the persistent DatabaseModel.
   • DataProvider read paths return the doc's current value; write paths (setRundown, setCustomFields, setSettings, …) are re-expressed as Automerge change() calls.
   • Granularity: the existing mutations already operate at per-entry / per-key level (rundown.entries[id] = …, order.splice(...), customFields[key] = …). Re-expressing them as Automerge changes at that same granularity gives good merge behaviour without a full rewrite of the mutation algorithms. Avoid replacing whole entries/order objects wholesale — that defeats per-key merge. The current commit() reassigns cachedRundown.entries = entries; the CRDT adapter needs to apply the delta instead.
   • lowdb persistence stays as a local durability layer (or is replaced by Automerge's own storage adapter). Either way the 3 s debounce semantics should be preserved.

2. Remote-change observer → existing side-effect path.

   • Subscribe to Automerge doc changes. When a change arrives from the peer (not from a local mutation), run the same post-commit side effects that a local edit would:
     • rebuild the loaded-rundown cache (rundownCache.init / runtimeState.updateAll),
     • runtimeService.notifyOfChangedEvents(metadata),
     • sendRefetch(RefetchKey.Rundown | …, revision, rundownId) to local clients.
   • This is the single most important hook: it makes remote edits indistinguishable from local edits to everything downstream (clients, timer, integrations).

3. revision semantics. Today revision is a per-rundown monotonic counter used only to tell clients "you're stale, refetch". With two writers it can collide. Options: derive the client-facing revision from the Automerge document heads/hash, or keep the counter as advisory and rely on the refetch always pulling current truth. Recommend deriving a stable version token from Automerge heads.

4. Playback intent channel.

   • Define a replicated PlaybackIntent ≈ RestorePoint + offsetMode, plus messages and auxTimers[1..3] intent ({playback, startedAtEpoch, duration, direction}).
   • Model as a LWW register stamped with a logical (Lamport) clock + originating peer id. Every runtimeService command updates the local intent and publishes it; the peer applies it if its stamp is newer.
   • The follower applies intent via a resume-style path (runtimeState.resume-like) that recomputes TimeOfDay fields from startEpoch + clock offset + project timezone — it does not copy startedAt/pausedAt verbatim (see §2.3 timing note).
   • Live ticks remain local; both nodes converge because they share intent + a common clock.

5. Clock synchronisation.

   • Both nodes must agree on epoch time within tolerance (target sub-100 ms for broadcast use).
   • Recommend an application-level offset estimate over the sync socket (periodic timestamped ping ⇒ Cristian's algorithm / NTP-lite), applied by the follower when interpreting startEpoch. Do not assume both machines are NTP-disciplined, but benefit from it when they are.

6. Conflict & authority model

• Document edits: resolved by Automerge (per-key map merge, RGA for arrays). Define a policy for the rare same-key concurrent edit (Automerge picks a deterministic winner; we may surface a "changed remotely" hint to the editor). Concurrent reorders of the same list are the main thing to test (array CRDT semantics).
• Playback: a human operator drives it; genuinely simultaneous conflicting commands are rare. LWW on the intent register (logical clock + peer id tiebreak) is sufficient for a 2-node system and far simpler than a leader-election protocol. Revisit if N>2 is ever needed.

7. Phased delivery

1. Phase 0 — Spec & spike. Lock requirements (this doc). Spike automerge-repo WS client/server between two local server instances; prove a doc round-trips.
2. Phase 1 — Document sync (data only). CRDT-back the DatabaseModel; remote-change observer wired into the existing refetch/cache side-effect path. Directional initial seed. No playback sync yet. Deliverable: edits on either node appear on both.
3. Phase 2 — Clock sync + playback intent. Offset estimation; replicate PlaybackIntent; follower derives ticks locally. Deliverable: start/stop/pause/roll/load/add-time mirror.
4. Phase 3 — Messages & aux timers. Extend intent channel.
5. Phase 4 — Resilience. Reconnection/catch-up, project-switch handling, auth hardening, conflict UX hints, observability (drift metrics, sync status).

8. Open questions / decisions needed

1. Library: confirm @automerge/automerge-repo (WASM core) vs alternatives (Yjs). Automerge matches the keyed-map data model and brittle-free merges; Yjs is leaner/faster but more text-CRDT oriented. Recommendation: Automerge.
2. Baseline-owner UX: when the two projects differ at initiation, is it always "push mine / overwrite theirs", or do we offer "pull theirs"? (Merging unrelated projects is explicitly discouraged.)
3. Persistence: keep lowdb as the local store and treat Automerge as the in-memory sync truth, or move durability to an Automerge storage adapter? Affects crash recovery and the existing flushPendingWrites/restore flow.
4. Timezone authority: anchor playback on the project's configured timezone (not each machine's local TZ). Confirm where that timezone lives / whether it must be added.
5. report data (run history): sync as part of the document, or keep per-instance?
6. Scope of N: is 2 nodes the hard ceiling, or should the intent/authority model leave room for more peers?
7. Multiple loaded rundowns / background rundowns: confirm the whole project document syncs (all rundowns), while only the loaded one drives the runtime on each node.

9. Risks

• TimeOfDay vs absolute epoch across timezones (the single biggest playback-sync trap; §2.3).
• Array/order merge semantics for concurrent reorders — needs explicit test coverage.
• structuredClone-and-replace mutation style must be converted to deltas or it will clobber concurrent edits and negate the CRDT.
• Document growth / compaction — Automerge history grows; plan periodic compaction/snapshots.
• Bandwidth on the local↔cloud link — fine for doc deltas + intent; would be a problem if timer ticks were ever streamed (they must not be).