diff --git a/docs/sync-engine/backend-plan.md b/docs/sync-engine/backend-plan.md new file mode 100644 index 000000000..5f9d58d7e --- /dev/null +++ b/docs/sync-engine/backend-plan.md @@ -0,0 +1,260 @@ +# Ontime Sync Engine — Backend Plan + +> Status: **draft / requirements**. Scope: backend only. The client/UI integration is +> deliberately out of scope and will be specified separately. + +## 1. Goal + +Run **two independent Ontime backends** (typically one **local** and one in the **cloud**) and +keep them in sync: + +- The user, from the client of *one* backend, initiates a sync between the two. +- After the initial sync, both backends hold the **same project data**. +- Ongoing **data changes** (rundown edits, custom fields, settings, …) propagate to both. +- Ongoing **playback actions** (start / stop / pause / roll / load / add-time, messages, + aux timers) propagate to both, so both report the same runtime state. + +The brief suggests leaning on a CRDT library (Automerge / `automerge-repo`) to do the heavy +lifting. This document validates that idea against the codebase and proposes a concrete +architecture. + +## 2. What the code actually looks like (validated) + +There are **three distinct kinds of state** in the server, and they have very different sync +requirements. This distinction drives the whole design. + +### 2.1 Persistent project data — *the document* + +`DatabaseModel` (`packages/types/src/definitions/DataModel.type.ts`): + +```ts +type DatabaseModel = { + rundowns: ProjectRundowns; // Record + project: ProjectData; + settings: Settings; + viewSettings: ViewSettings; + urlPresets: URLPreset[]; + customFields: CustomFields; + automation: AutomationSettings; +}; +``` + +- Persisted by `DataProvider` (`apps/server/src/classes/data-provider/DataProvider.ts`) via + **lowdb** to a JSON file, with a **3 s trailing-edge debounced write** (`persist()`). +- A `Rundown` carries `{ id, title, order, flatOrder, entries, revision }`. `entries` is a + `Record` — i.e. a map keyed by stable id. `order`/`flatOrder` are + arrays of ids. +- This is **collaborative-document-shaped data**. It is the natural fit for a CRDT. + +### 2.2 Rundown cache + transaction layer + +`apps/server/src/api-data/rundown/rundown.dao.ts`: + +- The **currently loaded** rundown lives in an in-memory `cachedRundown` plus derived + `rundownMetadata` (computed schedule: gaps, delays, group times, ordered lists). +- All edits go through `createTransaction({ rundownId, mutableRundown })` → + `rundownMutation.*` (add/edit/remove/reorder/applyDelay/swap/clone/group/ungroup/renumber) + → `commit()`. `commit()`: + 1. bumps `cachedRundown.revision`, + 2. re-processes derived metadata (`processRundown`), + 3. persists through `DataProvider.setRundown`. +- **Non-loaded ("background") rundowns** bypass the cache: read from disk, mutate, persist. +- `rundown.service.ts` wraps every mutation and, in a `setImmediate`, fires **side effects**: + - `updateRuntimeOnChange()` → pushes derived counts into runtime state, + - `notifyChanges()` → `runtimeService.notifyOfChangedEvents()` (timer) and + `sendRefetch(RefetchKey.Rundown, revision, rundownId)` (tells clients to re-pull). + +> Key takeaway: **mutations are funnelled through a single, well-defined chokepoint** with a +> post-commit side-effect hook. That hook is exactly where remote (synced) changes must also +> be injected, so that a change arriving from the peer triggers the same cache rebuild + +> client refetch as a local edit. + +The other persistent slices (`project`, `settings`, `viewSettings`, `urlPresets`, +`customFields`, `automation`) are written **directly** through `DataProvider` setters and do +**not** go through the transaction/side-effect layer — they emit their own `Refetch` from +their routers. Sync must cover these too. + +### 2.3 Runtime / playback state — *not document data* + +`apps/server/src/stores/runtimeState.ts` + `EventTimer` + `runtime.service.ts`: + +- Live timer state is **derived every tick from the local wall clock** (`timeCore.now()` = + `Date.now()`), recomputed at 30 fps and broadcast to clients at ~1 fps via `eventStore` + over the websocket (`MessageTag.RuntimeData`). +- The **entire playback state collapses to a tiny serialisable record** — the existing + `RestorePoint` (`services/restore-service/restore.type.ts`): + + ```ts + type RestorePoint = { + playback: Playback; + selectedEventId: MaybeString; + startedAt: MaybeNumber; // TimeOfDay (ms since local midnight) + addedTime: number; + pausedAt: MaybeNumber; // TimeOfDay + firstStart: MaybeNumber; // TimeOfDay + startEpoch: Maybe; // absolute epoch ms ← timezone independent + currentDay: MaybeNumber; + }; + ``` + +- `runtimeState.resume(restorePoint, event, rundown, metadata)` already **reconstructs a live + playing timer from this record** — this is the mechanism a follower backend will reuse to + adopt remote playback state. +- **Commands** (`start/startById/stop/pause/roll/load*/addTime/setOffsetMode`) all live on the + `runtimeService` singleton, decorated with `@broadcastResult`. External callers reach them + through `dispatchFromAdapter()` (`integration.controller.ts`) from WS/OSC/HTTP. +- **Messages** (`message.service.ts`) and **aux timers** (`AuxTimerService`) are additional + ephemeral runtime state held in `eventStore`, not in the DB. + +> Critical timing observation: `timeCore.toTimeOfDay()` uses the **machine's local timezone +> offset** (`getTimezoneOffset`). `startEpoch` is absolute and TZ-independent, but `startedAt`, +> `pausedAt`, `firstStart`, `clock` are all *TimeOfDay in the originating machine's TZ*. A +> local box and a cloud box in different timezones will **not** interpret a replicated +> TimeOfDay the same way. Playback sync must therefore anchor on **absolute epoch + the +> project's configured timezone**, and the follower must **recompute** TimeOfDay-derived +> fields locally rather than copying them verbatim. + +## 3. Does a CRDT fit? — Verdict + +**Yes, but only for §2.1 (the project document).** Automerge is a strong fit there: + +- Edits are keyed by stable ids (`entries[id]`, `customFields[key]`, rundowns by id), so + concurrent edits to *different* entries merge cleanly (Automerge maps merge per-key). +- It removes the need to hand-roll conflict resolution, op ordering, and incremental + catch-up after disconnection. + +**No for §2.3 (playback/runtime).** A CRDT is the wrong tool for real-time control: + +- Timer ticks must **not** be streamed over the network — each backend already derives them + locally from the wall clock. We only need to replicate **intent transitions**. +- Playback is a control-plane concern with a "last command wins" nature, not a mergeable + document. + +So the recommendation is a **hybrid**: + +| Domain | Mechanism | Library | +| --- | --- | --- | +| Project document (`DatabaseModel`) | CRDT document, incremental sync | `@automerge/automerge-repo` + WS network adapter | +| Playback / messages / aux timers | Replicated **intent** (LWW register w/ logical clock), recomputed locally | small in-house module over the same socket | +| Live timer ticks | **Not synced** — derived locally on each node | existing `EventTimer` | + +## 4. Topology & connection model + +- **Initiation is directional, ongoing sync is bidirectional.** "Sync now" from backend A + must choose a **baseline owner** (whose project seeds the shared document). Merging two + *unrelated* projects with a CRDT yields a union of both rundowns — almost never what the + user wants. So: + 1. On "sync", A and B establish a connection. + 2. The chosen baseline (say A) exports its current project as the shared Automerge document; + B **adopts** it (forks from A's document so they share lineage/history). + 3. From then on, both edit the *same* document and `automerge-repo` reconciles incrementally + and bidirectionally. +- **Who dials whom:** the cloud instance (`IS_CLOUD`) has a reachable public endpoint; the + local instance is usually behind NAT. The **local node dials out to the cloud node**, and + the cloud node acts as the `automerge-repo` sync server / relay. This is exactly the + `automerge-repo` WebSocket server/client split. +- **Sync targets the currently loaded project only** (one Automerge `DocumentId` ↔ one Ontime + project). Switching projects detaches/attaches the sync session. +- **Auth:** the sync socket must authenticate. Reuse the existing auth (`makeAuthenticateMiddleware` + / login flow + shared token). The "sync" action carries the peer URL + credentials. + +## 5. Integration points (where code hooks in) + +1. **DataProvider becomes CRDT-backed (the document).** + - The shared Automerge doc holds the persistent `DatabaseModel`. + - `DataProvider` read paths return the doc's current value; write paths (`setRundown`, + `setCustomFields`, `setSettings`, …) are re-expressed as Automerge `change()` calls. + - Granularity: the existing mutations already operate at **per-entry / per-key** level + (`rundown.entries[id] = …`, `order.splice(...)`, `customFields[key] = …`). Re-expressing + them as Automerge changes at that same granularity gives good merge behaviour without a + full rewrite of the mutation algorithms. **Avoid replacing whole `entries`/`order` + objects wholesale** — that defeats per-key merge. The current `commit()` reassigns + `cachedRundown.entries = entries`; the CRDT adapter needs to apply the *delta* instead. + - lowdb persistence stays as a **local durability layer** (or is replaced by Automerge's + own storage adapter). Either way the 3 s debounce semantics should be preserved. + +2. **Remote-change observer → existing side-effect path.** + - Subscribe to Automerge doc changes. When a change arrives **from the peer** (not from a + local mutation), run the same post-commit side effects that a local edit would: + - rebuild the loaded-rundown cache (`rundownCache.init` / `runtimeState.updateAll`), + - `runtimeService.notifyOfChangedEvents(metadata)`, + - `sendRefetch(RefetchKey.Rundown | …, revision, rundownId)` to local clients. + - This is the single most important hook: it makes remote edits indistinguishable from + local edits to everything downstream (clients, timer, integrations). + +3. **`revision` semantics.** Today `revision` is a per-rundown monotonic counter used only to + tell clients "you're stale, refetch". With two writers it can collide. Options: derive the + client-facing revision from the Automerge document heads/hash, or keep the counter as + advisory and rely on the refetch always pulling current truth. Recommend deriving a stable + version token from Automerge heads. + +4. **Playback intent channel.** + - Define a replicated `PlaybackIntent` ≈ `RestorePoint` + `offsetMode`, plus `messages` and + `auxTimers[1..3]` intent (`{playback, startedAtEpoch, duration, direction}`). + - Model as a **LWW register stamped with a logical (Lamport) clock + originating peer id**. + Every `runtimeService` command updates the local intent and publishes it; the peer + applies it if its stamp is newer. + - The follower applies intent via a **resume-style path** (`runtimeState.resume`-like) that + **recomputes TimeOfDay fields from `startEpoch` + clock offset + project timezone** — it + does *not* copy `startedAt`/`pausedAt` verbatim (see §2.3 timing note). + - Live ticks remain local; both nodes converge because they share intent + a common clock. + +5. **Clock synchronisation.** + - Both nodes must agree on epoch time within tolerance (target sub-100 ms for broadcast use). + - Recommend an **application-level offset estimate** over the sync socket (periodic + timestamped ping ⇒ Cristian's algorithm / NTP-lite), applied by the follower when + interpreting `startEpoch`. Do not assume both machines are NTP-disciplined, but benefit + from it when they are. + +## 6. Conflict & authority model + +- **Document edits:** resolved by Automerge (per-key map merge, RGA for arrays). Define a + policy for the rare same-key concurrent edit (Automerge picks a deterministic winner; we + may surface a "changed remotely" hint to the editor). Concurrent reorders of the same list + are the main thing to test (array CRDT semantics). +- **Playback:** a human operator drives it; genuinely simultaneous conflicting commands are + rare. LWW on the intent register (logical clock + peer id tiebreak) is sufficient for a + 2-node system and far simpler than a leader-election protocol. Revisit if N>2 is ever needed. + +## 7. Phased delivery + +1. **Phase 0 — Spec & spike.** Lock requirements (this doc). Spike `automerge-repo` WS + client/server between two local server instances; prove a doc round-trips. +2. **Phase 1 — Document sync (data only).** CRDT-back the `DatabaseModel`; remote-change + observer wired into the existing refetch/cache side-effect path. Directional initial seed. + No playback sync yet. Deliverable: edits on either node appear on both. +3. **Phase 2 — Clock sync + playback intent.** Offset estimation; replicate `PlaybackIntent`; + follower derives ticks locally. Deliverable: start/stop/pause/roll/load/add-time mirror. +4. **Phase 3 — Messages & aux timers.** Extend intent channel. +5. **Phase 4 — Resilience.** Reconnection/catch-up, project-switch handling, auth hardening, + conflict UX hints, observability (drift metrics, sync status). + +## 8. Open questions / decisions needed + +1. **Library:** confirm `@automerge/automerge-repo` (WASM core) vs alternatives (Yjs). Automerge + matches the keyed-map data model and brittle-free merges; Yjs is leaner/faster but more + text-CRDT oriented. *Recommendation: Automerge.* +2. **Baseline-owner UX:** when the two projects differ at initiation, is it always + "push mine / overwrite theirs", or do we offer "pull theirs"? (Merging unrelated projects is + explicitly discouraged.) +3. **Persistence:** keep lowdb as the local store and treat Automerge as the in-memory + sync truth, or move durability to an Automerge storage adapter? Affects crash recovery and + the existing `flushPendingWrites`/restore flow. +4. **Timezone authority:** anchor playback on the **project's configured timezone** (not each + machine's local TZ). Confirm where that timezone lives / whether it must be added. +5. **`report` data** (run history): sync as part of the document, or keep per-instance? +6. **Scope of N:** is 2 nodes the hard ceiling, or should the intent/authority model leave room + for more peers? +7. **Multiple loaded rundowns / background rundowns:** confirm the whole project document syncs + (all rundowns), while only the *loaded* one drives the runtime on each node. + +## 9. Risks + +- **TimeOfDay vs absolute epoch** across timezones (the single biggest playback-sync trap; §2.3). +- **Array/order merge** semantics for concurrent reorders — needs explicit test coverage. +- **`structuredClone`-and-replace** mutation style must be converted to deltas or it will + clobber concurrent edits and negate the CRDT. +- **Document growth / compaction** — Automerge history grows; plan periodic compaction/snapshots. +- **Bandwidth on the local↔cloud link** — fine for doc deltas + intent; would be a problem if + timer ticks were ever streamed (they must not be). +```