Declare what data goes where. The engine handles the rest.
LakeSync is an open-source TypeScript sync engine. Pluggable adapters connect any readable or writable system. Declarative sync rules define what data flows between them. Every adapter is both a source and a destination — local SQLite, Postgres, MySQL, BigQuery, or S3/Iceberg.
Consumer ↔ Gateway ↔ Any Data Source
Consumers push local deltas and pull filtered subsets from remote sources via the gateway. Adapters abstract the data source. Sync rules control what data each consumer receives.
Any source, any destination
Two interfaces: DatabaseAdapter for SQL-like systems and LakeAdapter for object storage. All three database adapters materialise deltas into queryable destination tables. Source connectors poll external APIs (Jira, Salesforce) and push changes as deltas.
Postgres / MySQL
DatabaseAdapter. insertDeltas, queryDeltasSince, getLatestState, ensureSchema.
BigQuery
DatabaseAdapter. Idempotent MERGE inserts. INT64 HLC precision. Clustered by table + hlc.
S3 / R2 (Iceberg)
LakeAdapter. putObject, getObject, listObjects, deleteObject. Parquet + Iceberg table format.
Jira
Source connector. Polls issues, comments, and projects via Jira Cloud API with cursor-based change detection.
Salesforce
Source connector. Polls accounts, contacts, opportunities, and leads via SOQL with LastModifiedDate cursors.
Custom adapters
Implement either interface for any data source. CompositeAdapter, FanOutAdapter, and LifecycleAdapter for advanced routing.
Declarative filtering via sync rules DSL
Bucket-based filtering with eq, neq, in, gt, lt, gte, lte operators and JWT claim references via jwt: prefix. The gateway evaluates rules at pull time and returns only matching deltas. Sync rules define what data each consumer receives.
Route data between any two adapters
Adapters are bidirectional — they implement both read and write. Sync rules define directional flows between adapters. The gateway reads from one adapter and writes to another.
Persistent outbox with automatic drain
Deltas queue in an IndexedDB outbox that survives page refreshes and process crashes. When connectivity returns, the outbox drains automatically and remote changes sync down.
Column-level LWW with HLC ordering
Concurrent edits to different columns of the same row are both preserved. Hybrid logical clocks (48-bit wall + 16-bit counter) provide causal ordering. Equal timestamps use deterministic clientId tiebreaking.
Poll external APIs and ingest as deltas
Source connectors extend BaseSourcePoller to poll external APIs on an interval and push changes as deltas. Memory-bounded streaming accumulation keeps resource usage predictable. Built-in connectors for Jira and Salesforce. Extend the base class for any API.
Design decisions
Adapter interfaces
DatabaseAdapter for SQL-like sources (insertDeltas, queryDeltasSince). LakeAdapter for object storage (putObject, getObject). Both are bidirectional.
Hybrid Logical Clocks
64-bit branded bigint — 48-bit wall clock + 16-bit counter. Causal ordering across clients without centralised coordination.
Column-level LWW
Conflicts resolved per-column, not per-row. Concurrent edits to different fields never overwrite each other.
Result<T, E> everywhere
Public APIs never throw. Error paths are explicit, composable, and impossible to accidentally ignore.
Real-time WebSocket sync
WebSocketTransport uses binary protobuf framing for push, pull, and server-initiated broadcasts. Auto-reconnects with exponential backoff.
Sync rules DSL
Declarative bucket-based filtering with eq, neq, in, gt, lt, gte, lte operators and JWT claim references. Pure function evaluation — filterDeltas() has no side effects.
Deterministic delta IDs
SHA-256 of stable-stringified payload. Same logical change always produces the same deltaId. Enables idempotent processing.
React bindings
LakeSyncProvider, useQuery, useMutation, and useSyncStatus hooks. Reactive local-first data access for React apps.
Source polling
BaseSourcePoller base class with memory-bounded streaming accumulation. Built-in connectors for Jira and Salesforce. Extend for any API.
Experimental, but real
14 packages, 3 apps. Core sync engine, conflict resolution, client SDK, React bindings, Cloudflare Workers gateway, self-hosted gateway server with WebSocket support, compaction, checkpoint generation, sync rules DSL, cross-backend flows, source connectors (Jira, Salesforce), and adapters for Postgres, MySQL, BigQuery, and S3/R2 are all implemented and tested. API is not yet stable — expect breaking changes.