Local-First, Now and Always!

Tue Aug 13 2024

Lukas Werner

Local-First Programming

Local-First programming is a set of programming principles that describes how data is managed by a program. Local-First allows the developer to program to a local copy of the data and not worry about how data gets synced between the different user’s devices or different users. The idea built on a technology called Conflict Free Replicated Data Types (CRDTs).

These are some of the great advantages to programming this way:

  1. No Spinners. All the data you need stays on your machine
  2. Offline. Since all the data is on your local machine you can work anywhere including without internet.
  3. Bankruptcy proof. Users can keep using the software regardless if the company that produced it continues to stay in business
  4. Collaboration. Documents can be shared without worry for conflicts

This kind of programming allows for software that is fundamentally multi-user (and multi-device) while also staying local and private.

Syncing SQLite via a Shared FS

A few weeks ago I saw this great blog post from Tonsky covering this talk given by Martin Kleppmann about the past present and future of local first. In that talk Kleppmann talks about this future syncing service that is generic across platforms and apps (sorta how iCloud storage objects work but cross platform). This syncing service doesn’t exist yet and isn’t close to being in the standardization process yet. Tonsky wrote how it could be possible to do this sync right now before this generalized syncing service protocol is done using shared drives.

Dropbox, Google Drive, iCloud Drive, Microsoft OneDrive, Syncthing and even thumb drives all use the same standardized abstraction, the file system. These services are vendor agnostic and local first. This makes them are a great way to start building local first apps. Just write your data to the disk, then the shared drive synchronizes it across your devices.

Great! Now we just need a way to save data to disk. There are hundreds of ways to do this. I personally prefer SQLite as it is a binary store that can be easily queried. Additionally, SQLite is one of the most deployed databases in the world with multiple copies all over your devices.1 Overall it is just really reliable and a good abstraction.

Cool so we have a sync protocol and we have a data store. How are we supposed to merge these two ideas together? This is where I’ve been stuck for the last little while. The main issue is our whole database is just one file. This is an issue because when the file gets synced there is undefined behavior. Any local changes could be overwritten by the sync service, or the sync service could provide two files forcing the user to merge two databases which is a terrible user experience.

This is where I was stumped until I read Hacker News the other day and saw a post about AutoMerge. Yes AutoMerge is a cool technology but I still wanted to use SQLite. But since I was curious I read through the comment section on that post and I rediscovered cr-sqlite. That was the moment I realized that the way to pull this off was to take the changes from each host and save them to a write only file. The app data store structure would look sort of like this:

./app.db
./app-sync/host1.changes
./app-sync/host2.changes

So to prototype this type of application I built a simple local-first Todo list app that I could later build in the data replication.

A demo of a Todo app running as a Text User Interface (TUI) from the terminal

To make this work is fairly simple first we open our database, second we upgrade our tables to Conflict Free Replicated Relations (CRRs) by running this simple function on the table:

SELECT crsql_as_crr('table_name');

Then we save the changes from a new table called crsql_changes that gets created by cr-sqlite to disk. If ‘real-time’ interaction is needed we can monitor the filesystem for writes to our files and update the database and UI real-time.

Source code of app is available on my GitHub.

Side note: This was the first project where I built a user interface using bubbletea. It is pretty fun and quite expressive.

Important Notes

Are there Disadvantages? A few.