File sync with chunks, hashes and conflicted copies
Updated Nov 10, 2026 · 4 min read
A sync service keeps the same folders on many devices. Change a file on your laptop and it appears on your phone a few seconds later. Doing that naively means uploading the whole file on every save, which is fine for a note and hopeless for a 2 GB video. Real sync services cut files into chunks, name each chunk by its contents, and move only the chunks that changed.
Chunks named by their contents
A client splits each file into chunks of a few MB and hashes each chunk with SHA-256. The hash becomes the chunk’s name. A file is then just an ordered list of hashes. This one idea, called content addressing, does a lot of work.
- Upload only what’s missing. The client sends the list of hashes, the server says which it lacks, and only those go up.
- Store each chunk once. Identical bytes give identical hashes, whether they’re in two versions of one file or the same installer saved by a thousand people.
- Resume broken uploads. Chunks that made it are already stored, so a dropped connection costs one chunk, not the whole file.
- Check integrity for free. If the bytes don’t hash to the chunk’s name, they’re corrupt.
Fixed chunks vs content-defined chunks
The simplest cut is every 4 MB. It works well when edits overwrite bytes in place. But insert one sentence at the start of a file and every byte after it moves along. Every 4 MB boundary now lands somewhere new, so every chunk changes and the whole file uploads again.
Content-defined chunking fixes that. A rolling hash slides over the file a few dozen bytes at a time, and wherever its value matches a pattern, such as the last 13 bits all being zero, that’s a boundary. Boundaries depend on the bytes around them, so after an insert they land on the same content as before. Only the chunk around the edit changes. Backup tools such as restic and borg work this way.
| Fixed size | Content-defined | |
|---|---|---|
| Where it cuts | Every N bytes | Where a rolling hash matches a pattern |
| Edit in place | One chunk changes | One chunk changes |
| Insert near the start | Every chunk changes | One or two chunks change |
| Cost | Almost free | A pass over the bytes |
| Chunk sizes | All equal | Vary around an average, with a min and max |
Metadata and blobs live apart
Chunks are large, written once and never changed, so they go in cheap blob storage. Everything else is metadata, meaning which files exist, where they sit in which folder, each file’s versions, and the chunk list of each version. Metadata is small, changes constantly and needs transactions, so it lives in a database. Keeping them apart lets each scale its own way.
One save, end to end
- The client notices the file changed, cuts it into chunks and hashes them.
- It asks the server which hashes are missing.
- It uploads only those chunks.
- It commits the new version, a list of hashes, naming the version it started from.
- The server tells the user’s other devices, over a connection each one keeps open.
- Each device fetches the new chunk list and downloads only the chunks it doesn’t have.
The order matters. Committing the version before its chunks are stored would let another device fetch a file that can’t be built yet.
How much it saves
Say you save a 300 MB file 20 times a day and each save touches two 4 MB chunks. Whole-file sync sends 6,000 MB. Chunked sync sends 160 MB. That’s about 5.8 GB a day saved, for one person and one file, and it’s why a sync service can run on phone data at all.
When two devices edit offline
Every commit says which version it was based on. The server accepts it only if that’s still the latest version, a check called compare-and-set. If your laptop and phone both edited version 1 offline, the first to sync wins cleanly. The second arrives based on version 1, which is no longer the latest, and the server knows it has a conflict.
A general sync service can’t merge arbitrary files. Mixing the bytes of two spreadsheets gives a corrupt file. So it keeps both. The first version keeps the name, and the other is saved beside it as a conflicted copy. Picking the later clock time instead would silently throw away someone’s work, and device clocks drift anyway. Apps that understand their own data, like a shared document editor, can merge edits instead, which is a different design.