← All articles

File sync with chunks, hashes and conflicted copies

Updated Nov 10, 2026 · 4 min read

A sync service keeps the same folders on many devices. Change a file on your laptop and it appears on your phone a few seconds later. Doing that naively means uploading the whole file on every save, which is fine for a note and hopeless for a 2 GB video. Real sync services cut files into chunks, name each chunk by its contents, and move only the chunks that changed.

Chunks named by their contents

A client splits each file into chunks of a few MB and hashes each chunk with SHA-256. The hash becomes the chunk’s name. A file is then just an ordered list of hashes. This one idea, called content addressing, does a lot of work.

  • Upload only what’s missing. The client sends the list of hashes, the server says which it lacks, and only those go up.
  • Store each chunk once. Identical bytes give identical hashes, whether they’re in two versions of one file or the same installer saved by a thousand people.
  • Resume broken uploads. Chunks that made it are already stored, so a dropped connection costs one chunk, not the whole file.
  • Check integrity for free. If the bytes don’t hash to the chunk’s name, they’re corrupt.

Fixed chunks vs content-defined chunks

The simplest cut is every 4 MB. It works well when edits overwrite bytes in place. But insert one sentence at the start of a file and every byte after it moves along. Every 4 MB boundary now lands somewhere new, so every chunk changes and the whole file uploads again.

Content-defined chunking fixes that. A rolling hash slides over the file a few dozen bytes at a time, and wherever its value matches a pattern, such as the last 13 bits all being zero, that’s a boundary. Boundaries depend on the bytes around them, so after an insert they land on the same content as before. Only the chunk around the edit changes. Backup tools such as restic and borg work this way.

Fixed sizeContent-defined
Where it cutsEvery N bytesWhere a rolling hash matches a pattern
Edit in placeOne chunk changesOne chunk changes
Insert near the startEvery chunk changesOne or two chunks change
CostAlmost freeA pass over the bytes
Chunk sizesAll equalVary around an average, with a min and max

Metadata and blobs live apart

Chunks are large, written once and never changed, so they go in cheap blob storage. Everything else is metadata, meaning which files exist, where they sit in which folder, each file’s versions, and the chunk list of each version. Metadata is small, changes constantly and needs transactions, so it lives in a database. Keeping them apart lets each scale its own way.

1missing chunks2commit3changedLaptophashes chunksSync serviceBlob storechunks by hashMetadata DBfiles, versionsPhonenotified
Chunks go to the blob store first, then the metadata commit makes the new version visible, then other devices are told.

One save, end to end

  1. The client notices the file changed, cuts it into chunks and hashes them.
  2. It asks the server which hashes are missing.
  3. It uploads only those chunks.
  4. It commits the new version, a list of hashes, naming the version it started from.
  5. The server tells the user’s other devices, over a connection each one keeps open.
  6. Each device fetches the new chunk list and downloads only the chunks it doesn’t have.

The order matters. Committing the version before its chunks are stored would let another device fetch a file that can’t be built yet.

How much it saves

Say you save a 300 MB file 20 times a day and each save touches two 4 MB chunks. Whole-file sync sends 6,000 MB. Chunked sync sends 160 MB. That’s about 5.8 GB a day saved, for one person and one file, and it’s why a sync service can run on phone data at all.

Whole file each save6,000 MB
Changed chunks only160 MB
MB uploaded a day, for 20 saves of a 300 MB file.

When two devices edit offline

Every commit says which version it was based on. The server accepts it only if that’s still the latest version, a check called compare-and-set. If your laptop and phone both edited version 1 offline, the first to sync wins cleanly. The second arrives based on version 1, which is no longer the latest, and the server knows it has a conflict.

A general sync service can’t merge arbitrary files. Mixing the bytes of two spreadsheets gives a corrupt file. So it keeps both. The first version keeps the name, and the other is saved beside it as a conflicted copy. Picking the later clock time instead would silently throw away someone’s work, and device clocks drift anyway. Apps that understand their own data, like a shared document editor, can merge edits instead, which is a different design.