Testing Sync at Dropbox (opens in new tab)

here's some of the greatest hits from memory:

1) when we upload a file, we first ask the server which blocks we need to upload. this allows us to deduplicate blocks with other files or previous revisions of files you have. I had written some code like...

  let blocks_needed: HashSet<BlockHash> = ...;
  let file_blocks: Vec<BlockHash> = ...;
  let to_upload: Vec<_> = file_blocks
    .into_iter()
    .filter(|b| blocks_needed.contains(b))   
    .enumerate();

  for (offset, block_hash) in to_upload {
    ...
  }

the bug here is that we're computing the offset into the file after filtering, so we'd be uploading incorrect contents to the server. we have protections elsewhere that would prevent file corruption, but finding this bug pre-commit with trinity was pretty awesome.

2) nontrivial interactions between different components in the system are hard to test manually and come for free with trinity. trinity will simulate cancelling file transfers, crashing the system, and even dropping the local databases (to simulate disk corruption). these are all hard enough to test in isolation, but knowing how to combine them is almost impossible. so, having "automated test generation" for these cases is really useful.

3) the "theory of trees" seems like it'd be pretty simple, but canopycheck has found some really interesting cases, especially around moves. when we started, we hadn't really thought about how to handle move cycles, concurrent moves that cross ancestor chains, ...

So... that’s weird. I have a Macbook Air - not the most powerful machine on the planet - and a nearly full 1 TB dropbox with ~3 million files. Initial resync on boot takes under a minute.

How much data (total size/count of files) do you have?

hey, can you file a bug from the app? it will collect a report that we can use to diagnose. there's a "Report Bug" option adjacent to "Preferences..."

I'm not Sujay, but I worked on this system once upon a time too! (not anymore)

1. It does redo the minimization phase, but the actual execution is extremely fast, so this cost is minimal. Storing test outputs gets pretty expensive when you are running millions of tests, and since there are very few failures, recomputing this is worthwhile

2. Yes and no! the article talks about this a little, but the "heirloom" system does essentially this, and the "native" filesystem variant of Trinity runs the same Trinity tester code against a real filesystem. The "no" is due to the issues with randomized testing -- since any operation that you do can affect the RNG, the exact operation that is run for a particular seed can change if you swap any part of the system. For regression tests, the operation sequence can be put into a separate, non-randomized test.

3. both

4. Testing in Nucleus is a sliding scale from "unit-test-like" to "integration-test-like" -- Trinity is mocking plenty of functionality; CanopyCheck is simultaneously testing many different components. It would probably be more accurate to say that CanopyCheck is testing a smaller subset of components with much greater fidelity, and Trinity is testing as much of the sync engine is practical.

yeah, we don't merge file contents. it's really hard to do this well, since you'd need to implement something like operational transforms for every different file type, and the cost of getting wrong is really high, since we'd be corrupting users' files.

I can see us perhaps doing this in the future for more targeted file types where the "rebase" behavior is well-defined, though.

I don't think Dropbox does merge at all does it?

The file object model wouldn't work for merging Word documents and such