> You can make a search for all users, which will tell you there are 97,616,627 users at the time of this writing, but you can only fetch at most 1000 results from a search, and they don’t come in any clear order, so you can’t just make the next search start where the previous one left off (or I didn’t figure out how).

> What you can do though is request accounts created in a certain time range. If you get the time range right, so that it has less than 1000 entries, you can paginate through it, and then request the next time range.

This reminds me of when I tried to add a google drive storage backend to camlistore/perkeep (because I had nearly-unlimited free quota at the time). One of the things a perkeep blobserver needs to be able to do enumerate all the blobs it has, in order. You can send millions of blobs to google drive without issue, but you can't directly paginate a search for them in sorted order.

You could just issue a search for all blobs under your perkeep drive folder, keep paginating the result until you run out of pages, and then sort in memory, but there's really no way of knowing how many blobs you're going to end up with and you might blow out your blobserver's memory.

Perkeep blobs are identified by blobrefs, SHA sums of the contents of the blob, so they look like sha-[0-9a-f]{64}. Google drive lets you search for files with a name prefix, so you can search for like /perkeep/sha-* and see if the result has a pagination token (indicating that there are more than 1000 results), and if so then you search for each of /perkeep/sha-0*, /perkeep/sha-1*, ... , /perkeep/sha-f*, each time checking to see whether there are too many matches. When there's not too many matches, you've found the prefix length that will let you fetch a bounded number of blobrefs, emit them to the perkeep client, and then release the memory before fetching more.

  /pk/sha-\*          1000+ results (non-empty pagination token)
    /pk/sha-0\*       1000+ results (non-empty pagination token)
      /pk/sha-00\*    1000+ results (non-empty pagination token)
        /pk/sha-000\*  193  results,
                       sort these in memory and emit to client
        /pk/sha-001\*  179  results,
                       sort these in memory and emit to client
        ...
        /pk/sha-fff\*  223  results,
                       sort these in memory and emit to client

I didn't end up landing the patch before I lost interest, partly because it was pretty much the first golang I had tried writing. It was fun working out the above details, though.