I think we’re almost on the same page (heh). Pagination may still make sense for a variety of reasons. Even if there’s no meaningful sever optimization gain, clients may benefit from a reduced response size (mobile, expensive data plans, low power devices). That sort of thing is where ensuring consistency (for the sake of brevity I’ll repeat, at a point in time, but there are other ways to allow clients to negotiate this) at the request level over multiple requests is useful, even if the underlying data is changing much faster than the client is consuming it.

It’s worth noting here that this isn’t just applicable to paginated lists. It can also be used where you want to let the client optionally, concurrently access related resources. It can be used for client-side concurrent transactional writes. It’s a barrel of monkeys.

For what it’s worth, I wouldn’t assume their volume of traffic was necessarily the reason the data was in such flux. It could be (and I strongly suspect) that their popularity algorithm just stinks (eg weighting 100% of 1 view over 90% of 100 views). Even so, a snapshot in time is probably a much easier lift/gain for a flailing algorithm than really digging into the data and analytics to get an optimal stable sort without it:

1. Take a snapshot of the query and throw it in a dumb cache.

2. Rate limit cache key creation rather than cache hit requests.

3. Throw the caches out (and repopulate if reasonable) every [vaguely not gonna break the bank age duration].

4. Forget about any further optimization unless you reallllly need it.

5. Document the problems with the suboptimal solution, have a retro so your junior devs can develop better instincts, get them onto the next user story.

6. Put a “should we improve this” on the backlog of backlogs.

7. Profit.