The big problem arises because players are able to interrupt after a set number of iterations of a loop to so what does your deck look like 25k iterations into the 51271 steps required to sort a 60 (edit: 59) card library?
Also, see what can happen with long-running combos, and it'll make a lot more sense: https://youtu.be/EXRnOhUfKwo
I'd do it for 98 cards though so you could apply the number in (c)EDH where you're likely to actually see this combo happen though. It would still be valid because if you can sort your library to an arbitrary state S in N moves you can sort it to a state S' that could result in S in any number of moves greater than N as well.
Maybe one could first divide the library into 3 piles and sort within each piles first.
In each N pass, we should see each element at least 3 times, and it must advance toward it's desired position at least once. since it can't be at the 'wrong' end of the triple multiple times.
Worst case in N^2 ops we should be done.
I'm curious about the distinction here. Is it needed? Isn't N divisible by N? Or is there a mathematically-meaningful difference?
as example: the library has 6 cards; the spell allows you to choose the top 3 cards and put them into the bottom in any order. So the spell only allows you to sort either the first set of 3 or the second set of 3 top cards, and there is no way to intermix the cards from the two sets of three => no way to sort the entire library.
Hope this make senses as to why N=3 is an exception as well!