That's an interesting metaphor because I picked it up somewhere else, but it might well be that Hunt and Thomas actually coined this phrase in "The Pragmatic Programmer".
If you think of orthogonality in geometry, you can describe any N-dimensional vector space through a set of N orthogonal vectors (say the unit vectors in X, Y, Z). "Orthogonal" in this case means that moving something along one of those (say the X axis) won't alter its position from any other base vector's perspective.
In the same way, "orthogonal implementations" is a way of saying that the things are independent from each other. You can find their explanation in the Google preview for the book [1].
So, say you write utility code for a couple of applications, for example: Message transmission and subscription management. Then, you should try and keep both as independent as possible from each other, so that as a downstream user I won't have to pull in or even "massage" both just to use one of them.
As I wrote before, that sounds totally obvious when said out loud; yet I constantly find new examples where someone found a way to unnecessarily "complect" [2] multiple things.
[1]: https://books.google.ch/books?id=LhOlDwAAQBAJ&pg=PT70&dq=ort...
[2]: If that word also seems strange to you, I highly recommend watching https://youtu.be/oytL881p-nQ
