To be clear, loading data from disk takes milliseconds and re-computing addresses takes nanoseconds. Yes, in less than one tenth the time it takes to read the part of the libraries you have linked into your binary from disk, you can locate and fixup any link references.

This part I don't get "no sensible system loads the entire executable when it's statically linked." If you're a statically linked executable, by definition the entire file is headed into memory, if there was something in the library you didn't use it got edited out in the link step. Now you may mmap the file and fault it in as you go along, but you are going to have the whole thing read.