If you have block D, and want to fetch block A, you don't actually know if it's the real block A unless you also have blocks B, and C. Block D (which you have) contains the hash of block C, which contains the hash of block B, which contains the hash of block A. You need the hash of block A to be able to verify that it is indeed the real block A.
Of course, because of things like SSL/TLS, you can be sure nobody tampered with the block on it's way from the server to you. With that in mind, ensuring you receive the real block just requires you to trust the server giving you the blocks, which may or may not be worth it depending on how much time/space it saves you. In some ways, the server would become your 'central authority' on the block-chain.
In reality though, I can't really imagine there's much they could do. Sending you fake blocks may work for a while but would fail if the client caches them and asks for the surrounding blocks (Their fake block isn't going to match the hash of the real block). I think it would be a bit hard to pull off for any length of time.
It suffices to have the headers of blocks B and C. Which, at only 80 bytes per header, is quite manageable.
Imagine that block n, when produced, in addition to the hash of the previous block n-1, gave the hash of block n-2, n-4, n-8, ..., n-2^n. This implies that the block size grows linearly, but it should give you the following: whenever you request and obtain a past block A and you have the current block D, you can use these pointers starting at D to easily request and obtain a sequence of blocks (of length log n) which allows you to authenticate A from D. (Of course the algorithm to reach from A is simply to request the first block authenticated in the header of D which is after A.)
