This is called certificate pinning (a.k.a. Trust On First Use or TOFU). If you use Firefox, just install Certificate Patrol.
If you are interested in the rationale, and how you can use technologies like DANE to make it even better, read the paper by Gabor Toth and Tjebbe Vlieg (http://staff.science.uva.nl/~delaat/rp/2012-2013/p56/report....).