Actually, you don't. A fortran compiler is optional, and used mostly for f2py and I think(?) for the LAPACK bindings. (The fortran bindings are more efficient than the C bindings for reasons I don't remember at the moment.)

Numpy's (and matplotlib's, actually) core is written in C.

Scipy is a different story, however. A lot of scipy's internals are written in fortran.

That having been said, fortran is a _very_ nice language for a significant portion of scientific programming tasks. IO in fortran is rather painful, though, i.m.o. Tools like f2py (which still doesn't really support fully modern fortran, unfortunately) make it very easy to combine fortran and python, for whatever it's worth.