As for statistics, I have to admit that this subject is something of a blind-spot for me. This is in part because the subject of statistics, per se, is actually a separate discipline from pure mathematics. For example, at many academic institutions, the statistics department will be on a separate floor from the math department, if it is in the same building. Statistics courses at these universities don't necessarily require a background beyond calculus; furthermore, it is often the applications to the social sciences, psychology, and business which motivate the subject matter (or, at the very least, justify funding for the department).

For these reasons, it can be often difficult to find theoretical treatments of statistics from a purely mathematical perspective. Of course, this is not inherent in the subject, and Gauss, one of the greatest pure mathematicians of all time, did groundbreaking theoretical work in the subject (e.g., the method of least squares, the "Gaussian" distribution). That said, Gauss was also an applied mathematician, and I would hazard a guess that he was driven to invent these things in the course of empirical studies (such as his work in astronomy, in which he determined the orbit of Ceres).

What approach you might want to take to studying statistics depends on your background. Since most of my background is in pure math, I actually don't have a whole lot of experience myself. Somebody who desired to study statistics from the perspective of a pure mathematician would probably need to first become solidly grounded in probability. This requires a background in real analysis, and then measure theory. If you don't already know your real analysis, then you still have a very large hump to pass over before being able to even understand what measure theory is all about.

Of course, you don't need a measure-theoretic background to understand and use statistics. In fact, I don't even know if you'd need (or want) that to study statistics from the point of view of a pure mathematician. It just happens that almost all pure mathematicians treat the subject of probability using the foundation of measure theory, and a mathematician would probably want to understand probability first before statistics.

That said, although I have not tried to read it, the book titled "All of Statistics: A Concise Course in Statistical Inference", by Wasserman, looks to me like a fast-track to clearly understanding statistics from a mathematical perspective, if there ever was one. I'd still recommend having some background in combinatorics, proofs, and some basic real analysis before attempting it. You might want to look at the prerequisites.

If you don't have a pure mathematics background, or aren't inclined to pursue one, you should just do what 99% of all basic users of statistics do: turn to an expert in the field you wish to apply the subject who has written a book on USING statistics. The book will probably be a hybrid of a crash course in statistics and a tutorial of how to use it. You'll want to learn how to use the program "R". In fact, since statistics is foremost an applied subject, this is probably the best approach anyway. For example, most of what I know about statistics (which isn't a whole lot) comes from learning about the parts I needed to do basic error analysis for undergraduate physics labs. Experimental physicists have their own books that cover the parts of statistics they use, and I imagine that it would be the same for most other subjects. You could probably learn a whole lot about statistics if you were to pursue, for example, machine learning.

One part of pure math that you should learn no matter what is the "method of least least squares". This is an easy application of linear algebra, is often in linear algebra textbooks, and is probably the most (over)used tool of statistics.

Finally, if you do want to learn more about probability first, I would recommend the books of Robert Ash. He has one on non-measure theoretic probability, called "Basic Probability Theory". It's available on his homepage, or as an inexpensive Dover paperback. Perhaps you could try all three approaches simultaneously (read the Ash book to learn about probability, the Wasserman book to try to learn some mathematical statistics (if you are so inclined, although I should warn you that it presupposes a background in undergraduate mathematics, and is somewhat expensive), as well as some third book, which goes straight to the statistics in the area you wish to apply it).

Let me put the disclaimer on all this that I have not attempted to carry out any of the recommendations here, so, unlike my previous posts, this is mostly speculative. It could be that your best bet to learning the subject is to find somebody who knows the subject and ask him / her instead!