Ignore the laws of statistics at your own peril.
Knowledge of statistics is the best investment one can make with one's time. A/B testing is the tip of the iceberg - a simple practical application, which can be exploited much better with statistics.
Learn about statistical distributions - Bernouilli, Binomial, Normal, Poisson and Hypergeometric at least, then Chi distribution to grasp real tests like chi2 goodness of fit, 2 way, calculating intervals - and you will see so many possible daily applications and avoid so many pitfalls (such as the ones outlined in the article)
BTW I saw in another comment something about significance. DeMoivre is not directly related to significance- it only means that if you know the population standard deviation, the standard deviation of any sample extracted from the population will depend on the size of the given sample - ie smaller samples will go below and above the expected value much more often
Consequently, if you try to deduce the SD of a population using a sample, the bigger sampler will give the best results (ie smaller or more accurate intervals)
Do you have any numbers to back that up? ;-)
I'd just say I understand the basic ideas such as DeMoivre or the Central Limit Theorem, and it helps me a lot.
First I'll assume you have a basic understanding of probabilities (odds, dices, cards, etc).
If you don't yet get probabilities, try the CK-12 books probabilities and advanced probabilities book - free on the kindle, and easy to read : http://www.amazon.com/CK-12-Probability-Statistics-Course-eb...
After that, my recommendation is to study the distributions suggested - from Bernoulli to Hypergeometric - and any source you "understand" will do.
The important thing is not the source, but to understand how these things work together, how they "articulate" - i.e. why taking a bunch of samples that follow any distribution will get you something that follow a normal law (LLN, CLT, etc) - even if the law they follow has a big hole in the middle, that'll where the mean of the normal law will be. Or under which conditions you can replace a law by another law, etc.
Then it's a good time to learn what moments do - how they shape the graphs you get. After that, you can try intervals - calculate intervals given a population parameters to see how a sample can predictably differ, then from a sample of a given size how you can estimate the population parameters.
After learning all that, to bind all this knowledge I'd suggest the free courseware on MIT 15_075 (even reading only the slides online on http://ocw.mit.edu/courses/sloan-school-of-management/15-075...)
I've recently "refreshed" my knowledge of statistics, and used the slides from 15 075 as a base. They get to the point and give a better mathematical understanding - something important to build your knowledge on a solid base after you understand how the things work together and what to go down the rabbit hole.
The course suggests the Tamhane and Dunlop book (which I haven't purchased yet but which is on my buy list) ; some other people recommended it to me for the demonstrations - I did the E(S^2n) E(S^n-1) by hand and I would love to see the proof for the Chi2 stuff, because I usually understand better after I see or do the demonstration.
Regarding Chi2, "Introduction to business statistics" has a great chapter #13, giving practical application, but I strongly suggest you understand the basics first - it's too easy to make mistakes with statistics.
Yes I don't fully trust myself with a tool as powerful as statistics - it takes a professional - but even with my limited understanding, I can see the value it provides, the warnings it gives (ie the article read like some basic logical stuff, but then I realized it wouldn't have been that obvious if I hadn't known basic statistics.)
To get the "logic of science" part, you also need to have (IMHO) some fairly decent grasp of combinatorics, for which I quite recently stumbled upon one of the best books in this field: "Notes on Introductory to Combinatorics." (I like the links to many of Polya's gems of "How to solve it.")
For many other references, a quick HN search for publicly available references will result in other endorsements, too (a preliminary version of the Jaynes' book used to be available, too)
- Any of Wainer's books (the author of the original pdf)
(Added later): books that explain the history of statistical thought are surprisingly good, because they explain the context and the problems the statistics were originally meant to solve. I really enjoyed "the lady tasting tea" and I think I learned from it.
I think a much larger problem is in the underlying assumptions that are made. For instance, assuming that an experiment on animals can be applied to humans (sometimes it can, sometimes it can't). These can be more nuanced and much harder to detect than a simple math error.
Also, the importance of truly random sampling is not emphasized enough. Even medical researchers are guilty of using international cluster sampling to make generalizations about the population. Overlooking sources of bias like geography, culture, lifestyle differences, etc.
I find overpopulation to be one of the greatest concerns that we are not adequately dealing with. Albert's bacteria in a bottle analogy made me think of Elon Musk's attempt to bring human life to other planets.. will we not just be expanding our time of growth? Maybe we should start choosing a way to reach a population growth rate of 0% and see what countless problems are resolved.
It seems to me that we have a choice:
1. Figure out how to cut our population growth to 0%, and then educate everyone on the planet as why they should do this instead of breeding to the max. Our kids will be happier than us.
2. Breed to the max until everyone on earth (and maybe Mars) is only just hanging on by their fingernails, probably living extremely unpleasant lives of violent competition and constant starvation. Our kids will be less happy than us.
Sadly I think option 1 is unlikely due to the tragedy of the commons.
I'm going to repost a my comment about this very concept as related to startups from a while ago because I believe HNers will appreciate it - it's from an article called "Startup School And Survivor Bias" (hope that's ok :)
Source: http://news.ycombinator.com/item?id=4685042
============================================================
Startups: never have so many understood so little about the statistics of variance present in the outcomes of small samples.
People like to speak of 10x productivity, non-stop work and geniuses - but the reality is much less interesting. A large number of small teams working on many different problems will by definition have a great variance in outcomes just by random extraneous factors (also known as the law of small numbers and insensitivity to sample size).
> A certain town is served by two hospitals. In the larger hospital about 45 babies are born each day, and in the smaller hospital about 15 babies are born each day. As you know, about 50% of all babies are boys. However, the exact percentage varies from day to day. Sometimes it may be higher than 50%, sometimes lower.
For a period of 1 year, each hospital recorded the days on which more than 60% of the babies born were boys. Which hospital do you think recorded more such days?
1) The larger hospital
2) The smaller hospital
3) About the same (that is, within 5% of each other)
56% of subjects chose option 3, and 22% of subjects respectively chose options 1 or 2. However, according to sampling theory the larger hospital is much more likely to report a sex ratio close to 50% on a given day than the smaller hospital.
Relative neglect of sample size were obtained in a different study of statistically sophisticated psychologists
-- http://en.wikipedia.org/wiki/Insensitivity_to_sample_size
> A deviation of 10% or more from the population proportion is much more likely when the sample size is small. Kahneman and Tversky concluded that "the notion that sampling variance decreases in proportion to sample size is apparently not part of man's repertoire of intuitions. For anyone who would wish to view man as a reasonable intuitive statistician such results are discouraging."
-- http://www.decisionresearch.org/pdf/dr36.pdf
Taking lessons as gospel from these "10x" events is by definition foolhardy and merely an extension of the bullshit pushed by the entire "Good To Great" Jim Collins business book industry.
It's like taking lessons from survivors of the Titanic on how to survive the sinking of a ship. It's quite simple - be a young female child with a life vest and rich parents (or in startup land - a young upper-middle class male living in California during a venture bubble, a cyclical investment in the Valley with a convergence of secondary technologies, above average intelligence and a college degree from a reputable university).
I have a personal rule with any kind of advice or explanation coming out of anyone working in a "soft" industry - if it's vague - it's bullshit. All of the advice given at these events are bullshit by this definition. So are many other things - and yeah it doesn't preclude me from spouting it. Or using the advice at my discretion.
But honestly - startup founders literally have no idea why things take off and they have no idea why they win. That's why they have to keep pivoting - it increases their luck surface area and their ability to gain traction - after which they simply must hold on tight while surfing the wave.
YouTube was a dating site - didn't work - pivot - video traction - venture up - ride.
PayPal was a Palm Pilot app - didn't work - pivot - traction - venture up - ride.
Google sold corporate search - didn't work - pivot - copy PPC from Overture - lever up - traction hits - ride.
Instagram - started with a location checking HTML5 app 2 years too early - pivot - copy PicPlz and Hipstamatic - hit traction - lever up - ride.
Angry Birds - fail at hitting nearly every game in the past decade - pivot - take a shot at the iPhone - hits traction - lever up - ride.
Of the startups that didn't pivot - they either skipped the pivot thanks to previous side projects/companies or already had traction - and all they had to do was lever up and ride.
I'm going to make this clear - there is absolutely, positively nothing wrong with this - not at all - it is merely reality and not particularly unfair.
People stating pointless platitudes that success is due to things like "Be 10x more productive", "Commitment" and "People, product, and philosophy" are simply wasting their breath, other people's time and confusing what actually happens. These things may or not be either actionable, predictive or sufficient for success.
Here's my list of startup advice:
Be alive. Be male. Be young. Don't have health issues. Be born in America or move there. Enter the cycle after a recession. Speak English. Enter a growing/new field where the level of competition is low and so is the sophistication of your competition. Surf cost trends down from expensive to mass consumer markets. Work bottom up - on small things. Be of above average intelligence. Have family support. Have a college degree.
Oh and most importantly of all: Get fucking lucky.
The hindsight/survivorship biases in combination with faulty causality and the narrative fallacy will completely hose your thinking - so be careful.
More interesting stuff:
http://en.wikipedia.org/wiki/List_of_biases_in_judgment_and_...
http://en.wikipedia.org/wiki/Black_swan_theory
http://en.wikipedia.org/wiki/List_of_fallacies
http://en.wikipedia.org/wiki/List_of_memory_biases
http://www.econ.yale.edu/~shiller/behfin/2000-05/rabin.pdf
Disclaimer: Biases rule your thoughts and mine - this post is also subject to both bullshit and biases (mostly bullshit - I do love that word). Think for yourself.
Collins primarily looked at established, large businesses. You won't catch him saying that his ideas primarily applies to start-up success. His point is that start-ups are all about luck, and he points out how many a lucky start-up blew their lead because they couldn't figure out how to establish a company.
So, rather being in opposition to your thoughts, there is a lot of common ground.
Other than that, some great thoughts. I would add that I have seen brilliant people that could not network or were not resilient. I think both of these need to be added to your startup advice, and are important success factors.
Sounds like you have some solid advice, pivot often.
Like twitter, right?
This would be a neat theory, if girls somehow used an average of the two X's... which seems compellingly logical, though the (current) theory is that only one X is used, chosen at random. http://en.wikipedia.org/wiki/Barr_body
These seem to be well-known cases in which you really do get a phenotype that's a function of both X chromosomes.
A guy has to use Mom's X. I think the paper's argument still holds, because there's a larger amount of possibilities for women than men, but I can't take this much further without circular reasoning.
For the dude there's a .5 prob of getting a great X and a .5 prob of getting an unknown X.
After looking up the book from which this chapter is excerpted, I followed other recommendations from Amazon to another very useful book,
http://www.amazon.com/When-Can-You-Trust-Experts/dp/11181302...
When Can You Trust the Experts: How to Tell Good Science from Bad in Education by Daniel T. Willingham, a very astute psychologist with an interest in education policy.
Why is this so important? The fact, that the variability increases with smaller sample size was ignored completely by the protagonists in the provided examples. Realizing weather this inverse effect is linear or not doesn't seem to be the main problem in peoples intuition.
disclaimer: I have poor understanding of statistics.
The rest of the examples had nothing to do with the specific relationship between standard deviation and sample size, but with the more basic fact that a relationship exists. This observation is arguably the more important one, and is poorly argued in the chapter. It's also why some people always demand error bars, though I personally prefer plotting individual data points where possible.
The last example, while interesting, had very little to do with the equation (despite a claim to the contrary), which makes me believe the topic was an afterthought.
The point of this article is that a sample only accurately reflects the whole in some ways. Variability in particular scales with the square root of the sample size. And since misconceptions about variability have been at the heart of many controversies (male vs. female intelligence, school size, cancer risk, etc.), De Moivre's equation is important; even dangerous in the sense that ignorance of it has led to billions of dollars wasted.
