A guide to private alpha testing (opens in new tab)

I'd be curious to learn how others approach private alpha (how long to stay in it, who to include, what to test, how to think about feature additions/pivots, etc...)