Yes, perhaps you don't have to argue all of that. But Damore did. Perhaps if he hadn't made those assertions, his memo wouldn't be seen as particularly interesting or controversial.
But we don't have to engage in such hypotheticals. Let's consider the premise that Damore's memo is simply a call to debate based on science and facts. In my opinion, such a non-biased, open-minded essay would not include this kind of assertion:
> We have extensive government and Google programs, fields of study, and legal and social norms to protect women, but when a man complains about a gender issue issue affecting men, he’s labelled as a misogynist and a whiner [10]. Nearly every difference between men and women is interpreted as a form of women’s oppression. As with many things in life, gender differences are often a case of “grass being greener on the other side”; unfortunately, taxpayer and Google money is being spent to water only one side of the lawn.
Only someone ignorant of basic facts and history would make this sweeping claim. And only someone ruled by his emotions would be unable to take a step back and do a Google search to see if anti-discrimination law has been used on behalf of men. Given that kind of massive error, or inability to recognize one own's ignorance, I don't think we should continue assuming that Damore's premises are factual or coherent in their arrangement. Which is why I'm giving him the benefit of the doubt in saying that it'd be better if he could have actual empirical evidence on the harm of Google's diversity programs. Just because his memo contains facts doesn't mean that it's truthful, if those facts are used loosely to further the author's emotional appeals.
edit: Fixed the formatting to show that [10], in the original memo, refers to a footnote and contains a hyperlink.
The footnote:
[10] “The traditionalist system of gender does not deal well with the idea of men needing support. Men are
expected to be strong, to not complain, and to deal with problems on their own. Men’s problems are more
often seen as personal failings rather than victimhood, due to our gendered idea of agency. This
discourages men from bringing attention to their issues (whether individual or group-wide issues), for fear
of being seen as whiners, complainers, or weak.”
The URL of the hyperlink (the text of which is "misogynist and a whiner"): https://becauseits2015.wordpress.com/2016/08/06/a-non-femini...
(Pretty sure "becauseits2015.wordpress.com" would not pass as a useful source on Wikipedia, but YMMV.)