That hypothesis is sensible.
According to The Male Brain, by psychiatrist Louann Brizendine, males have 2.5x the space devoted to sexual drive in the hypothalamus compared to females.
By the time males reach puberty, guy’s sexual occupation and fertile female tracking circuits run in the background non-stop like an unkillable daemon process.
That’s absolutely going to divert brainpower.