The power losses on a perfectly smooth road are aero and rolling resistance. But once you get onto bumpy surfaces, now you also need to consider the power losses of the bumps on the bike AND on the rider because the rider is the one supplying the power, even the power which does not transmit through the cranks.
Although I do agree with you partially, if you're racing, ride what ever makes you fastest. Energy efficiency isn't paramount, getting to the finish first is. But if you're not racing, then really consider if the tradeoff of comfort is worth it.
The BRR website's test protocol uses a fairly smooth metal drum for testing. This is good for testing rolling resistance but not for testing total system losses over rough roads. So although BRR is a great resource, take it with a grain of salt.