The whole idea of handicapping is to make the race more fair, not less so.
http://en.wikipedia.org/wiki/Hull_speed
If you want to race two different boats (and they're almost all different) you will have to use some kind of handicap.
So any race without a handicap would immediately penalize the shorter boats. And then it becomes a race of who has the most money. The three different kinds of races (handicap, rating, one design) mean that everybody gets to compete how they want. Once you enter a race under those conditions when you win, you win for real.
And no-one in their right mind would enter a boat into a race they'd already lost before it started so you can consider any of that very hard fought for, sailing is tremendously competitive.
In the late 80's in between a lot of other stuff I did back then I worked for a while for a very state-of-the-art sailmaker in the Netherlands to write CAD software to create sail designs and cut-plans. So I'm not a sailor or anything like that but I've read quite a bit about it and worked with people that spent as much of their lives on the water as off and I know that they would take a remark like that in a very bad way.
For two boats that are equal the differentiating factors in actual speed (so otherwise equal conditions) are crew, sails, stays, mast. It's not rare to see a better crew out-sail a much better boat.
One boat is a cruise, two boats is a race.
The weather during the different time frames may vary wildly. It may work to your advantage, or to your disadvantage — on average, it will be canceled out. But rating/handicap systems sure make that there's more luck involved than in one-class races. Not that that's a bad thing.
