Honestly from what you're saying, I'd pass. Time is too precious. I mean, if you think you might have a Jamie and Adam type of relationship (mythbusters) where you don't always see eye to eye but you always do amazing work together, that's great. But if you don't see that now, and have that professional respect already, then I dunno.

Have you discussed this with him? And if not, why not? You'll be tackling much bigger issues together in the future, so tackle this one. What does he think? It might be as simple as saying "You know, we have a lot in common and I think we could work together, but our personalities just don't seem to click. I'm concerned about working together long term. What do you think?"