I love the opportunity to sit and program as half of a pair. I'm sure it's not for everyone, but I too was once one of those who resisted the idea. Part of my resistance was my fear that people might realize I didn't know as much as they thought I did. I got over that. Many people who have given an honest effort to pairing (done by the rules) have found out that it's actually very enjoyable and effective.
Another common resistance is misunderstanding of what pairing is, and of what benefits you might get from it. To be effective, you can't just sit near someone else and expect magic to happen. The rules are simple, but not obvious. It makes perfect sense to me why someone would hate pairing after doing it poorly.
The least obvious rule is "change pairs frequently." A typical conception is that pairs are married at the hip for days at a time or even weeks. No, I'd slit Tim's wrists and he'd slit mine were that the case. Instead, we switch pairs often. The tedium of dealing with one person all day long aside, one of our goals in switching often is to ensure that not just two, but at least three people contribute to the solution of a task.
The downside of frequent switching is the overhead cost of context switching. It takes time to explain things! But that's where the synergies in the original XP practices come in: If you're doing TDD well, following simple design, coming up to speed on the test at hand isn't a terribly difficult proposition. And in fact, the need to minimize context switching overhead is a subtle force in the direction of improving code quality.