They are quite annoying, as are cyclists. Sounds like quite an infruriating moment, but I don't think I would've got to the point of getting out of my car to have a word with him. Knowing the sorts around here that would've given him a reason to find where I live and key my car or something of similar effect. A couple of hand gestures and toot of the horn is enough in most cases, luckily I've only had to do that once so far.
My dad once did the simple thing of over-taking a bike, in a completely legitimate place when the biker was going 10MPH under the limit. The biker then sped up and followed my dad back home (only a few miles, but still), my dad pulled the car onto the drive and got out. The biker then pulled up and got off the bike and started ranting and raving, my dad just walked into the house and said "I did nothing wrong" and shut the door. Guy stood there for a few minutes banging on the door before getting the message and buggering off.
Flipping idiots think they own the road.