Maybe it’s just me, but I’m not sure I’ll ever feel the same way about David Letterman again.

Since the late night talk show host made his dramatic confession on TV last week that he was being blackmailed for $2 million for having numerous affairs with female employees, I’ve been talking to different people about Letterman. It seems like everyone basically wants to give Dave a free pass because a) well, he’s Dave, and just about everyone likes Dave, b) we all hate extortionists, and c) we’re all pretty sick of these silly sex scandals.

Blackmail is wrong, no doubt about it. And it certainly sounds like Robert “Joe” Halderman, the Emmy Award-winning “48 Hours Mystery” producer who was arrested for the alleged extortion plot, is a real piece of work.

But that doesn’t mean Dave should completely get off the hook. After all, this is a guy who has been more than comfortable taking potshots at peccadillo-prone politicians and actors in his monologues and Top Ten lists for decades. And then he’s fooling around with his female employees? The people for whom he signs their checks? The folks who would likely give their eyeteeth to work for a major celebrity like David Letterman? (Dave reportedly even kept a secret bedroom at his studio for his trysts.)

One friend said to me over the weekend, “Can you blame him? It’s good to be king. Why not? Who cares if he was their boss? He wasn’t married at the time, and he’s David Letterman. More power to him. Anyone would do what he did. They’d be crazy not to.”

Another person said to me, “Why was it unprofessional or unethical? Lots of people sleep with their bosses. It’s nothing new, older than the hills. David Letterman is a very powerful man, a celebrity. He didn’t do anything wrong. No one has ever filed a sexual assault or harassment complaint against him. These women knew what they were doing. It was consensual. He was just being a guy.”

This was the reaction (believe it or not) from Kim Gandy, former president of the National Organization for Women: “I don’t really care who someone sleeps with, as long as it’s not coerced and as long as there’s not some explicit or implicit promise of favors or the like. It’s another adult—it’s not a minor. If that’s all it is, he’s a single guy and he had a fling.”

Meanwhile, one CBS insider praised Letterman’s attitude toward women on the set. “I’ve worked in a lot of places, and [`Late Show’] is one of the better places for a woman,” the insider told Fox News. “Dave’s not a groper.” (How noble.)

In general, the reaction from the public has been muted and uncharacteristically forgiving. The comic geniuses at “Saturday Night Live” barely touched the Letterman mess last weekend, and it appears that Dave’s advertisers and viewers are sticking by him.

I don’t want to sound like a choirboy here, but something’s not kosher. I know that employers have flings with their employees from time to time. But this certainly sounds like more than simply a little misguided moment due to an affair of the heart. It sounds like someone having a real pattern of taking advantage of a situation because of his celebrity, influence and prestige. In some circles, that’s known as an abuse of power.

Certainly, it wasn’t illegal. And Dave admitted that what he did was “creepy” and “terrible.” But still, something feels wrong here. I guess I just expected more of Dave. And maybe of the rest of us.

Stupid Americans and their ‘holier-than-thou’ BS. Grow up already and stop with the voyeursim. It’s not like he’s a pedophile or something. They were consenting adult women with whom he had sex. Big deal. Again, please grow up and move on to more important issues like healthcare reform, education, cleaning up and incarcerating the heinous criminals on Wall Street, etc.

And what about his son? And the mother of his son? Was he faithful to her?

Yes, this is show business. And yes, it’s wrong. We may in time find that the State of New York and U.S. labor laws agree that it was wrong.