The only time I get annoyed about karma is when someone rates me down for a stupid reason. Other than that I don't really care.
The system was designed to filter out posts that don't belong in the forums, not to be used as a way to "punish" someone who writes something you don't agree with. If you get into an argument with someone, it's almost certain that you'll be rated down, most likely by the person you're arguing with.
I don't know what the US constitution says about censorship, but ours forbids it in every way ("Censorship is forbidden and can in no way be reinstated"). Sure, you have to risk the consequences of saying what's on your mind (someone calling you an idiot, for instance), but you have a right to say it. It got us into trouble with the Middle-east because they get more butthurt over religion than anyone else in the world, but dammit, we said it! Or drew it, actually.
What I don't care about is the personal karma thing. It's fun to watch your personal rating go up, sure, because it means you're doing something right and the community likes you, but once you're at 20k posts, you're pretty much locked into your rating anyway.