The most common reason I downvote is because a comment is some sort of logical fallacy. Mostly ad hominem, where instead of contributing to an intelligent discussion, insults get thrown around. I do not enjoy mud-slinging and would rather engage in an intelligent discussion.

In this case a downvote reflects the low quality of the comment more than my disagreement with the poster. While I'm hesitant to upvote a comment if I disagree with the author's position, I at least will not downvote it.