"Just because I'm a man doesn't mean I'm a misogynist."
"Just because I'm white doesn't mean I'm a racist."
"What are you talking about? I worked really hard to get where I am today!"
And the kicker is, most of the time those people are right: they're not misogynists, or racists, or slackers. They did work hard to get where they are. But that's not what "privilege" implies. Privilege doesn't have to be something you insist upon, or some malicious plan to one-up your fellow human beings.
Most times, we're born into it, accustomed to it, and blind to it. Trust fund kids didn't finagle their placement in the parent lottery; but I doubt we'd say they weren't privileged as a result of that accident of chance. Of course, a trust fund kid's privilege is obvious. It's less obvious in other cases. Which is what I think the following comic illustrates rather well.
DailyKos on Facebook (and from tumblr @cartoonpolitics before that).
Privilege is rarely something we're choosing to exercise. And yet we can still benefit from it, even if we're not actively seeking it out. Reaping those benefits inadvertently, however, doesn't absolve us of the duty to eradicate injustice when we see it. And privilege very often manifests as injustice, be it social, legal, etc.