I'm completely convinced it has little to do with size and everything to do with demeanor and how we project ourselves. I'm fit but only 5'10", 175 and I've been told numerous times that people thought I was an LEO. You don't have to walk around like Officer Levitt for people to pick up a vibe.
Getting back to HCountyGuy's post above about profiling and stereotyping. Let's put it in the context of dogs.
You see a Dobie, a Rottie, a Shepherd, a Pit, and deservedly or not you're going to react differently than you would to a Cocker or a Beagle even though the latter two may be more inclined to inflict a bite.
Size, reputation, ability, traveling alone or in a group, coming toward or moving away, eye contact or not...all these things are going to be on our radar whether based upon current circumstances or "reputation". I see it as a tool. No more, no less. If our actions are solely based upon what we actually encounter, no harm, no foul...no matter what is considered "politically correct".