The rumour I heard is that Polian regularly contacts the League after games to complain about calls that didn't go the Colts' way. If so, it's obvious why the referees are so intimidated.
And the facts are apparent to any neutral. Yes, I don't like the Colts -- but no less than I like, say, the Steelers, and remembering the Polamalu call still makes me angry. As DaB says, you don't need to just watch the Pats. Personally, I simply can't watch a Colts game unless the Patriots are a part of it -- and then I go through expecting the worst.