I think it all starts with beating them on their home field and hoisting the AFCCG trophy TWICE in '01 and '04.
Taking it one step further, Brady and Belichick have pretty much owned them in the W/L column over these past 15 years.
The icing on top would be the cheating allegations, which their fanbase fully believes are true. In my opinion, they hate the Pats just as badly as they do the Ravens, Browns, or Bengals. I can't tell you how many people around Pittsburgh couldn't make up their minds as to who they were rooting for in the 2011 and 2012 AFCCGs, so that really says something in my opinion.