While I do think Felger often goes overboard just for dramatic affect, he has said on many occassions that he doesn't consider Moss a "locker room cancer" in the true sense of the label in that Moss didn't constantly bring everyone down with b*tching and complaining, etc..
He has stated that he feels Moss is "high maintenance" though and that may have impacted how Brady ran the offense in an effort to keep him involved. As it's been pointed out several times, Brady was at his best finding the open receiver and Felger points to the record setting year of 2007 where that changed.
So, the main reason why he's been down on Moss is that in his opinion, his presence on the field contributed to Brady altering how he used to play and made him less consistent.
Felger and Mazz can still both be annoying d*cks though.