Carroll's agent will fan the flames with this speculation every time there's a vacancy to see if they can eek any more money out of USC.
While it might have some slight adverse impact on recruiting, the more it happens the more potential recruits will disregard the annual rumors. Besides, all Carroll needs is a 2 week window in which it seems he'd consider a move to wrest more $$ from USC... then he repairs any damage with recruits and continues to have consistent good years.
The aspects of Carroll's personality that worked to his detriment in the NFL are precisely the same personality aspects that work to his favor in College.
Carroll's smart enough to know that by now.