Roster attrition and injuries derailed the early dynasty.
2005: Law, Patton, Fauria, Andruzzi, Johnson, Traylor, Phifer left via free agency or retired. Bruschi was lost for half a year after his stroke and was never the same player after. Harrison and Light were lost for the year.
2006: Branch and Givens left via free agency, and McGinnest was cut. Harrison was lost for the year in week 16
Bad fortune prevented the coronation of the 2007 team and ended the 2008 season in its first quarter. Bad drafting delayed the rebuild. Look at the list of high picks that busted or disappointed during this era.
2006: Maroney, Jackson, Thomas, Mills (only Gostowski panned out)
2007: Meriweather
2008: Wheatley, McConnell, Crable (only Mayo and Slater were successful)
2009: Butler, McKenzie, Tate, plus Chung was slow to develop
The 2009 - 2013 seasons were rebuilding years. The defense took a full five years as a result of bad drafting. The offense rebuilt itself overnight - twice. It's a testament to the greatness of TB12 that the team stayed competitive during these years.