Huge? Probably not, but the poor drafting has probably cost your team at least 1 Super Bowl. When I ran through the drafts I was shocked to see just how bead New England has been after the first round. The round 2 and 3 track record almost defies logic. I think it really strengthens his resume as one of the greatest coaches ever, but if they had a better picks they would have more titles, IMO.
The Patriots hit a bad spell from 2006-2008, and 2009, while an improvement, has turned out to be largely disappointing as well. As an overall proposition, though, the team is at the top of the drafting heap. Take the second round, for example, which is a round famously derided by the critics. Not counting 2011, which saw injuries keeping the team's picks on the sideline and, therefore, unavailable for analysis, here's the list of draftees under Belichick:
2010 10 42 Rob Gronkowski TE Arizona
2010 21 53 J. Cunningham LB Florida
2010 30 62 Brandon Spikes LB Florida
2009 2 34 Patrick Chung DB Oregon
2009 8 40 Ron Brace DT Boston College
2009 9 41 Darius Butler DB Connecticut
2009 26 58 S. Vollmer T Houston
2008 31 62 T Wheatley DB Colorado
2006 4 36 Chad Jackson WR Florida
2004 31 63 Marquise Hill DE Louisiana State
2003 4 36 Eugene Wilson DB Illinois
2003 13 45 Bethel Johnson WR Texas A&M
2002 33 65 Deion Branch WR Louisville
2001 17 48 Matt Light T Purdue
2000 15 46 Adrian Klemm T Hawaii
That's 7 hits in 15 picks, which is right around the 50% one should expect, and that's with two players (Brace, Cunningham) who've still got at least a chance to turn things around.
While I certainly agree that the drafting in 2006-2008/2009 had a significant negative impact upon the Patriots, I'd also note that some big free agent misses, and some poor trades, have done the same thing. It's great for us to point to one area or another as "the" reason the Patriots failed in any given year, but it's really been a total package issue, IMO. If Joey Galloway and Shawn Springs don't flame out, for example, 2009 probably goes a lot differently, regardless of the drafting.
P.S. If you want to take a look at an interesting round for success, take a look at round seven. Since the 2000 draft and through 2010, Belichick has found hits with Pass, Givens, Banta-Cain, Cassel, Andrews (STs, until his criminality cost him his career), Edelman and Deaderick. 7 out of 21 in round 7 isn't bad at all, especially with at least 5 of the 7 (Edelman could be argued, but I'll call him ST only) being positional hits, and that doesn't count Ethan Kelly, who had a four year career with 9 starts.