This is probably a side effect of the CBA, limiting practice time. As the season progresses the defense should noticeably improve. The injuries have exacerbated that situation.
In 2016 that occurred in week 3. In the first two games the Patriots allowed 45 points, 44 first downs and 801 total yards in their first two games, In the next game the Pats D pitched a shutout.
The Patriots began 2015 by allowing 26.5 points per game, 408 yards per game, and 23 first down per game. They followed that up by allowing only 23 points over the next two games.
In 2014 the Pats gave up 33 points and 25 points in week one. Soon after that allowed 41 points, 443 yards and 26 first downs. Many infamously wrote them off at that point.
Certainly the defense is a concern as of this moment, it's not something you can completely right off. Don't forget the Patriots were without three starters on defense today. Hopefully the injuries will not continue to pile up. Let's see how they look next week. It's still way too early to declare how the defense will perform for the remainder of the season.