I seriously doubt that it actually happens myself, but there are some reasons why they could actually make it work better than you might think.
It certainly seems as though Belichick has really made an effort into having Brady get the rid of the ball quickly for various benefits. This past season he led the NFL in quickest release time, with under 2.5 seconds. That would go well with a player like Amendola, who excels in space and can rack up some YACs. It would also continue to benefit the offense's approach of taking advantage of quick moves off the line, protecting Brady, and hiding some of the weaknesses of the offensive line.
The size differential isn't much at all. Amendola is only an inch shorter than Llyod and about 10 lbs lighter. He has experience in McDaniel's offense already, and also offers a great replacement to either one of Welker/Edelman (assuming they may not keep both). He returns punts, and he does it pretty well at times too.
I don't think it'll happen, but there seems to be a somewhat limited WR market if they try to replace Llyod. Do I think it would be the best match? No, but many here were fine with going out and getting Hartline to replace Llyod, so I do think it could be done better than you may initally think. One thing you need to remember is that Amendola played primarily in the slot where his YPA is going to be lower than a guy like Llyod. He also played in the horrid offensive hell which is St.Louis too, so that may come into play when assessing the stats.