Note: IANAL
Absent court proceedings the NFL has no right to the phone.
However, once litigation commences the phone may well be subject to discovery.
If Brady waits too close (and no, I don't know what "too close" is) to the start of litigation to destroy the phone then (again, IANAL!!) once litigation does start and it comes out that the phone has been destroyed, the court can penalize him for spoliation of evidence.
However (and again, IANAL!!), if the phone is destroyed "far enough" before things start (I don't claim to know what the boundary of "far enough" is, either), then spoliation is off the board.