The Scriptures talk of Jesus having "brothers". The word used for "brothers" was the Greek word "Adelphos". Adelphos was used to describe brothers, cousins, and even kinsman. Given that Jesus gave his Mother to John at the crucification, it seems unlikely that he would have had brothers or sisters.
That being said, the most important evidence to establish Mary's perpetual virginity would have been the Early church fathers who belived in the Perpetual virginity of Mary.
Heck even Protestant reformers like Luther, Zwingli, and Calvin (in his biblical commentary) believe that Mary was a perpetual virgin.
http://www.ewtn.com/faith/teachings/maryc2.htm
http://www.catholic.com/tracts/mary-ever-virgin