Even with panic selling, it will return again and push higher as it is an undervalued tangible finite asset. Also, the banks are not the largest owners of gold reserves, countries are, central banks just hold them:
Since we (US) don't use gold to sustain our currency (no gold standard), we, as a nation, are holding it for some other purpose.