Over the countless years, such a large amount of wealth had accumulated inside the Kangra fort treasuries that such an amount could not be carried on the backs of the camels or in vessels, and neither could the mathematicians calculate it nor could the writers record it. It has also been said that this fort has more amount of jewelry and a greater quantity of gold, silver and precious stones than even the Royal Treasury of India at any point in time. Quite predictably, the fort has been subjected to countless attacks over the course of history and countless rulers have tried to bring the fort under their control as it has been said, “He who holds the Kangra Fort, holds the hills''. The Maharaja Of Kashmir, Raja Shreshtha made the first attack on this fort in 470 AD. The first records of foreign invasion on Kangra Fort date back to 1009 AD, at the time when Mahmud of Ghazni invaded India. The immense amount of treasures inside the Kangra Fort captivated Mahmud Ghazni. The Turqik Sultan of Delhi, Muhammad Bin Tughlaq was the next ruler to come invade and capture the fort in the year 1337 AD.

After his death in 1351 AD, his successor, Feroze Shah Tughlaq conquered Nagarkot in the same year. Another interesting fact to note is that Mahmud was the first enemy to ever step foot inside the fort walls. His attack on the city of Kangra was so swift that the Garrison commanding the fort and town could not even close the walls of the city as the Katoch armies had been greatly reduced in number that day because the Raja of Kangra, Jagdish Chandra was out on a campaign against the Raja of Kaluta which is present day Kullu on the same day of the invasion. When the sultan gained access to the town, the sultan blackmailed the garrison commander with taking the lives of all the people and citizens of the town, and under these circumstances he was let into the Kangra fort, and after taking control of the fort, he killed all of its occupants.

The rulers of the provinces of Juzjani, Altuntash, and Asightigin, which were the sultan's chief chamberlains, were appointed in charge of the gold and the silver. The sultan himself took the sole charge of all of the jewels. The immense Kangra Fort is said to have 21 treasure wells, and each well was 4 meters deep and 2.5 meters wide. The Sultan of Ghazni looted eight wells, the British in the 1890’s found five more wells and it is very well believed by the locals that the fort still has five treasure wells hidden deep inside the fort somewhere. A British commanding Garrison held the fort until the massive earthquake of 7.8 magnitude on the Richter scale that killed almost 20,000 people in 1905. The earthquake destroyed the roots of the Kangra Fort immensely. What no ruler, invader or army could do, was done by nature itself. Most buildings in the towns of Kangra, Mcleodganj and Dharamshala were also destroyed by this earthquake.