Thank you, Wikipedia. (see previous post)

The use of the swastika was associated by Nazi theorists with their conjecture of Aryan cultural descent of the German people. Following the Nordicist version of the Aryan invasion theory, the Nazis claimed that the early Aryans of India, from whose Vedic tradition the swastika sprang, were the prototypical white invaders. It was also widely believed that the Indian caste system had originated as a means to avoid racial mixing. The concept of Racial purity was an ideology central to Nazism though it is now considered unscientific. For Rosenberg, the Aryans of India were both a model to be imitated and a warning of the dangers of the spiritual and racial "confusion" that, he believed, arose from the close proximity of races.

Thus, they saw fit to co-opt the sign as a symbol of the Aryan master race.