Could this be partially attributed the geography, natural resources available in the region. AKA both the Spanish Empire and contemporary society saw the same poverty inducing industries available in those regions?
"""
At this point the Spanish focused on the people of the Inca Empire. [..] Citizens were divided into encomiendas, with one going to each of the conquistadors [..]. The encomienda was the main institution used for the control and organization of labor in the early colonial period, but it soon faced a vigorous contender.
In 1545 a local named Diego Gualpa was searching for an indigenous shrine high in the Andes in what is today Bolivia. He was thrown to the ground by a sudden gust of wind and in front of him appeared a cache of silver ore. This was part of a vast mountain of silver, which the Spanish baptized El Cerro Rico, “The Rich Hill.” Around it grew the city of Potosí, which at its height in 1650 had a population of 160,000 people, larger than Lisbon or Venice in this period.
To exploit the silver, the Spanish needed miners [..]. They sent a new viceroy [..] Francisco de Toledo, whose main mission was to solve the labor problem. [..] To find the labor he needed, de Toledo first moved almost the entire indigenous population, concentrating them in new towns called reducciones — literally “reductions” — which would facilitate the exploitation of labor by the Spanish Crown. Then he revived and adapted an Inca labor institution known as the mita, which, in the Incas’ language, Quechua, means “a turn.”
Under their mita system, the Incas had used forced labor to run plantations designed to provide food for temples, the aristocracy, and the army. [..] In de Toledo’s hands the mita, especially the Potosí mita, was to become the largest and most onerous scheme of labor exploitation in the Spanish colonial period. De Toledo defined a huge catchment area, running from the middle of modern-day Peru and encompassing most of modern Bolivia. [..] The Potosí mita endured throughout the entire colonial period and was abolished only in 1825. Map 1 shows the catchment area of the mita superimposed on the extent of the Inca empire at the time of the Spanish conquest. It illustrates the extent to which the mita overlapped with the heartland of the empire, encompassing the capital Cusco.
Remarkably, you still see the legacy of the mita in Peru today. Take the differences between the provinces of Calca and nearby Acomayo. There appears to be few differences among these provinces. Both are high in the mountains, and each is inhabited by the Quechua-speaking descendants of the Incas. Yet Acomayo is much poorer, with its inhabitants consuming about one-third less than those in Calca. [..] In Calca and Acomayo, people grow the same crops, but in Calca they sell them on the market for money. In Acomayo they grow food for their own subsistence. These inequalities, apparent to the eye and to the people who live there, can be understood in terms of the institutional differences between these departments — institutional differences with historical roots going back to de Toledo and his plan for effective exploitation of indigenous labor. The major historical difference between Acomayo and Calca is that Acomayo was in the catchment area of the Potosí mita. Calca was not.
[..] Throughout the Spanish colonial world in the Americas, similar institutions and social structures emerged. After an initial phase of looting, and gold and silver lust, the Spanish created a web of institutions designed to exploit the indigenous peoples. [..] Though these institutions generated a lot of wealth for the Spanish Crown and made the conquistadors and their descendants very rich, they also turned Latin America into the most unequal continent in the world and sapped much of its economic potential.