Well it is racial. What is really sad that you want to still be one of the good guys while supporting a clearly immoral thing. It would be more respectable, and less pathetic, to just accept your pride in the actions of the colonizers, no matter what the consequences of those actions were for the colonized.
And what property rights are you even referring to? The relic in question is a physical item that can be moved without affecting those that continue to possess the stolen item in any way. It's certainly not something those who possess the item have earned. It's not even land that you have long settled, which may be much harder to part with.
Continuing to hold on to it just says, "Yeah our ancestors were a bunch of bandits. We don't apologize for it". Why would anyone move on when one of the parties holds such an attitude? Let alone the aggrieved party.