I think you are misrepresenting GP & parent's comments.
Yes, absolutely, totally, Brits have a well-deserved reputation of colonisation.
But as a hypothetical conundrum, who would you return the relics from a long expired society to -- the current (arguably quite distinct, religiously & culturally) administrations of those lands?
What moral right is exercised (or exercisable) of relics of, say, Atenism, crafted 3 to 4 thousand years ago -- locals with an orthogonal religion & culture, or foreigners with an orthogonal religion and culture?
(Personally I instinctively lean towards your take, albeit a little less abruptly - but I think it's all quite complicated - partly with the bizarre 'cultural birthright' thing, partly curator cred, less so the accessibility claims.)