You must clone the slice on both write and read, right?
I get that cloning incurs a memory allocation and a copy operation, but this is the price for safety when concurrent access is possible or your data may be bodified outside your structure.
You could probably intern immutable keys, or avoid storing if keys already exist and are immutable, or use an object pool (like sync.Pool) to reduce allocations if this happens at scale. Anything else I am missing?