It probably is 5.5GB. UV caches versions of packages and symlinks to them but pytorch is infamous for many different versions especially for different features.
And since its compiled dependency even if UV was to attempt the the more complicated method of symlinking to use a single version of identical files it wouldn't help much. You'd probably need to store binary diffs or chunks of files that are binary identical, at that point your code would probably start to resemble a file system in user space and time to switch to a particular version of the files (ie create thek as qctual files in filesystem) would be much higher.
Also I believe uvs cache is separate from the pip cache so you could have different copies in both.
I think there's a uv cache prune command. Arguably it should offer to install a from job to do it periodically
