undefined | Better HN

0 pointssieve1y ago0 comments

ROCm. About 40%. But there is duplication there as well. Two 16GB folders containing the exact same version.

0 comments

Run rmlint on it, it will replace duplicate files with reflinks (if your fs supports them — xfs and btrfs do), or hardlinks if not.

sieveOP1y ago

Thanks! Hearing about this for the first time. Never felt the need before.

forrestthewoods1y ago

Does uv have any plans for symlink/hardlink deduplication?

sieveOP1y ago

Not sure. The simplest solution is to store all files under a hashed name and sym/hardlink on a case to case basis. But some applications tend to behave weirdly with such files. Windows has its own implementation of symlinks and hardlinks. They simply call it something else. Perhaps portability could be an issue.

mixmastamyk1y ago

The article says it already hard links duplicates. But likely not able to help if you are using multiple versions of interpreter and lib.

IgorPartola1y ago

Sounds like a great use case for ZFS’s deduplication at block level.

sieveOP1y ago

I use ZFS everywhere EXCEPT on this drive. Not willing to have ZFS on the primary drive till native support lands in the kernel (so, never).

hyperbrainer1y ago

Have you tried borg [0]? Also, why not BTRFS?

[0] https://borgbackup.readthedocs.io/en/stable/index.html

1 more reply

22c1y ago

or, you know.. symlinks

flakes1y ago

Main issue with symlink is needing to choose the source of truth— one needs to be the real file, and the other point to it. You also need to make sure they have the same lifetimes to prevent dangling links.

Hardlink is somewhat better because both point to the same inode, but will also not work if the file needs different permissions or needs to be independently mutable from different locations.

Reflink hits the sweetspot where it can have different permissions, updates trigger CoW preventing confusing mutations, and all while still reducing total disk usage.

1 more reply

j / k navigate · click thread line to collapse

0 comments

homebrewer1y ago

Run rmlint on it, it will replace duplicate files with reflinks (if your fs supports them — xfs and btrfs do), or hardlinks if not.

sieveOP1y ago

Thanks! Hearing about this for the first time. Never felt the need before.

forrestthewoods1y ago

Does uv have any plans for symlink/hardlink deduplication?

sieveOP1y ago

mixmastamyk1y ago

The article says it already hard links duplicates. But likely not able to help if you are using multiple versions of interpreter and lib.

IgorPartola1y ago

Sounds like a great use case for ZFS’s deduplication at block level.

sieveOP1y ago

I use ZFS everywhere EXCEPT on this drive. Not willing to have ZFS on the primary drive till native support lands in the kernel (so, never).

hyperbrainer1y ago

Have you tried borg [0]? Also, why not BTRFS?

[0] https://borgbackup.readthedocs.io/en/stable/index.html

1 more reply

22c1y ago

or, you know.. symlinks

flakes1y ago

Hardlink is somewhat better because both point to the same inode, but will also not work if the file needs different permissions or needs to be independently mutable from different locations.

Reflink hits the sweetspot where it can have different permissions, updates trigger CoW preventing confusing mutations, and all while still reducing total disk usage.

1 more reply

j / k navigate · click thread line to collapse