Depends on the package manager. Some package managers support overlays so you can keep your package modifications as a layer on top of the base package.
I know you can using Nix for sure. I think Arch/pacman (Arch Build System) and Gentoo/portage (since it is source-based) also have this concept, but I’ve never used it. No clue about other package managers.
The downside is that package updates will require a recompilation, but you can use binary caches to centralise this slightly.
