Most GPU encoders are part ASIC, part regular GPGPU compute. That's what allowed AMD to increase the quality of their encoders via a driver update, but also why you can't add easily add entirely new codecs.
That said, outside of GPUs there are both FPGA-based and pure-ASIC hardware encoders that beat not just NVENC but even software encoding in terms of quality AND performance, such as AMD's Alveo lineup. https://www.amd.com/en/products/accelerators/alveo/ma35d.htm...
