That's funny, because I've found something similar to be true for me. I wonder if that's a function of some measurement of interactive visual information.
For instance, I have a green screen CRT that I can work on at any nit as long as the contrast is basically infinite (text dim or bright), but as soon as the rest of the screen becomes illuminated, it's like I'm peering through a literal fog.
Contrast that to a dimmed 4k 10-bit panel, and I find myself dropping out of focus or losing my place. I'm looking for a particular OLED panel right now on AliExpress to test with.