>I as a client care about 400s because that means I fucked up.
Not necessarily. If the user enters in the wrong CC number and gets a 402 or 422 back then you don't really care.
>As a server I don't care about 400s because (within constraints) I don't care if my clients have fucked up.
Not necessarily. 404s can be caused by your application routing or a bad link you're generating.
>Ad a server I care about 500s as that means I've fucked up.
Not necessarily as 501 and 505 can be expected behavior
>As a client I care about 500s only so much as to know if I should give the server a break and try somewhere else.
This one is pretty safe