>Isn't their primary concern latency?

It is, but at some point you can't do much more about the network latency; shaving off a couple dozen us, might the difference of getting the price/quote or missing it out. Especially, when you play versus similarly equipped firms.

Depends on how low the latency already is. If you're already <0.5ms away network wise then perhaps shearing off 0.1ms from a 1ms operation could make a significant difference.