They wont look at you because its not their primary way of “seeing” or spatializing the world or their etiquette

Their ears might briefly move in your direction, their tail might flick in one of their frustrating ways, they might not move a muscle (which is a conscious response for them as well)