I actually do entirely ignore it because I think it's purely a creation of the author for the purpose of making his point. Caste society is not a symptom of modernity, in fact the opposite. It's an anachronism in the book, sort of like putting your ideological opponent into a Nazi uniform and giving him a tiny mustache and making him speak with a scary German accent.

Slavery (which is really what it is in the story) you are more likely to enounter in the Antebellum South than in a modern metropolis and high modernity. Limiting the intellectual capacity of your subjects would never be the point of the kind of society that Huxley envisions, and it's one example of how he does not take the thing he aims to critique seriously.

A society so advanced it could realize Huxley's fears is not at the same time going to build itself a medieval army of slaves. Even at the time of Huxley's writing Taylorism was already well past its peak and criticism of mechanized production was everywhere.