Skin color, wealth, relatives, philosophies, habits, and infinitely more characteristics are used to determine who will and will not benefit you or not benefit you and it can change with every scenario.
I agree with comparing caste-ism to racism, and very well may be the same. It is very similar in that they classify someone as part of a tribe because of their ancestors. But it is also quite similar to classifying someone based on their accent, which side of town they grew up in, or what university they attended, because they all end up being a function of who one’s ancestors are also.
This is fair. The salient difference for me emerges in immigration. The fact that someone can look at an Indian American speaking American English, dressed in Western garb and educated in America and determine something about their parents' caste puts caste closer to race than e.g. wealth, religion or sexual orientation.
When I try to get to the root cause of why people want to categorize others, I often find it is about utility. Surely, it can be evil too such as someone wanting to punish others or getting a feeling of superiority from viewing others as being below. But it can also be about going with who you think will have your back when the situation gets hairy (which can include covering up your misdeeds), or simply someone who will help you.
In that sense, an Indian American speaking American English, dressed in Western garb and educated in America who is likelier to have many cousins and family who are engineers/doctors/lawyers/business owners could be more useful than a visually identical Indian American who is not as likely to have that family background.
The compounding effects of networking really hamper efforts to increase equality of opportunity.