I get the same feeling of momentary compulsion to exactly answer a greeting when confronted with two very common greetings in Thai: "Have you eaten rice yet?" and "Where are you going?"

To this day, I am still not entirely confident in how I should respond with to the prior.

In regards to the English "how are you?" greeting, think of it as a formulaic inquiry meant only to ensure there isn't anything absolutely horrible going on with you at that exact moment - anything lesser equates to "fine" or "I'm well, thanks."

More broadly, it is an appropriate situation to note anything out of the ordinary going on - "oh wow, I am so tired this afternoon!" etc to scope the ensuing situation. The greeting really is a question that equates to being asked "is there anything particularly out of the ordinary going with you right now I should know about that would impact us talking?" If there isn't, "I'm well, thanks."

(I know you know all this, but your comment got me musing on the topic.)

I (a native english speaker) frequently ask this question at the start of conversations. and I am actually seeking information- it is not an empty ritual.

The following are in the context of a workplace conversation; other types of conversations may vary slightly depending on the scenario.

Specifically, I am trying to understand how to set the tone for a conversation. If you are feeling stressed, busy, exhausted or frustrated, I may keep banter to a minimum, decide to ask for a meeting later instead of engaging a full conversation now, or even decide that whatever caused me to get your attention in the first place is less important and offer to help you instead of asking for your help.

If you are feeling bored, content or happy, I might ask for more direct help than limiting the conversation to simply getting an answer to a question.

No matter what the answer is, I'm also trying to use showing an interest in you to set the tone of the conversation to one of camaraderie and collaboration, rather than direction, accusation or competitiveness.

There's a ton of nuance involved, no set rules, and the actual amount of time I am expecting to spend on the topic correlates pretty strongly to how well I know you. I'm not asking to be your therapist or your friend, but I am hoping for an honest answer, and if anyone expresses that they are struggling, I will offer to help however I can.

How is it different from 'Wie gehst?', which is "How's it going?" which has the same rules as 'How are you?'

As a native speaker of English, when someone asks me that I do try to express how I am, but I know my response is being interpreted relative to a baseline of how people answer this question. If I'm having a rough day I will answer differently from how I would on a good day. But while doing this I'm balancing all sorts of other considerations: how much time I can spend away from other topics, whether I want a closer or more formal relationship with the other person, what they want, etc. I modulate my response relative to the communicative baseline, so my interlocutor, if they're fluent, can infer that I'm doing well or poorly, I'm in haste or willing to take some time, that I put great stock in the my relationship to them or little.

I think this implicit communicative baseline is a huge, invisible barrier to communication among people who are apparently fluent in a common language. You can still understand the question as serious and answer it honestly conveying how you are to the speaker, and have it come off as fake or formulaic to people not aware of the baseline. I'm sure this is true for all languages. The problem, of course, is acquiring a knowledge of these baselines and the context in which they apply is extremely difficult, often even for native speakers. The native speakers find it difficult to introspect about this and explain why they interpret things as they do. Because it is invisible to them, it is difficult for them to teach this to someone else. And it is difficult for them to realize someone else is not doing this and therefore not be offended by non-natives, or people with ASD or whatever, not communicating relative to this baseline. A Dutch person saying something bluntly isn't "just being honest". They are just comporting themselves relative to the Dutch baseline. A Japanese person using non-confrontational polite formulations isn't being dishonest. They assume you are familiar with the Japanese baseline (while not necessarily even being aware this is their assumption).

I think it is common for people to believe people from their native culture come in all sorts but people from other cultures all have personalities in a tight range. They're all lazy or wily or emotionless or angry or cold. I think what they're perceiving is the way one communicative baseline deviates from another. They take this difference as a deliberate, communicative modulation away from their baseline, the honest, neutral one. To them, the other person's neutral state is not neutral. They always speak as they they're angry, say, or in a hurry, or trying to deceive. It is analogous to the way people perceive themselves as having a neutral, invisible accent and all other people speak in some quirky way.