I feel the opposite way about the English reading "5 times 3". In English, the subject comes first, so I would expect the sentence to mean take 5, use "times" as a verb, and 3 as the adverb. Likewise, if you read it as "5 multiplied by 3", you would expect to take five, three times.
Exactly. And this is why we use math operators rather than English terms for math operations. If you want to specifically mean 5 times a group of 3, define a new operator for it, like 5 ○ 3. Don't 'overload' the x operator with your own arbitrary meaning.