For the record, my native tongue is much more gendered than English (it has gendered nouns, verbs and adjectives; not just pronouns) - I don't understand how it's useful at all, I don't miss it in English.
Personally I agree, for me gender holds little importance, if I'm being most true to myself I identify as non binary simply because I don't really identify with a gendered label. That motivation has also lead me to being ok with being gendered male, because it just doesn't matter to me.
I understand I need to look outside of my own experience to see the importance people place on gender though. You can say all you want that most people don't care, but I feel if you misgender people, a lot of them would be very upset. Trans people are very vulnerable to suicide because of this, to diminish the importance of gender (this includes the binary, of which many trans people want to fit into) to these people is to be at best lacking in empathy.
Now if you're approaching this from a gender abolitionist angle where you believe all this attachment to gender is socialized and that we should push to de-emphasize genders role in society, then I believe that's a far more defensible position, but I feel you need to at least recognize the importance gender has to people today (socialized or not) if you're to have any hope in bridging that gap with people.
So I'm only a grammatical gender abolitionist :) I don't see the point of gendering people when I talk about them unless I talk specifically about their gender. As a happy side effect, this would also massively reduce the risk of accidentally misgendering someone.