I don’t think anyone "deserve" to inherit. It’s just money you never worked to get and that you get because your parents had extra money.
So, morally, why would the brothers deserve more the money than the ex girlfriend ? Maybe they were still in good terms. Maybe she brought more joy in his life than his brothers. Maybe not but enough for her to get her share. Or just maybe he didn’t care enough about this and should be considered ok with the outcome.
You don’t deserve 1 more million on your bank account because you happened the brother of someone. Yes it may happen eventually but that’s just luck.
And I’m not taking side for anyone here. I can totally understand why the brothers would be upset and I would probably be if I were in their shoes. But at the same time, as someone who doesn’t count on his brother’s death to become rich, I’ll have a hard time being empathetic.
They should at least be upset about the carelessness of your brother rather than the ex girlfriend.
tl:dr; You are allowed to think that doing the right thing is not to give the money back because it was never meant to be theirs.