You can ignore it for now but how much will they push it over time to make it harder and harder to ignore? Will grandma accidentally buy tokens due to some dark pattern UI and then call you about it? Will you get constant messages from scammers trying to get you to send them tokens?
This might not happen but I feel it is a valid concern for a platform you wish to stay on longer term.
edit: And since the CEO of the token company has said he wishes to make future monetary donations to Signal they have an incentive for making the token do well.
It's not just the fact that it exists that leaves the bad taste, it's the fact the decision was made, what that implies about the mentality/goals of the people making such a decision and that it will from that point on hog resources. And those aspects won't go away by not using it.
Not to mention that with the number of scams with cryptocurrencies It's not inconceivable that either Apple or Google will eventually ban them from App store (apps using cryptos)
But it seems like a waste of resources by Signal, they could be making the chat part of the app better instead.