While I agree that public control of public buildings is relatively vague in modern times, it still exists to some extent. If a mayor wanted to tear down a beloved building and replaced with an ugly one (as judged by the esthetics of the public in the town), they would face significant backlash and may lose a future election based on that: people in certain places care a lot about the look of their town (and in others, only vaguely).

Even beyond electoral politics, many cities have public NGOs and other organizations that seek to shape this sort of thing from an early stage through various legal means (and sometimes even through civil disobedience, like tying themselves to a building to protect it). If they are broadly in line with the tastes of the people, they tend to thrive; if they are not, they will often die out.

And yes, in certain cities and towns, people actually like the way grafitti looks and are bothered when someone goes and whitewashes a beautifully painted wall. That's perfectly fine, and is a part of the culture and esthetics of that place (and here, destroying the art that people enjoy is an act of violence against the public and/or the artist). But it's also perfectly fine for other places to want neat walls with clean textures, and marring their beautiful walls with grafitti would be an act that goes against the public.