You could do that, sure. Whenever I want to schedule a meeting with someone in Zurich, I could reason “15:00 UTC is about an hour before solar noon for me and — looking it up — about four hours past for him. Reasonable time for a meeting.” This would prove so useful in practice that people would rapidly start referring to times that way when talking to people from abroad: “sorry I couldn’t take your call, it was seven hours past solar noon for me; I was having dinner.” Everyone would memorize their location’s offset from UTC and use it colloquially all the time.

Oh wait, we’ve reinvented time zones!

Basically, the point of time measured in hours and minutes is to be useful to humans in daily life. And it’s useful to have a common language for things relevant to humans.