IMO a reason is because nobody seems to bat an eye when faced with a "TUI" made out of lines, boxes and garish colors but if a GUI is made like that it is suddenly ugly so toolkit developers put a ton of effort into shiny looks.
But try a GUI made with Win32, Motif, Tk (not Ttk) or even Gtk1 and it'll be as responsive as any terminal application (of course it also depends on the application, there isn't much the GUI can do if the application abuses it or is sluggish for other reasons ).