My Emacs session has been running for two weeks and I have 560 open buffers, which amounts to a memory usage of 620 MB. I don't think it's unreasonable for most every interactive operation to be instant.
I don't blame Lisp for the slowness, except insofar as the language promotes the use of data structures like simple lists that eventually create bottlenecks, and that the Elisp implementation is not very fast compared to something like SBCL.
disclosure: I'm currently writing a small node server in emacs. I'm happy with both as they are very responsive, but emacs does chug occasionally like a bro at a frat house.