One level lower, you can use Clash's signal-level simulator. Basically it gives you a synchronous stream of signal values, either as a lazy list (for "offline" simulation), or as an automaton that you can turn the crank on by feeding it the next clock cycle's inputs (for "online" simulation, i.e. where you want to do IO to compute the next input from the previous outputs). So at this level, you'd take your Pong circuit and use the automaton interface of the simulator to feed the virtual "pushbutton" states computed from e.g. keypresses, and then consume the output to do the rendering. Or simulate the whole circuit end-to-end and feed its output into a VGA interpreter, which you also get to write in Haskell.

If you need to debug at the Verilog level, you can use Clashilator (https://github.com/gergoerdi/clashilator) to automate FFI-ing into a Verilator-generated simulation.