Maybe I'm expecting too much of HN, but I've seen these same two top level comments under myriad ML posts.
Sorry for the meta-discussion that's gotten us further away from this really remarkable paper.
It's completely speculative. There is no evidence at all that Spiking NNs really work better is any circumstances.
Speaking as someone who has worked in the ML field, it feels to me like advocates for them are caught up in the biological plausibility argument. That's an interesting branch of research, but has very little to do with how AI should be implemented using transistors. In some ways the "neural networks" name has done a great disservice because people keep getting caught in the trap of comparing them to how the human brain works.
Transformers have a sequence context, but it constructs its own context dependent notion of orderliness with attention.
Persistent or recurrent activation states can extend the context window past the current tokenizing limitations. Better still would be dynamic construction where new knowledge can be carefully grafted into a network without training, and updates over the recurrent states feeding back into modifying learned structures.
Spiking networks might provide a clear architecture to achieve some of those goals, but it's really just recurrence shuffled around different stages of processing.
