If you say "Alexa, stop by the way", it'll get the device to stop responding with follow-ups for ~24h. I ended up creating a routine that runs every night at 4am to lower the volume to zero, automatically say "stop by the way" to the device, and then raise the volume a minute later, and Alexa has stopped with the follow ups