Is running a virtual machine per each connected device really necessary? Why do you do it?
I mean, I'm working at a IoT cloud too, but about 7 years ago we managed to connect 250.000 IoT devices to one erlang instance running on 8 core machine with 24gb ram as a stress test with simulated devices (rest of system was rabbitmq to share measurements from devices and riak for storing/retrieving them, plus websocket api endpoint). Those were gps trackers and some hubs. Surely you don't need 192 core server to connect 5000 devices?
You can absolutely simulate more IoT-devices. But in this case I don't have to simulate receiving OTA updates, rebooting, keeping state. It has some upside but it isn't the reason I've done it.
I think I mentioned that in both this and the previous piece that I started out doing the VMs for fun and that's still kind of the central point. "What can I do with 192 cores?" We'll have a good use of the artifacts of this work for testing the framework itself in ways where devices are useful.
As you say, if I wanted to stress-test the OTA service for real I could definitely just set up some libraries to make multiple connections instead of starting the whole machinery. You can make a lot more connections, throw a lot more work at the system. And depending on how fancy you want to get you can simulate a lot of device behavior.
Given that I happened to have a couple of thousand full-fidelity devices in my pocket we got some incidental testing of the service.
