The new Rack socket hijacking API (opens in new tab)

That depends on the application server's concurrency model. If you have enough threads then it won't be a problem. While threads don't have the "web scale to 100,000 users!!!" reputation, you can still get very far with them on most workloads.

On evented application servers, you can integrate with the event loop API.

And, I'm not 100% sure whether the spec allows this, but it would appear that on application servers that are not evented, you can even offload the socket to a thread running an event loop. For example Phusion Passenger's implementation allows this.

How it scales is an interesting question.

Could it, for example, replace a COMET server that is maintaining 100 connections (all doing long polling)? 1000? 10000?

Could it also be hacked into doing Websockets upgrade (and subsequently maintaining a large number of concurrent websockets conns)?

It depends on the webserver you use, not Heroku. The webserver must implement the hijacking API of Rack.

Also, you don't necessarily have to run an HTTP server on Heroku. They'll run whatever you want (as long as you change your procfile), so I think you can just go nuts. I think the only restriction is that you can't have the socket open for more than 30 seconds (though that may be 30 seconds of the socket being idle).