Indeed. Staff at the hosting service I use for professional purposes have always been very down on WP. My understanding from our various conversations is that for any systems where they were providing some sort of managed security service, it was a time sink to keep everything up to date. For any shared hosting systems where they weren't also managing the security, there would be frequent compromises and then that would get things blacklisted, so potentially other customers using the same shared resources could be negatively affected in at least two different ways. They don't seem to have become noticeably more positive about any of this in recent times despite the arguments about WP security being better these days, which suggests that there is still enough of a problem to be concerned about.

Since Jetpack code is also what we run on WP.com (tens of billions of pageviews) it goes through a huge amount of performance tuning and optimization. The way modules work when you turn them off they don't have any overhead, similar to turning off a plugin. If you were using literally one thing it might feel like a lot, but as soon as you use 2-3+ things Jetpack does it's a lot more efficient than separate individual plugins to accomplish the same task.

It's not like features you don't use slow you down, or that the < 1 MB of code they take on disk is important.