Since Jetpack code is also what we run on WP.com (tens of billions of pageviews) it goes through a huge amount of performance tuning and optimization. The way modules work when you turn them off they don't have any overhead, similar to turning off a plugin. If you were using literally one thing it might feel like a lot, but as soon as you use 2-3+ things Jetpack does it's a lot more efficient than separate individual plugins to accomplish the same task.