Yep you're right, its the framework bootstrapping that's the killer, especially when those frameworks are expensively parsing config files each time (Yaml, XML...).
Which modern frameworks never do. The parsed data is usually cached in a php file. Then the (compiled to bytecode) files are cached in opcache. The interpreter doesn't even hit the filesystem on every request.
