I would guess that a lot of the slowness is not from PHP itself, but the way your specific projects are using it. With the vanilla PHP project, the slow down may be caused by poor design decisions from other developers.

With Drupal, the code quality of the core is generally okay, however the added inertia may come from a "square peg, round hole" situation, where the CMS is being used for an obscure purpose for which it wasn't really designed. This is quite common in the wild, probably due to the low barrier to entry and the fact that people will use what they are comfortable with to solve every problem under the sun (not dissimilar to the prevalence of Excel throughout the world of finance for a variety of completely inappropriate use cases).