You're also underestimating how hard rockets work while still in the atmosphere. For example, the shuttles SRB work entirely within the troposphere and stratosphere. They're about a million pounds of propellant each, and together they make up 70% of the shuttles lift off weight. If you eliminated the need for the oxidizer in the SRBs, you'd save nearly half the entire weight of the shuttle. Because of the non-linearity of the rocket equation, saving weight produces compounding advantage, so this would be huge.
The key goal of an orbital launch vehicle is generating the necessary speed for orbit (over 8,000 m/s, around mach 25). The difficulty of reaching the altitude of low Earth orbit is inconsequential in comparison. A rocket has the advantage that it can do its accelerating wherever it's more convenient, so the typical flight profile is first up and then over, because it's a hell of a lot easier to accelerate and travel at high speeds above most of the atmosphere. For example, the Falcon 9 reaches an altitude of 5km before it even goes supersonic, and will reach an altitude of 30km within the first 2 minutes of launch.
An airbreathing engine however needs to stick around in dense enough atmosphere for its engines to work. And if a vehicle relies on a significant amount of airbreathing then it needs to spend a significant amount of time in that denser atmosphere. And that means that it needs to do more of its accelerating in denser air, which means that it will encounter higher aerodynamic forces, higher drag, more heat issues, a higher max-Q, etc. Those sorts of forces tend to be the "long poles" that aerospace vehicles are designed around, it dictates everything from the materials used to the type of construction to the service life of the vehicle's frame, etc. This is something that positively cannot be avoided for an airbreathing vehicle.
Sure, the SRBs generate a ton of thrust on the Shuttle, but they also help push the Shuttle quickly to higher altitudes and lower air pressure. Before the Shuttle hits mach 2.5 (of 25) it is already at an altitude where atmospheric pressure is 1% of sea level.
As I said before, mass isn't the big driver of cost in orbital launch vehicles, cost comes from complexity which comes from operational complexity (flight profile, staging, etc.) and design complexity (engines, control systems, handling, etc.) A vehicle which saves fuel but increases operational complexity is not a cheap vehicle. Fuel costs around $1,000 a tonne, whereas an engine can easily cost $10,000 / kg.
The biggest win that a vehicle like Skylon would have initially is that it might make it easier to make reusable launchers. If that's the case then even an expensive launcher which can be reused only a handful of times might still be useful in reducing overall launch costs. But if an entirely rocket based vehicle can be made to be reusable then it's very unlikely to have better overall economics or operating characteristics, for all of the reasons I've listed previously.
