Unfortunately, simplistic explanations that are understandable to a child don't always model reality. If it were that easy, we'd have solved energy security and climate change a very long time ago.
The (real) arguments about transitioning gradually have very little to do with past investments in fossil fuel production and much more to do with whether we yet have the ability to manufacture and deploy better alternatives efficiently at scale.
Once upon a time, those PV panels you mentioned cost more in resources to develop, manufacture and install than they were likely to save over their working lifetime. Of course, the technology has come on a long way since then, but if we'd all gone out and stuck panels all over our roofs in those early days, it would have been counterproductive.
Today we face some similar questions with the lithium batteries currently used in popular EVs. It's all very well looking at Tesla aiming to manufacture a few hundred thousand vehicles per year, but total global car supplies (and remember, cars aren't the only vehicles on the road) are approaching one hundred million per year. Even assuming we could magic up 100 Gigafactories to produce that many batteries -- and keep in mind that Tesla's first Gigafactory is on a five-year build programme that hasn't finished yet -- we'd need about a million tonnes of lithium per year to sustain production using the technology and processes we have today. That's an order of magnitude more than the current annual global supply, and at that burn rate we'd deplete the entire known global reserves in 1-2 decades. There are analogous concerns over the availability of other essential elements for the production of current technology EV batteries, such as cobalt.
So if you think we are ready and anyone who says otherwise is some sort of apologist, I invite you to explain how we're going to scale up battery production, gather enough raw materials to go into those batteries, and presumably develop a suitable recycling programme, so everyone can be switching to EVs within the kinds of timescales that environmental lobby groups are currently calling for. Otherwise, you're really just arguing for changing our infrastructure to use a different set of natural resources that still has a finite supply that we'd be in danger of exhausting.