The point that there is no shortage of tax havens in the EU itself, including leading EU nations, is true.
Luxembourg, City of London, Liechtenstein, Monaco, Netherlands, Belgium, the Vatican, Switzerland (sort of), Malta, all for slightly different reasons. I've even read articles from these nations where they point out that the only way they'll ever distinguish themselves is with tax law. Or rather: that they'd be broke if they weren't tax havens.
And then we're remaining silent about "royal" tax havens that are individual companies which get preferential tax treatment from parliaments (e.g. Shell) and, uh, I guess "presidential" Total. These deals, for individual companies, are far more atrocious (and forced on tax departments) than any tax haven you'll find. They are directly enriching favored individuals in a particular country, and get tax advantaged for it.
And to add a more "creative" situation to terminate, Cyprus is a double tax haven (both the Greek and Turkish parts of the Island are tax havens, for different reasons, interesting for different types of individuals and/or companies, but the fact that money can cross the line, even if people cannot, offers interesting opportunities as well). Neither side of the Island will ever have anything to offer the world other than some small tourism industry, so ... for them as well being a tax haven is the only option (other than, I guess, war).
So what do you do about the problem that many of these countries have no real options other than being tax havens?