The US launched the first violation of visa free travel agreements, which the EU had to accept but could not have done first.

Given a world where they would have had to back down on US/Canada, it would have been a diplomatic disaster to go through and try to justify why they are not going to back down with the other countries.