You are also aware that the EU specifically allows to limit the freedom of movement on "grounds of public policy, public security or public health"?
You are also aware that the recent migrant crisis of the past few years has caused these kinds of barriers to be raised in response? Proving that such restrictions are not just mere hypotheticals?
The European Union is, after all, not a suicide compact.