Trade was heavily restricted, governed, and controlled in the ancient world. Empires were built on collecting transit fees, cities often restricted the entry of sailors aboard merchant ships, and control of rivers and who could use them were dearly held by rulers the world over. There are a number of cities around the world today that started as trading cities, specifically designated places where merchants were allowed to visit to trade but restricted from going anywhere else.
The Roman Empire was a unique example among the thousands of years of history because along with their transportation network of roads they did actually have a couple of passport-like systems in place. Now a third of the people were literally slaves, so they definitely weren't going anywhere without permission, so maybe Rome is not a great starting point for what I presume to be the argument against passports. But the Romans and other Mediterranean civilizations did check your identity when you travelled. Romans had documents they called diplomas for people on official business and several civilizations used clay tablets called tesserae as a sort of ID card, but by far the most common method was simply having known people vouch for you.
Basically nowhere in the ancient world from the Middle East to the pre-colonial Americas could you just show up in a foreign village without people asking questions that you ought better have good answers for. There are a handful of exceptions, but this has never been the norm, and even with passports and visa restrictions it is definitely far easier today than it ever was in the past. I expect it will get easier and easier into the future. Perhaps quite soon in fact as the global population begins to decline and people become more and more valuable.
