The central question here is "what is money". The general argument you are laying out is not some "crackpot theory", it is one alternative of how you can conceive of money. And in fact that conception of money is the one which has been dominant throughout nearly all of history. If you think of coins in history, they were valuable because they contained valuable materials, they had inherent value. This concept of a "gold coin", which is valuable as a piece of gold, translated into paper money as a piece of paper representing the inherent value of that piece of gold.

This is quite different to the modern conception of money which does not represent anything besides itself. The reason a banknote is valuable is not because it represents something actually valuable, but because the government enforces it as a means of exchange and you are thus able to use it to purchase goods and services.

None of these theories are "wrong" or "absurd", they are both valid ways to look at money. The reason people care is because in one scenario the government clearly has more control over it, if you imagine yourself as the central bank of the roman replublic, the only way you can create new gold coins is by actually acquiring gold to make them. You can inflate the currency by reducing the amount of gold in a coin, but this obviously does not affect the gold value in already existing coins. With modern currencies things look very different. Creating new money is trivial, the government can create it from thin air. And thus the government has to be far more responsible with its greater power.

The whole thing gets infinitely more complex, but at the core is a debate about the question of what money actually is.