Well, the Federal Reserver produces a paper currency, the dollar, and gold has historically been a competitor of paper currencies. So it makes sense that the Fed would not want competition in currencies, and would attempt to both disparage and devalue it.
The Federal Reserve is also intimately bound up in fractionally reserved expansion of the paper currency, so it is reasonable to assume that they might use the same basic technique with gold. And that, in fact, is the historical origin of banking, in the modern, leveraged sense: gold assayers began loaning out gold on a fractionally reserved basis, allowing them to collect interest beyond their core assets. Worked great until people ran on them.
So I assert that it is less irrational to believe that there is a possibility that central banks engage in manipulation of gold prices than it is to believe that there is a dinosaur living in the subway under manhattan.
YMMV.