Even with that setup I have unfortunately had a bad experience just using Qwen2.5-27B. I asked it once to take a large PDF of a book and find and quote all instances which mentioned food. After churning for a long time it eventually gave me several interesting excerpts, only one of which was real and the rest were hallucinations/confabulations.

I hope we can get to the point where even a small distilled model at the 7B-30B level avoids hallucinating.