It should be clear that brute-forcing a passphrase is much more complicated than a PIN number, in the best case for you (worse case for an attacker).
Just because your passphrase may essentially be a PIN look up into a lookup table doesn't mean the attacker knows that or has access to the same lookup table.
You can also add additional entropy via punctuation or casing.
The point to a random passphrase is to try to avoid "human" mistakes like over favoring a subset of words, and rerolling words you don't like potentially makes your collection of passphrases more susceptible to analysis or social engineering (word association) attacks.
Like I said, it's generally better to pick a word-list you are comfortable with all the possible words than to subset a word-list you aren't entirely comfortable with.
The goal of something like Diceware is to be easy for humans to memorize but also still true random (see: xkcd's battery horse comic). If you don't need to memorize it, then yes, why not entirely generate a random sequence of letters/numbers/symbols/emoji.
