I believe you’re correct.
However because HAVING takes place after all the results are fetched and processed (so it can do GROUP BYs) HAVING can’t use indexes.
So while the two will give you identical results if you’re not using a GROUP BY, the HAVING version could be thousands of times slower. Or worse.