Implementations details of fizzbuzz are ugly. There are a couple different ways to implement it (not all use the modulo operator!), but not matter how you implement it there is a special case that doesn't elegantly fit your algorithm and in turn your code is ugly in ways it seems like a better design could fix.

That you cannot find a an elegant solution to what seems like it should have one is what makes it difficult. A good sign of a good programmer is then you ask them about their code after it works they will say they don't like this solution and would like to spend time making it cleaner. If you know there isn't an elegant solution of course you won't bother spending that time, but if you don't know that if seems like there should be a better answer if you can just restructure the code a little.