It's not any more of a black art than programming is. There's no magic to it, you just have to know what you're doing. Also, you should use some kind of scheme (like bayesian optimization) to optimize the hyperparameters of many experimental models. That's the best way to make sure you're getting the best results (but of course takes a while, so you do have to be selective about which parameters you optimize or what combinations you allow).

And I'll also aknowledge that "programming" is a vastly more mature field than "deep learning" or even "machine learning". So it's fair to argue that there's much we don't know, but there's more and more we do.