This is iOS. I haven't been able to find the ability to lower the minimum volume, and based on posts in the support forums, this seems like a common complaint that has yet to be addressed by Apple. That said, if you happen to know something that I don't, I'd be delighted to hear it.
Settings, Sounds, Headphone safety, reduce loud sounds, adjust the slider accordingly. I had to turn this on because my IEMs were way too loud on the lowest setting.
I had that on the lowest setting already, it doesn't make a difference. I suspect that iOS already thinks its playing music quite softly and therefore won't know to down correct.