I hear you. My $0.25 ... I bet the expert authors of many music theory books would be capable of thinking in mathematical terms but (now I'm guessing) there is likely some undeserved "ew, yuck, math!" culture in the arts so rather than turn off their audience, they avoid talking about the quantitative underpinnings of why things sound the way they do.
Two more enjoyable books on the math and physics of music (though, again, probably not far enough up the tree of abstraction for chords):
"The Science of Musical Sound" by Pierce (lovely little book, not too deep though, quite coffee tableable)
"Fundamentals of Musical Acoustics" by Benade (old book, considered a classic, reads like a science text)
Good luck!