You are right that attention is a complex phenomenon, but keep in mind that people who evaluate ADHD are aware of the limitations -- that is why TOVA isn't typically the only mechanism used for diagnosis. ADHD evaluation often involves multiple sessions in which other possible explanations are ruled out or explored, along with questionnaires from friends and family who know the person -- incidentally some of the questions on the ones I collected from my friends/family were about my driving!
Even with this, there are still limitations, but there's no need to throw the baby out with the bathwater -- the goal is to produce an actionable explanation that we can use to reduce suffering, and for many people this explanation does exactly that.