Blame Alfred Nobel, he set up the original categories. According to Wikipedia his goals were prizes "which annually recognize those who 'conferred the greatest benefit to humankind'". Perhaps he didn't consider math as directly benefiting mankind.