Of course it is not insurmountable problem, but for lot of other units milli- tends to represent the lower end of "human scale"; millimeters, milligrams, milliliters, milliwatts, etc are as small as most people will ever use in day-to-day life.
In some ways it would be more convenient if the base unit would be e.g. 1e-4 days (or some other power of 10) so day would be 10 kilos. Understandably it would be lot less elegant but more practical. Deci- would be .86 seconds which is kinda nice.