Using the gram would not have removed the prefixes from all commonly used units.
In the beginning, the liter was a much more frequently used unit of volume than the cubic meter.
A liter was defined as the volume of a kilogram of water. In a system were the gram was the unit of mass, the corresponding unit of volume was the milliliter.
Which of the gram and the kilogram or which of the centimeter and meter were chosen as the units of mass and length did not matter much for mechanical units, in the way they were used in practice in the 19th century.
A definite choice of the base units has become important only after a bunch of new physical quantities have been defined for use in the theories of electricity, magnetism, heat and light, in the second half of the 19th century. When dealing with so many different physical quantities, not using unique base units would have caused too much confusion. While this necessity has been recognized, for many years 2 different choices for the base units were widespread, that based on meter-kilogram (used mostly by engineers) and that based on centimeter-gram (used mostly by theoreticians). Meter and kilogram were more typical for the sizes of practical machines, while centimeter and gram were more typical for the sizes of laboratory experiments.