Yes, the filesystem access was slow both in Intel processor and now M1. I suppose that is to do with the Virtualisation involved. It felt a little faster accessing filesystem on the M1, but that’s just me and insignificant difference.
I’d say if you’re reliant on Docker for work, and that includes filesystem sharing, Macs may not be your best choice.