That's strange, in my experience it shouldn't have 6x slowdown. Probably might be due to several factors, but here's your test, running on my system without Docker:
Ryzen 3900X (host machine)
$ dd if=/dev/urandom bs=4k count=10k | gzip >/dev/null
10240+0 records in
10240+0 records out
41943040 bytes (42 MB, 40 MiB) copied, 1.02284 s, 41.0 MB/s
qemu-aarch64-static
$ dd if=/dev/urandom bs=4k count=10k | proot -R /tmp/aarch64-alpine -q qemu-aarch64-static sh -c 'gzip >/dev/null'
10240+0 records in
10240+0 records out
41943040 bytes (42 MB, 40 MiB) copied, 3.33964 s, 12.6 MB/s