fish passes the -G flag to 'ls' by default, which produces colorized output, but stats each file. So fish is actually outputting more information than zsh, and what your test is really measuring is the speed of '/bin/ls' vs '/bin/ls -G'
To compare ls speeds for the same workload, run it directly: /bin/ls. I expect no difference, because it is the same binary.
If you wish to compare globbing speeds, do not use bare ls, which does more work in fish. Instead, invoke the ls binary: '/bin/ls * '. Or better yet, just echo: 'echo *'.