You could imagine creating a modified Go assembler that is more generic and not tied to Go's ABI that could accomplish the same effect as LLVM. However, it'd probably be better to create a project like that from scratch, because most of Go's optimizations happen before reaching the assembler stage.
It would probably be best to have the intermediate language that QBE has and transform that into "intermediate assembly" (IA) very similar to Go's assembly. That way the IL stage could contain nearly all the optimization passes, and the IA stage would focus on code generation that would translate to any OS/arch combo.