For those use-cases you might just be better off with either contract tests or simply making the payload itself typesafe with something like protobuf.
OpenAPI/Swagger is basically inextricably bound to REST/JSON APIs over HTTP. I also think they're quite abstruse in terms of defining non-trivial interfaces.