Scaling Data Pipelines on Kubernetes (opens in new tab)

(airbyte.com)

3 pointsdavinchia4y ago2 comments

2 comments

why didn't you use grpc?

Also, a standard kubernetes client (e.g. the go client), uses cache on the client side for queries, and does not hit the api server.

davinchiaOP4y ago

Since this is executed from Java code, we did not want to have to bundle kubectl (I'm assuming that's what you are referring to) with the Jar. We also did not want to run an exec command.

Are you asking why not GRPC between the various job pods?

j / k navigate · click thread line to collapse

2 comments

streetcat14y ago

why didn't you use grpc?

Also, a standard kubernetes client (e.g. the go client), uses cache on the client side for queries, and does not hit the api server.

davinchiaOP4y ago

Since this is executed from Java code, we did not want to have to bundle kubectl (I'm assuming that's what you are referring to) with the Jar. We also did not want to run an exec command.

Are you asking why not GRPC between the various job pods?

j / k navigate · click thread line to collapse