stormbeard

I have a solution that solves a set of problems that keep showing up in ML workloads. The kind of systems I'm talking about are ones where:

- You have a GPU attached to each instance.

- Each request takes anywhere from 10ms to 2min.

- There's a hard limit on the number of in-flight requests/queries (I assume because of the GPUs).

Normally, I see people fronting the instances with software load balancers, but this doesn't work very well for reasons. Assuming I have a solution in the form of a fancy load balancer, how would I go about monetizing it? Let's assume the solution is non-trivial to create, but very straightforward to use (essentially a drop-in replacement).

I ask because I don't think I can just "sell a fancy load balancer" like it's the late 90s or something. Modern companies appear to always have more complicated products and I just want to sell a straightforward piece of infrastructure that solves a fairly hard problem.

Thanks in advance.

I don't know much about consulting, so I wanted to ask some of the more knowledgeable folks here. Are there companies out there that would need consulting services for things related to microservice resilience, load shedding/balancing, and/or other things that fall under the L7/RPC networking umbrella? Think along the lines of retry storm prevention, fair rate-limiting, etc. "Backpressure stuff".

I've run across so many of the same problems at various unicorn/FAANG companies and as I've become more experienced I'm curious if there's a way to leverage this experience other than continuing to be a "staff+ engineer" at another Facebook/Uber/Snap/etc.

Thanks in advance for any advice/comments.