Ask HN: How can I monetize a load balancer for ML applications?

2 pointsstormbeard1y ago10 comments

I have a solution that solves a set of problems that keep showing up in ML workloads. The kind of systems I'm talking about are ones where:

- You have a GPU attached to each instance.

- Each request takes anywhere from 10ms to 2min.

- There's a hard limit on the number of in-flight requests/queries (I assume because of the GPUs).

Normally, I see people fronting the instances with software load balancers, but this doesn't work very well for reasons. Assuming I have a solution in the form of a fancy load balancer, how would I go about monetizing it? Let's assume the solution is non-trivial to create, but very straightforward to use (essentially a drop-in replacement).

I ask because I don't think I can just "sell a fancy load balancer" like it's the late 90s or something. Modern companies appear to always have more complicated products and I just want to sell a straightforward piece of infrastructure that solves a fairly hard problem.

Thanks in advance.

10 comments

sargstuff1y ago

Is there accessible documentation which covers installation & non-functional requirements (aka hardware/software requirements & how to setup/use the solution)

stormbeardOP1y ago

I can write these things, and I assume it will be necessary for anything someone pays for, but I'm mainly just asking a question so I can learn about how people would monetize these things (if it's even possible to) nowadays.

sargstuff1y ago

Ok, backing up a few steps. Can look at things that have been done in the past related to licensing/usage fees relative to computational power. aka statistical/mathematical / operating system / database packages/software.

Modern cloud processing has 'fees/payment' tied to scale/balancing per use requirements (not necessarily tied specifically to gpu's).

sargstuff1y ago

what was done to access "doesn't work very well for reasons"? aka monitored systems in questions and saw ...... ?????

What were the "reasons" for "doesn't work very well? aka trying to do goolgle search type work on 2mb intel 486 oover a 2mb network and expecting to be able to compete with google is never going to work out.

What type of load balancing? Load balancing typically has to be tuned/adjusted based on end usage requirements/production environment (not just per factory setting)

stormbeardOP1y ago

From what I've seen, there's usually some hard requirement that a single instance of the application can handle N in-flight requests. This is fine when you have a single load balancer that uses some flavor of "least request" or "least connection" balancing algorithm. The problem arises when you have a fleet of load balancers- they don't share state, so each LB can't know the state of any individual backend without sending a request. This results in a failed request if the backend is already at full capacity.

Your tone is coming off as condescending and I'm not sure if it's intentional. I guess I should mention that I'm _very_ familiar with how load balancing works. This is a real problem and you can't "tune" your way into a consistent global view of the backend states.

sargstuff1y ago

But do have control over how the data parcels causing load are partitioned/preped before being parceled out. (which requires knowledge of resource metrics/topology-- distributed resources within computing box vs. distributed external computing resources ).

Pre-modern computing, load balancing was a telecommunications field thing. Cloud computing is the modern 'load balancing' take.

1 more reply

talldayo1y ago

> but this doesn't work very well for reasons.

Which reasons? In my experience/exposure, people are perfectly happy with Proxmox on a big GPU-laden boxen.

stormbeardOP1y ago

I didn't mean to imply that this is a problem with arbitrary machines that have GPUs. There are specific kinds of applications that use GPUs and because of how they work have hard limits on the number of tasks they can concurrently process on those GPUs.

Either way, I feel like these details are orthogonal to my original question. Do you think it matters?

JSDevOps1y ago

Why doesn’t software load balancers work? Also surely you just implement queuing?

j / k navigate · click thread line to collapse

Ask HN: How can I monetize a load balancer for ML applications?

2 pointsstormbeard1y ago10 comments

I have a solution that solves a set of problems that keep showing up in ML workloads. The kind of systems I'm talking about are ones where:

- You have a GPU attached to each instance.

- Each request takes anywhere from 10ms to 2min.

- There's a hard limit on the number of in-flight requests/queries (I assume because of the GPUs).

Thanks in advance.

10 comments

sargstuff1y ago

Is there accessible documentation which covers installation & non-functional requirements (aka hardware/software requirements & how to setup/use the solution)

stormbeardOP1y ago

sargstuff1y ago

Modern cloud processing has 'fees/payment' tied to scale/balancing per use requirements (not necessarily tied specifically to gpu's).

sargstuff1y ago

what was done to access "doesn't work very well for reasons"? aka monitored systems in questions and saw ...... ?????

What type of load balancing? Load balancing typically has to be tuned/adjusted based on end usage requirements/production environment (not just per factory setting)

stormbeardOP1y ago

sargstuff1y ago

Pre-modern computing, load balancing was a telecommunications field thing. Cloud computing is the modern 'load balancing' take.

1 more reply

talldayo1y ago

> but this doesn't work very well for reasons.

Which reasons? In my experience/exposure, people are perfectly happy with Proxmox on a big GPU-laden boxen.

stormbeardOP1y ago

Either way, I feel like these details are orthogonal to my original question. Do you think it matters?

JSDevOps1y ago

Why doesn’t software load balancers work? Also surely you just implement queuing?

j / k navigate · click thread line to collapse