The factors that I think would make this service most valuable are low cost (think, lower than GPU's on AWS or similar, even at scale), high burst capability from cold start (1000QPS is a good target), and of course low cold start delays (< 1s, or .5s).
This led me down a rabbit hole in years past and the technical solution seems to be generally, the ability to swap models in and out of GPU ram very quickly. Possibly using NVIDIA's unified memory subsystem.
We don't have any cold start delay! In our custom environment, you can do exactly what you are describing (running both CPU and GPU code). We provide you with access to the GPU and the CUDA libraries installed. It's basically lambda (minus the cold start) with GPU access.
We can scale a lot very quickly depending on how much you need.
Are you willing to talk a bit about how this all works? I assume you host the hardware yourself somewhere, which in the days of AWS et al must be pretty tough to pull off, especially with these specs. Where do you get the hardware from these days with the crypto craze?
https://NN-512.com (open source, free software, no dependencies)
With batch size 1, NN-512 is easily 2x faster than TensorFlow and does 27 ResNet50 inferences per second on a c5.xlarge instance. For more unusual networks, like DenseNet or ResNeXt, the performance gap is wider.
Even if you allow TensorFlow to use a larger ResNet50 batch size, NN-512 is easily 1.3x faster.
If you need a few dozen inferences per second per server, this is the cheapest way. And you're not depending on a proprietary solution whose parent company could go out of business in a year.
If you need Transformers instead of convolutions, Fabrice Bellard's LibNC is a good solution: https://bellard.org/libnc/
> If you need a few dozen inferences per second per server, this is the cheapest way. And you're not depending on a proprietary solution whose parent company could go out of business in a year.
Definitely the cheapest way.
We've been in business for more than a year already actually :)
I'm giving performance comparisons versus TensorFlow, which I consider to be a standard tool.
People who use your proprietary, closed, black-box service are dependent on the well-being of your business. You could vanish tomorrow.
To me, adding GPUs into the devops mix typically increases the complexity significantly, and I would definitely pay money to someone who can just take my model, host it, and let them deal with the complexities around it.
This sounds confusing to me. Surely it is possible to craft a neural network that takes longer to process?
> Max. model size: X GB
Do you really mean model size or should this also include the size of the intermediate tensors?
The full screen option on the YouTube video is turned off by the way, so it is impossible to read without leaving your website.
Overall, this offer looks quite competitive. Are you planning to offer your service in the EU in the future?
The model size is the zipped size of your model that is uploaded to Inferrd (either through the SDK or the website).
I'll fix the full screen problem right away, thank you for reporting.
We only have servers in the United States at the moment but are looking to have servers all around NA and EU very soon.
Nice to hear!
> We only have servers in the United States at the moment but are looking to have servers all around NA and EU very soon.
Sorry, my question was not quite clear. What I actually wanted to know was more along the lines of being able to use your service in Europe legally. For example, I can not find a privacy policy or a way to get a GDPR data processing agreement.
I’ve never heard of that type before and I wasn’t able to find anything with google.
Furthermore more, the lack of company information (address, company registration nr etc) and the fact that it’s not clear where the servers are located geographically makes me a bit hesitant.
CUDA 11.3.0
cuBLAS 11.5.1.101
cuDNN 8.2.0.41
NCCL 2.9.6
TensorRT 7.2.3.4
Triton Inference Server 2.9.0
I'm new to deploying to production inference so I'm not sure if those are easily portable across such platforms or not really.
Edit: Actually, I didn't spot the free tier of 1000 requests. I wonder how you avoid the problem of a lot of users leaving defunct/disused models running while still keeping them hot - presumably some kind of limit to the model count?
