I have had to submit a few pull requests to various projects along the way, but didn't find the ecosystem prohibitively lacking.
That isn't all of them but those are the main ones.
In techempower benchmarks it exceeds 2 million http requests/second and it's a full REST framework.
And if you use the fiber support through Quasar you can pretend most things are normal blocking code.
Have to tried it or is this a case of NIH?
I'm fascinated to see they are running that all on a single node. Its a massive amount of state aggregated from billions of events that needs to be served at extremely low latency, but couldn't it be partitioned somehow??? Google Fi/Spanner and BigTable have certainly been developed to support these issues. I've been trying to dig up what infrastructure powers Google AdX, but I haven't found anything. AdWords seems to be tied to Spanner, but AdX is/was an entirely different beast. In any case I'm quite certain that it isn't running pacing on a single, gigantic node.
Suffice to say, I spent 730$ within _seconds_, so fast actually Googles systems couldn‘t even switch off fast enough to prevent 7,3x overspend, and the only thing that prevented stupid me from a five digit spend was probably choosing an unusual ad size.
Fascinating stuff indeed :)
The bidder/pacer state is not necessarily massive, and certainly it does not consist of all the gazillions of past events. Depending on the strategy/bidding model, it can range from a few MB to several GBs, something that can fit in a beefy node.
> Google Fi/Spanner and BigTable have certainly been developed to support these issues.
I doubt any external store can be used with so low latency constraints (2-10ms) and high throughput (millions RPS). Perhaps Aerospike but even that is a stretch to put it in the hot-path. At this scale you're pretty much limited to fetch the state in memory and update it asynchronously every couple of minutes/hours.
Source: I also work in ad tech.
For anyone else confused it's probably Google F1 and Spanner.
Is it golang or pony or F$? CoreFX mention in the end confused me more.
It already does most of what you want and has support for native epoll transport.
I'm not sure what led Vert.X to be discarded, maybe not a Java shop? But we've used it extremely successfully for high performance REST and I know of several high profile tech companies that swear by it.
There's nothing I know of that compares with Vert.X in performance, stability, and popular adoption
While the bandwidth benchmarks are fun to see and write blog posts about, we also care a lot about keeping tail latency at or below 2ms than we do about getting more bandwidth at this point.
1 - See httpbeast in the latest round of the Techempower benchmarks: https://www.techempower.com/benchmarks/#section=data-r17&hw=...
Also why is evey immature native language considered, but the two speed demon languages without garbage collection - ISPC and C++ - are nowhere to be found?
Looks like libuv directly in C11? (not F# as before edit).
I am learning Clojure so I would like to know if anyone knows of the highest performant applications written in it.
This makes me curious - was it the language or the runtime characteristics?
Imagine if a crime syndicate would brag about their efforts to make their worldwide criminal activities more efficient.
Therefore I think the best we can hope for is that engineering breakthroughs achieved in profit driven fields will gradually leak into other fields where they can actually be used to improve people's lives.
Breakthroughs achieved would be funded by taxpayers and customers of these agencies and the patents for technology produced would be more likely enter the public domain.
As I read this article I was thinking "I bet the first comment on the article will just be complaining about the effort going into ads", and I wasn't wrong.
Honestly, it's becoming tiresome. Low latency computing is a great area to talk about, and instead it's derailed by this essentially off-topic discussion.
Because that's what happens if you don't have web advertising. Free things disappear without revenue.
Or maybe you'd prefer to go back to the days of randomly-targeted or "PUNCH THE MONKEY" ads. Because THAT'S what happens without ad auctions and targeting.
The reality is: advertisers and ad-supported sites WANT to show you a relevant ad that you're likely to click (modulo obvious bad actors). That's how they get paid. Anything else is, by definition, "[wasting] users' time and attention."
I run some services that are paid for by advertising and I see this myself: the second I start trying to charge money, I'm now the one service that people see as costing them money when all my competitors are "free." And the associated entitlement is toxic.
To answer your first question, imagine a world where advertising doesn't exist and we simply pay for those things.
I was reluctant to admit this since ads paid my way through university. But as ads make less and less money, it's harder and harder to ignore the truth. And while ads are viable, they impede the necessary cultural transformation we'll need to form a healthier relationship with the goods and services we want.
It actually boils down to an expectation of slave labor for a lot of things, which people don't want to hear.
They might disappear, but not without replacements. Hosted email, videos, photos, discussion boards, and search are all services that would exist with or without a business model built on ad money. I don't care if they are free or paid. But the fact is that the privacy-depraved model wins because it's provided for free to the eyeballs.
And yes, I'd rather have punch-the-monkey ads. I'd even go so far as to say that I'd prefer my search results and most content-provider's offerings not ordered and filtered based on my previous behaviors.
The web doesn't work without free services. For everyone who has extra money they would gladly use to pay for everysite they visit the majority won't and you will end up buying overpriced bundles from your phone company and your privacy will be worse. Free allows users a chance not to be tracked. Once you start involving money you can easily be traced.
Facebook? There are plenty of sites like facebook without users. Charge for an account.. reduce the userbase, the platform loses value without the free users.
Was I supposed to respond “oh no, current ads are so much better”?
That doesn't match my experience at all, but maybe you can chalk it up to targeting?
Also you could learn a lot about tunnel building, and tunnel detection.
https://www.cnbc.com/2019/05/22/irish-data-privacy-watchdog-...
Found this article great, not many places to see 5m req/s, let alone on a single node.
I'm really interested in hearing more about those databases!
