HN has been interwoven with Quickwit's journey from the very beginning. Looking back, it's striking to see how our progress is literally chronicled in our HN front-page posts:
- Searching the web for under $1000/month [0]
- A Rust optimization story [1]
- Decentralized cluster membership in Rust [2]
- Filtering a vector with SIMD instructions (AVX-2 and AVX-512) [3]
- Efficient indexing with Quickwit Rust actor framework [4]
- A compressed indexable bitset [5]
- Show HN: Quickwit – OSS Alternative to Elasticsearch, Splunk, Datadog [6]
- Quickwit 0.8: Indexing and Search at Petabyte Scale [7]
- Tantivy – full-text search engine library inspired by Apache Lucene [8]
- Binance built a 100PB log service with Quickwit [9]
- Datadog acquires Quickwit [10]
Each of these front-page appearances was a milestone for us. We put our hearts into writing those engineering articles, hoping to contribute something valuable to our community.
I'm convinced HN played a key role in Quickwit's success by providing visibility, positive feedback, critical comments, and leads that contacted us directly after a front-page post. This community's authenticity and passion for technology are unparalleled. And we're incredibly grateful for this.
Thank you all :)
[0] https://news.ycombinator.com/item?id=27074481
[1] https://news.ycombinator.com/item?id=28955461
[2] https://news.ycombinator.com/item?id=31190586
[3] https://news.ycombinator.com/item?id=32674040
[4] https://news.ycombinator.com/item?id=35785421
[5] https://news.ycombinator.com/item?id=36519467
[6] https://news.ycombinator.com/item?id=38902042
[7] https://news.ycombinator.com/item?id=39756367
[8] https://news.ycombinator.com/item?id=40492834
Anyway tantivy is great! I love pg_search https://www.paradedb.com/blog/introducing_search (which appears to be built by another company, but on top of tantivy, which is a great feature of open source)
Now, I am worried about development being stalled after this acquisition. How does further developing tantivy in the open helps Datadog's bottom line?
Re: Tantivy. I'm hopeful the community Paul and the Quickwit team have built on top of Tantivy will continue to flourish. I'm sure Datadog will build product(s) with Quickwit, which is built on Tantivy and will contribute to it. Many other companies like ours (ParadeDB) and other databases also integrate it. I can't speak for others, but we'll contribute whenever possible. We're currently working on supporting nested documents in Tantivy, for example, and hoping to upstream this work.
While it's reasonable to be concerned, I'd say this is a win for Quickwit, Tantivy and, of course, the well-deserving team behind them.
As someone who never managed to built a fond relationship with Apache Lucene based products (Solf, Elastic). I was extremely happy to see Tantivy in open source.
BM25 scoring, proper asian language support, speed, memory foot prints, etc - amazing job! Thank you so much!
https://github.com/quickwit-oss/tantivy
IMHO Datadog made a smart move!
If Tantivy itself just stays permanently under Apache2 licence and find a sustainable path to co exist with the rest of open source community - it's all good guys. You are more than deserve a commercial success.
> We will be focused on building a new product with Datadog, and to ensure our open-source community can continue, we will soon release a major update of both Quickwit with a relicense to Apache License 2.0 and tantivy.
So, it looks like we'll get a more liberally licensed Quickwit, but reading between the lines suggests development of it is might otherwise be winding down? It has been pretty nice and stable in my experience, so I can't really complain much. But I was really looking forward to what else it could bring.
Congrats to the team, in any case!
They will stop fulltime day-to-day effort in it themselves, probably because they have been relocated to writing a similar service but closed and integrated in DD, but it seems they want to opensource the current product with a OSI compliant license in the hopes that the community picks up the tab.
I think that's a nice trade. Could have been much worse.
By the way, also note that DD is not a total stranger in the OSS space. They actually opensourced their observability pipeline tooling for general use as Vector, which is a rock solid product. - https://vector.dev/
https://www.datadoghq.com/blog/datadog-acquires-timber-techn...
But Vector is something that complements Datadog's offering very well, so I think that makes sense for them to be good stewards of it. Quickwit is something that somewhat actively competes against them, which is a big difference. I suspect that unlike Vector, Quickwit is probably going to stop seeing any development in pretty short order, unless the devs now can consciously go out of their way to dedicate extra hours to it.
To be clear, I think that the relicense is great, and I think it's very possible that Quickwit will be picked up/forked by someone and maintenance will continue, because it's very good, and I'd really love to see someone do metrics for it as well. So, I'm not all gloomy or anything like that.
OrioleDB continues to be a fully open source and liberally licensed. We're working with the OrioleDB team to provide an initial distribution channel so they can focus on the storage engine vs hosting + providing lots of user feedback/bug reports. Our shared goal is to advance OrioleDB until it becomes the go-to storage engine for Postgres, both on Supabase and everywhere else.
Happy to hear any concerns you have
I have met the founders of all 3 of these companies and can assure you they all care tremendously about bringing their work to the world.
ParadeDB is independent and without plans to sell anytime soon, though :)
It also looks like most of DD's observability acquisitions are either integrated directly (seemingly with a full rewrite) or look a lot like aquihires for senior folks, so I wouldn't hold my breath here.
I can't imagine they feel great about Quickwit getting bought by a competitor after that.
The good part for the rest of us is it's a signal that there's likely some appetite for a fork if Datadog screws the pooch.
Binance built a 100PB log service with Quickwit (228 points, 6 months ago, 195 comments) https://news.ycombinator.com/item?id=40935701
Show HN: Quickwit – OSS Alternative to Elasticsearch, Splunk, Datadog (145 points, 1 year ago, 51 comments) https://news.ycombinator.com/item?id=38902042
Rhetorically, why was it time for this?
Practically, the answer is right there: the VCs wouldn't accept a mere rapidly-growing company with great tech. It's either an up round so they can mark up the value on their portfolio, or if the market isn't hot enough for a high-priced Series A, force an exit.
Remember that this is a decision you make when you accept the seed money, not when it comes to looking for further money. If you want to build a profitable growing company, bc funding is not the path to that.
If you suggest something they don’t like, they’ll stall until you run out of money, or force you out.
The sentence in the blog post is a tad misleading. I suspect François is not really talking about VCs that had already invested in quickwit, but about the usual flow of other VCs who contacted us, to know about the company and be part of our eventual series A.
It just generally felt like we were "at a crossing".
No one twisted our arm.
Regardless, in my case, it's not my first rodeo; I didn't give up any board seats or majority control. I mostly raised from angels that I know personally and trust me. And we're a high-margin B2B business (apparently similar in those limited respects to Quickwit) so I don't feel particularly capital constrained without raising additional rounds.
So I don't have a board pushing me for "valuation events." But I am very familiar with this toxic dynamic from my previous experience.
Genuine question: has the company changed enough in the interim to deserve a second look?
We did get absolutely burnt by other manifestations of the DataDog approach: The billing model was (is?) very much not good, transparent or predictable and staying on top of costs was close to a nightmare. The way surprise costs and contract changes (triggered by them) was handled did not feel honest.
The product itself is great but from my perspective it's absolutely not worth having to deal with their business side of things and the risks, costs (money, time, attention) and stress associated.
If I were a Quickwit customer I'd start looking for alternatives.
Datadog is a pain in the ass. I've got two emails and a voicemail from them just this week. We are not an active customer.
Heroku/Salesforce is also a pain in the ass. It causes enough friction with legal that I'll spend whatever effort it takes to replatform our workload just to not have to have those unending inbound calls.
NS1 was easy-peasy, but post-IBM I now receive a PDF invoice for $50 once per month with no credit card-based billing options and have to remind finance to cut a paper check. I'll be rehoming our DNS as soon as we decide on where to move it to.
tl;dr: the business experience is part of the product
As my ex manager once told - there is no such thing as nice people in P&L statement. Someone has to pay
It's very easy to be anxious and see the path to the dark side here.
However one of possible outcomes - there will be a valid open source competitor to Grafana ecosystems, and this along secure the rest of scene from relicensing. There is a chance it will be all win-win with clear sustainable path and no money and power struggles for founders.
I want to stay on the positive here. Time will show.
You may want to check out SigNoz [1] - takes a all in one app approach compared to different modules for each signal approach which Grafana takes
Disclaimer : I am one of the maintainers
Damn I never get sales pitches like that.
- It is non-trivial to setup and operate. It requires properly configuring and orchestrating numerous components - distributor, ingestor, querier, query frontend, compactor, consul, ruler, memcache, index service, etc. - see https://grafana.com/docs/loki/latest/get-started/architectur...
- Its' configs tend to break with every new release, since some configs become obsolete and some configs change names and/or structure.
- It is very slow on "needle in the haystack" queries such as "search for logs containing some rarely seen trace_id", since it needs to read, unpack and scan all the logs at object storage. This may be also very expensive if the object storage service charges for read IO.
- It doesn't support log fields with big number of unique values (aka high-cardinality log fields). It you'll try storing logs with high-cardinality fields into Loki, then it will quickly explode with enormous RAM usage. Recent Loki releases provide half-baked solution for this problem - "structured metadata" ( https://grafana.com/docs/loki/latest/get-started/labels/stru... ), but it is still experimental and requires non-trivial configs (aka it doesn't work).
https://github.com/SigNoz/signoz
PS: I am maintainer at SigNoz
> Organizations in financial services, insurance, healthcare, and other regulated industries must meet stringent data residency, privacy, and regulatory requirements while maintaining full visibility into their systems. This becomes challenging when logs need to remain at rest in customers’ environments or specific regions, hindering teams’ ability to attain seamless observability and insight. To help our customers meet these requirements without sacrificing visibility or introducing multiple logging tools, we are pleased to announce that Quickwit—a popular open source distributed search engine—is joining Datadog.
That makes sense. Datadog has been pure SaaS the whole time, which is unusual. Buying a good db engine like Quickwit would be a smart head-start into the on-prem segment which is a natural expansion opportunity.
I've previously made the prediction that Datadog is the new Cisco - can expect lots of acquisitions to be made going forward.
What happened to Vector the last opensource they bought? Are they still hired?
I joined Datadog after the Vector acquisition and now currently am the manager for the Community Open Source Engineering team that works on Vector open source.
Just confirming strongly what ripley12 said, as a person with direct involvement in OSS at Datadog.
I haven't kept super close tabs on it but last year we were hiring for a role to do tech lead stuff and OSS community building for Vector, and yes several of the original Vector employees still work here.
My guess is you will be billed per gig or something but not nearly the cost of shipping your logs to DD
I loved that I was able to setup Quickwit on AWS lambda and have a good cloud based search engine for $0.01 / month.
If you're used to traditional Enterprise pricing it's fairly priced for the value you get but anybody coming from self-funded or VC it's very expensive. If you're already using Splunk you can afford it.
One of its best features is it's consumption priced not by seats so easy to open up for all of eng including product/QA teams and not just devs to use which is great for breaking down barriers.
It's a bit like aws in that it's a platform - use of one product tends to encourage using more from their suite.
I used to log large apps using kibana and elastic search. Also using the clusterfuck that are all AWS tries at this (cloud watch, log insights.amd whatnot).
Nothing compares to what DD give you in observability.
Having said that, DD should only be an AWS feature. They should buy them for a couple of billions and integrate it as a service for all of AWS infrastructure.
Showing ALL the logs isn’t cool. Showing SOME logs at random is cool.
