The config makes this easy, but my favorite part is the fact that the CPU and MEM of the vector processes barely even registers on my metrics charts. I can't even tell you what their actual and requested resources are because I haven't bothered to look in a while.
It's one thing I never have to worry about. I could use more of those.
I think the interesting part is the overlap between observability metrics for operations and expanding in to BI metrics with the same tools.
I built Logflare so you could get your structured logs into BigQuery directly, so now you this :)
Hopefully we'll have a Vector sink soon, but until then I think they support POSTing batches to HTTP endpoints so you can do that and we'll take any JSON and go straight to BQ with it after migrating your schema for you (automatically based on the incoming payload shape).
Edit: Awesome, my complaint that you couldn’t scrape the Prometheus federation endpoint has fixed.
I'll go ahead and open an issue to get that addressed, but in the future please feel free to do so yourself for anything that's tripping you up! We really value this kind of feedback and try to address it as promptly as possible.
The diagram and succinct summary give you a very high level view, but the Docs link at the top takes you this:
I get that you want to sell this to executives, but the people that point executives to products to buy are the developers like me, and if don't understand what it does, I can't see executives finding your product.
Just give examples of what on-the-ground problems is solves, converts kafka messages to s3 objects or what?
Lack of novel data enrichment is not a problem that I have, but I could do with something that streams k8s pod logs to kafka or s3 or whatever. Does this solve that?
That being said, we do know of a few cases where Vector's in-flight processing and routing capabilities were enough that a full Kafka-based pipeline was no longer needed. This ability to push computation out to the edge and reduce the need for centralized infrastructure is one of the aspects of Vector that we're most excited about.
