undefined | Better HN

0 pointsgeorgyo3y ago0 comments

This sounds truely nightmarish and costly. Even moderate volumes of data are going to add up very quickly cost wise.

I see the term zero-ops. But maintaining and debugging this pipeline is going to require some ops, even if you are not managing VMs.

0 comments

shaeqahmed3y ago

Using and maintaining Matano is a fraction of the cost compared to popular non-serverless alternatives like ELK or Spunk. Matano is specifically designed for petabyte-scale security analytics use-cases that don't fit in a traditional SIEM.

The serverless data ingestion pipeline means you don't need to over-provision for ingestion (Logstash and Splunk Forwarders are notorious for related costs / ops in high scale use-cases) in the write path. For reads, since Matano queries Iceberg tables backed by highly-compressed parquet files on object storage you won't pay anything close to what you would for a database or search engine based SIEM.

mdaniel3y ago

> For reads, since Matano queries Iceberg tables backed by highly-compressed parquet files on object storage you won't pay anything close to what you would for a database or search engine based SIEM

Where do you show an example of querying anything? There's an empty "detector" in the examples directory, which I guess gets called once per row of this 20MiB/s alleged elsewhere?

Anyway, I find comparing this to Splunk to be a bit premature

FridgeSeal3y ago

Tools like Spark, Trino, etc can be pointed at parquet/iceberg/etc files in S3, and they'll let you issue SQL queries against the files directly. Means it integrates out of the box with whatever data tooling is being used in your org already.

j / k navigate · click thread line to collapse