Traditionally, this was unfeasible: Postgres is a rowstore database that’s 1000X slower at analytical processing than a columnstore database.
(A quick refresher for anyone interested: A rowstore means table rows are stored sequentially, making it efficient at inserting / updating a record, but inefficient at filtering and aggregating data. At most businesses, analytical reporting scans large volumes of events, traces, time-series data. As the volume grows, the inefficiency of the rowstore compounds: i.e. it's not scalable for analytics. In contrast, a columnstore stores all the values of each column in sequence.)
For decades, it was a requirement for businesses to manage these differences between the row and columnstore’s relative strengths, by maintaining two separate systems. This led to large gaps in both functionality and syntax, and background knowledge of engineers. For example, here are the gaps between Redshift (a popular columnstore) and Postgres (rowstore) features: (https://docs.aws.amazon.com/redshift/latest/dg/c_unsupported...).
We think there’s a better, simpler way: unify the rowstore and columnstore – keep the data in one place, stop the costs and hassle of managing an external analytics database. With Hydra, events, traces, time-series data, user sessions, clickstream, IOT telemetry, etc. are now accessible as a columnstore right alongside my standard rowstore tables.
Our solution: Hydra separates compute from storage to bring the analytics columnstore with serverless processing and automatic caching to your postgres database.
The term "serverless" can be a bit confusing, because a server always exists, but it means compute is ephemeral and spun up and down automatically. The database automatically provisions and isolates dedicated compute resources for each query process. Serverless is different from managed compute, where the user explicitly chooses to allocate and scale CPU and memory continuously, and potentially overpay during idle time.
How is serverless useful? It's important that every analytics query has its own resources per process. The major hurdles with running analytics on Postgres is 1) Rowstore performance 2) Resource contention. #2 is very often overlooked - but in practice, when analytics queries are run they tend to hog resources (RAM and CPU) from Postgres transactional work. So, a slightly expensive analytics query has the ability to slow down the entire database: that's why serverless is important: it guarantees the expensive queries are isolated and run on dedicated database resources per process.
why is hydra so fast at analytics? (https://tinyurl.com/hydraDBMS) 1) columnstore by default 2) metadata for efficient file-skipping and retrieval 3) parallel, vectorized execution 4) automatic caching
what’s the killer feature? hydra can quickly join columnstore tables with standard row tables within postgres with direct sql.
example: “segment events as a table.” Instead of dumping segment event data into a s3 bucket or external analytics database, use hydra to store and join events (clicks, signups, purchases) with user profile data within postgres. know your users in realtime: “what events predict churn?” or “which user will likely convert?” is immediately actionable.
Thanks for reading! We would love to hear your feedback and if you'd like to try Hydra now, we offer a $300 credit and 14-days free per account. We're excited to see how bringing the columnstore and rowstore side-by-side can help your project.
