The next generation of Materialize (opens in new tab)

The poor man's version of Materialize that I implemented is the following:

Step 1) Find all "paths" between tables

* Use the postgres information schema to get all relations * Use npm library graph-cycles to see if there are any graph cycles. If so.. some relations go on a blacklist. * Use npm library topopsort to sort the graph * Traverse the graph and find all possible paths from and to tables * Generate SQL queries to look up "affected" other tables. Input: table name + id Output: list of table names + ids

Step 2) Track changes in database using a transactional outbox

* Create outbox table fields: id, timestamp, event, table_name, table id, jsonb payload * After running migrations have a script that ensures every table has triggers on insert,update,delete that would insert a row on each of these events in the outbox

Step 3) Compute ( You want to do reporting for a certain table ( target table ) which has relationships with other tables and for this table you want a "materialized" view )

* Have a script on a loop that takes all the outbox entries since last time processed * Use the queries from step1 to find out which ids of "target table" are affected * Then only materialize / compute those ids * Store in Elasticsearch ( in our case )

This is not a solution if you are Facebook scale. But as a small SAAS company with not too many transactions this works brilliantly. And with more tenants you can just scale this up by sharding the inbox.

Bonus points: have elasticsearch (ingest node) and a postgres replica on the same host as the "compute" script. So if you have a lot of queries calculating the "dependencies" you get better performance.

sorry for my terrible explanation

I've worked on specific instances of this problem at a few companies and its hard! The number of ways distributed systems can fail only serves to compound the complexity here as well. Each time I've built one of these systems its taken years to get it almost good enough to keep customers happy and to meet our performance goals. However, the resulting system were plagued by deep set correctness bugs that were the bi-products of early decisions made by engineers who didn't understand correctness.

It's really exciting to see a system getting built by people who know what they are doing and will be hopefully a more "correct" solution.

This is really impressive!

I've been following Materialize as their blog posts are a great source of inspiration when working on OctoSQL[0] (a CLI SQL dataflow engine), but was a bit surprised with how few data sources they were supporting (basically Kafka and Postgres based on their docs), but now that they're switching/pivoting to being a database themselves, this makes much more sense.

I also think the architecture is really cool. Cloud-native is the way to go for modern databases and will make adoption much easier than something you'd have to host on bare metal. One question though, does this mean the open-source version is basically deprecated now and further development is closed-source, or does the open-source project represent the "compute" part of the "next gen Materialize"?

Congrats and good luck with further development!

[0]: https://github.com/cube2222/octosql

From a technical point of view Materialize seems absolutely fantastic - everything that the Firebase databases are trying to be, except with full SQL support (well, minimal SQL support - there's no RETURNING, ON CONFLICT or json - but probably good enough, we at least get JOINs and CTEs).

But unfortunately they only seem to be interested in enterprise customers: Not only is it not open source, but there's no open sign up and pricing for compute nodes isn't publicly available!

The downside is they pivoted away from supporting running it yourself. The technology is certainly exciting but also changes the target demographic. I'm curious to see how it plays out

Wow that's sad, I was really excited by the promise of this project. But going cloud native means I won't be able to use it in any of my projects any time soon.

I’m starting to play with differential-dataflow in a new Rust project. On the one hand, it’s cool. On the other, it could use better documentation.

With Materialize the database, it really depends on price/performance whether I could use it (could it be really cheap when idle, like Aurora Serverless?), but two things that would make it easier to use would be the ability to purchase it through AWS Marketplace and to deploy it/use it in my organization’s own AWS account. As an enterprise dev team senior manager I then do not have to go through a vendor approval process or deal with my procurement department, nor do I need to worry about third party data control.

A lot of folks are reacting to the fact that it seems like this new version won't be able to be self-hosted. And while I get why that's a turn-off to many orgs, there's a massive market for turnkey, managed data products like this. You can get really far without having to staff data or infra engineers. The way I see it, the next phase of "tech" company will deemphasize the amount of in-house engineering outside of their core areas of innovation. There will, of course, be many companies that remain engineering-driven, but I see that as less the assumed norm going forward.

Does this fix how much of an insane memory hog materialize is? Some queries are just impossible if you can’t use disk. This is why I was forced to stick with Flink. Even though materialize makes things appear stupid simple and easy with SQL, I found that you can only do the most simple streaming views with it. You can’t even do unique counts with this for very long without breaking—and there’s no probabilistic alternatives.

Bad for big data. Great for small and simple data sets. But who is using Kafka with small data?

Also they do not integrate at all with custom data types in Postgres IME. E.g. an enumeration in your table will mean materialize can’t read the table as a source. Lame.

I use PG with an alternative materialized views implementation[0] that is pure PlPgSQL and that exposes real tables that can be used to write to in triggers, and where the views can be marked stale too.

This means hand-coding triggers to keep the materializations up to date, or else to mark them as out of date (because maybe some operations would be slow or hard to hand-code triggers for), but this works remarkably well.

As a bonus, I get an update history table that can be used to generate updates to external systems.

In principle one can get the AST for a VIEW's query from the PG catalog and use that generate triggers on the tables it queries to keep it up to date. In practice that's only trivial for some kinds of queries, and I've not written such a tool yet.

[0] https://github.com/twosigma/postgresql-contrib/blob/master/m...

I've been following the Materialize project and Frank McSherry's work for a long time and seeing them go cloud native should really democratize the use cases for the mainstream. yes there are trade-offs associated with this but this will make it usable by the vast vast majority of the market.

Also I love the tagline "Consistency, Scalability, Low Latency: Pick Three"

There seems to be an overlap with ksqlDB functionality. ksqlDB computation part is based on Kafka Streams. https://github.com/confluentinc/ksql

Can someone please explain, without technical buzzwords, what Materialize is?

I checked the docs but they mention streaming DB, Timely and Differential Dataflow which I don't know either.