For us this package is most important as the query engine that powers Dolt:
https://github.com/dolthub/dolt
We aren't the original authors but have contributed the vast majority of its code at this point. Here's the origin story if you're interested:
https://www.dolthub.com/blog/2020-05-04-adopting-go-mysql-se...
- Don't use "mysql" in the name, this is a trademark of Oracle corporation and they can very easily sue you personally if they want to, especially since you're using it to develop a competing database product. Other products getting away with it doesn't mean they won't set their sights on you. This is just my suggestion and you can ignore it if you want to.
- Postgres wire/sql compatibility. Postgres is for some reason becoming the relational king so implementing some support sooner rather than later increases your projects relevance.
https://github.com/dolthub/doltgresql
Background and architecture discussion here
Dolt can do the same two directions of MySQL binlog replication, and also has its own native replication options:
Not enticing enough to build a business around, due to it being that bit too different and the persistence layer being that bit too important. But the sort of thing that I'd love it if the mainstream DBs would adopt.
I didn't realise the engine was written in Go, and honestly the first place my mind wonders is to performance.
[1] https://docs.dolthub.com/architecture/storage-engine/prolly-...
We needed to build a custom storage engine to make querying and diffing work at scale:
https://docs.dolthub.com/architecture/storage-engine
It based on the work of Noms including the data structure they invented, Prolly Trees.
https://docs.dolthub.com/architecture/storage-engine/prolly-...
The default proxied database is dolt. I'm guessing this is extracted from dolt itself as that claims to be wire-compatible with mysql. Which all makes total sense.
We didn't extract go-mysql-server from Dolt. We found it sitting around as abandonware, adopted it, and used it to build Dolt's SQL engine on top of the existing storage engine and command line [2]. We decided to keep it a separate package, and implementation agnostic, in the hopes of getting contributions from other people building their own database implementations on top of it.
[1] https://github.com/dolthub/dolt [2] https://www.dolthub.com/blog/2020-05-04-adopting-go-mysql-se...
> No transaction support. Statements like START TRANSACTION, ROLLBACK, and COMMIT are no-ops.
> Non-performant index implementation. Indexed lookups and joins perform full table scans on the underlying tables.
I actually wonder if they support triggers, stored procedures etc.
The bundled in-memory database implementation is mostly for use in testing, for people who run against mysql in prod and want a fast compatible go library to test against.
For a production-ready database that uses this engine, see Dolt:
On the JVM you have things like Cassandra, Elasticsearch, Kafka, etc. each of which offer performance and scale. There are lots more examples. As far as I know they don't do any of the things you mention; at least not a lot. And you can use memory mapped files on the JVM, which helps as well. Elasticsearch uses this a lot. And I imagine Kafka and Cassandra do similar things.
As for skillset, you definitely need to know what you are doing if you are going to write a database. But that would be true regardless of the language.
It is also true that the JVM and the GC are a bottleneck in what they are able to offer. Scylla and Redpanda's pitch is "we are like this essential piece of software, but without the JVM and GC".
Of course, having a database written in Go still has its pros and cons, so each to their own.
In Go, I’ve found that with a little bit of awareness, and a small bag of tricks, you can get very low allocations on hot paths (where they matter). This comes down to using sync.Pool and being clever with slices to avoid copying. That’s footgun-performance tradeoff that’s well worth it, and can get you really far quickly.
Also I suspect this project isn't for holding hundreds of GB of stuff in memory all the time, but I could be wrong.
Writing no-alloc is oftentimes done by reducing complexity and not doing "stupid" tricks that work against JIT and CoreLib features.
For databases specifically, .NET might actually be positioned very well with its low-level features (intrisics incl. SIMD, FFI, struct generics though not entirely low-level) and high-throughput GC.
Interesting example of this applied in practice is Garnet[0]/FASTER[1]. Keep in mind that its codebase still has many instances of un-idiomatic C# and you can do way better by further simplification, but it already does the job well enough.
Odds are if you need one written in Go then you’re requirements are somewhat different. For example the need to stub for testing.
Then we can finally have multiple database engine support for WordPress and others.
Somewhat relatedly, StarRocks is also MySQL compatible, written in Java and C++, but it's tackling OLAP use-cases. https://github.com/StarRocks/starrocks
But maybe this project is tackling a different angle. Vitess MySQL library is kind of hard to use. Maybe this can be used to build ORM-like abstraction layer?
We here at DoltHub use it to provide SQL to Dolt.
Given how important the DB layer is I would be careful to use something like this in production, but if it allows speeding up the test suite it could be really interesting.
TLDR; Because go-mysql-server existed.
https://www.dolthub.com/blog/2022-03-28-have-postgres-want-d...
We have a Postgres version of Dolt in the works called Doltgres.
https://github.com/dolthub/doltgresql
We might have a go-postgres-server package factored out eventually.
Could I swap storage engine with own key value storage e.g. rocksdb or similar?
... it would be a wild future if Vitess replaced the underlying MySQL engine with this (assuming the performance is good enough for Vitess).
if someone is considering running it, they're probably considering it against the actual thing. and I would think the main decision criteria is: _how much faster tho?_
We expect that it's faster than MySQL for small scale. Dolt is only 2x slower than MySQL and that includes disk access.
In dynamic language land, we tend to use real DBs for test runs. So having a faster DB wouldn't hurt!
