Show HN: GoSQL – A query engine in 319 LoC (opens in new tab)

(github.com)

81 pointsarchiewood1y ago9 comments

I've always been curious about how SQL engines actually work.

So I built a minimum viable SQL engine in Go.

- Supports CSV files as tables

- Supports SELECT, FROM, WHERE, LIMIT

It's very simple:

1. Parses query string

2. Converts it into an AST representation

3. Executes the query against the CSV

4. Returns the results

9 comments

stevekemp1y ago

Nice job.

You can see this post for the start of a guide in implementing something very similar "Writing a SQL database from scratch in Go":

https://notes.eatonphil.com/database-basics.html

(Use the tag "sql" to find the later parts. Sadly not linked directly from that first one.)

eatonphil1y ago

Thanks for mentioning! One of the most fun parts of this series I think is handling indexes on INSERT and actually making use of them based on (effectively) pattern matching on WHERE clauses.

> (Use the tag "sql" to find the later parts. Sadly not linked directly from that first one.)

There's a "Note" section right below that title that links to the other posts. :) I guess it is UX feedback that this was not obvious to spot.

archiewoodOP1y ago

Wow this is much more complete than mine, kudos.

iamcreasy1y ago

Nice. I also wanted to know the details behind database engine and ACID compliance. So, decided to follow Database Design and Implementation by Edward Sciore, and re-implemented the database in Python: https://github.com/quazi-irfan/pySimpleDB

This db treats file as raw disk and reads and writes in blocks. In the book, your step 3 and 4 will be a start of a transaction that uses recovery manager to log changes introduced by the query, and buffer manager to page in and out file blocks in memory. This book uses serializable isolation, so if buffer pool is full and can't page in new block or if another transactions are writing to that same block - the newer transaction will be rolled back after a brief wait.

lainga1y ago

Try this talk about sqlite!

https://www.youtube.com/watch?v=ZSKLA81tBis

zh24081y ago

Feel like you can achieve something similar in duckdb? duckdb allows you to query local csv, parquet, and even remote ones?

archiewoodOP1y ago

Oh definitely, I love DuckDB.

This was a learning exercise for me as much as anything

coredog641y ago

This is very handy given the recent de-emphasizing of S3 Select by AWS.

ddmf1y ago

Nice one, thanks, I'll have a look through this.

j / k navigate · click thread line to collapse

Show HN: GoSQL – A query engine in 319 LoC (opens in new tab)

(github.com)

81 pointsarchiewood1y ago9 comments

I've always been curious about how SQL engines actually work.

So I built a minimum viable SQL engine in Go.

- Supports CSV files as tables

- Supports SELECT, FROM, WHERE, LIMIT

It's very simple:

1. Parses query string

2. Converts it into an AST representation

3. Executes the query against the CSV

4. Returns the results

9 comments

stevekemp1y ago

Nice job.

You can see this post for the start of a guide in implementing something very similar "Writing a SQL database from scratch in Go":

https://notes.eatonphil.com/database-basics.html

(Use the tag "sql" to find the later parts. Sadly not linked directly from that first one.)

eatonphil1y ago

Thanks for mentioning! One of the most fun parts of this series I think is handling indexes on INSERT and actually making use of them based on (effectively) pattern matching on WHERE clauses.

> (Use the tag "sql" to find the later parts. Sadly not linked directly from that first one.)

There's a "Note" section right below that title that links to the other posts. :) I guess it is UX feedback that this was not obvious to spot.

archiewoodOP1y ago

Wow this is much more complete than mine, kudos.

iamcreasy1y ago

lainga1y ago

Try this talk about sqlite!

https://www.youtube.com/watch?v=ZSKLA81tBis

zh24081y ago

Feel like you can achieve something similar in duckdb? duckdb allows you to query local csv, parquet, and even remote ones?

archiewoodOP1y ago

Oh definitely, I love DuckDB.

This was a learning exercise for me as much as anything

coredog641y ago

This is very handy given the recent de-emphasizing of S3 Select by AWS.

ddmf1y ago

Nice one, thanks, I'll have a look through this.

j / k navigate · click thread line to collapse