First, it let's you get started quickly and prototype. You can write unit tests, make sure everything is working as expected, then count queries and make sure you're being efficient with your SQL engine.
Second, and even more importantly, it's crucial in the definition of the app and the Schema. Thinking in high level "classes and objects" helps with the abstraction and the design of the apps. Even if you then default to raw SQU queries, thinking and building your model with class abstractions is huge.
Finally, there are some "tiny details" (but in my eyes, very important) that everybody oversees:
* Migrations: the way Django has designed migrations is just marvelous. We've migrated tons of data and changed the structure of our DB multiple times without any issues.
* Troubleshooting and "reporting": the ability to fire a quick shell, load a few models in Django's REPL, and do a few `.filters()` is for me key. On top of that, we add a Jupyter server connected to our read replica and we can do all sorts of reporting VERY QUICKLY. Not everybody needs a Data Lake :)
PS: We've ran Django sites accessed by millions of people per month. And we never had an issue with the ORM. Yes, sometimes I have to tell more junior devs that the gymnastics they're doing with `select_related` and `prefetch_related` can be more easily and effectively resolved with a query. But that's it. I'd say less than 1% of all the queries in our codebase have to be migrated to raw SQL.
It has happened that a dev in our team was populating an endpoint that used inheritance and when looking at the number of queries we were over 100.
But the solution in those cases is just OUTER JOINs and CASE. Especially since we use Postgres in the backend and it works so well.
So yes, there are some pitfalls, but they're greatly overshadowed by the advantages of the ORM.
It's like understanding memory allocation if you're writing c. It's just part of the job
I put that word in quotes because I don't want to imply it's a problem - it's just something you need to know how to handle.
Too often I come across Python projects that are hyped and when I dive into it I find it rather underwhelming to say it politely. Invariably it turns out that those people don´t know any other language than Python.
I see that as a general problem; Python is the language for beginners this day with an endless amount of tutorials, but it seems lots of those starters don´t get to have a look outside the Python world. I fear this is getting an even bigger problem because AI models are trained on a vast range of Python, not because it is better, but simply because it is the PHP of these days.
I don´t mean to imply that I know you have that "narrow profile".
I agree that the value in Python is for quick prototypes in case you know Python well. Outside of that (at the risk of a language war), I think one does better look into modern .net core or the JVM for a general purpose, high quality and highly efficient language/platform.
On the topic of ORM I think EF Core (.net core) and Doctrine (php) is strictly better.
> Too often I come across Python projects that are hyped and when I dive into it I find it rather underwhelming to say it politely. Invariably it turns out that those people don´t know any other language than Python.
To give a nice counterexample: Python's hypothesis library is really great, and compares favourably to its Haskell inspiration of QuickCheck.
Edit: That said, learning multiple programming languages (paradigms) is a very good idea. A reasonable dose of e.g. Java, .NET or C++ for example is great for understanding how programming languages and software should not be written. Learning modern JavaScript makes it obvious how Python really suffers from weak functional programming support, very bad performance and a horrible packaging system.
The widespread use of Python is a problem? People should learn java? Okay.
I'll note you don't talk about the usefulness of the projects themselves, only your opinion of what the code "looks like".
I use https://pypi.org/project/django-nplusone/ for instance. Sentry also warns of these by the way.
Actually it seems they are: https://code.djangoproject.com/ticket/22492#comment:11
.prefetch_related (for whole models) and annotate/Subquery (for fields or aggregates) have existed for many years, alongside a pile of aggregate functions which have existed forever and have improved.
Whether or not you use the tools given to you is a sign of developer quality and experience. You can easily avoid n+1 99% of the time but you have to appreciate the problem exists.
I think the Django project demanding some competence is okay.
Meanwhile Django's ORM just does the job perfectly fine in the 99% common case, and for the 1%, again 99% of the time it has enough escape hatches to solve the hard problems without making things excessively complicated. But $DEITY save you if you're in that 0.01%...
I don't know how people build websites in any speed without such a tool. I guess they just waste time duplicating effort on an admin UI or pay for one of those tools that can generate one from an API (meaning they have to also build an API).
It's such a massive time-saver I switched to django after a week or so.
And it encourages CRUD thinking instead of thinking about business processes and user experience
It's great for tiny/personal projects but in an organisation it can be a trap IMO
- the main reason to use django today is the admin UI
- and the only reason to use django ORM is the admin UI
It feels like much of the value in that is achieved even better by using a proper database client, say, JetBrains‘ database integration. I certainly wouldn’t let any business people touch a Django admin panel, or at least those I’ve seen built by our data guys. Way too technical.
And don’t even get me started on extensibility. I wanted to have like, an additional page to include a Grafana iframe plus a bunch of buttons. Something that would take me about five lines in Laravel, for example. Good luck even searching the documentation for this…
It means you can focus on your business logic while buying time to get you through the initial development.
Also, I believe your code for creating an empty "data" migration is missing the "makemigration" command itself.
The last time I needed to do that, I ended up crying "uncle" and writing manual SQL. I wasn't happy about doing it, but I was happy that the framework left me an escape hatch so that I could.
Thanks, I will update the data migration code!
Routes, form validation, REST API, templating (if you don't need react), auth, etc.
You'll probably wind up recreating a lot of ORM and surrounding functionality at lower quality
is it possible that the fact this article is written in 2006 simply makes it dated? it seems very catastrophizing but we've come a long way and are just more aware of how to work with or around the shortcomings and flaws of ORMs.
I've often written programs where I'm trying to encapsulate pure in memory state into business objects and run into the same type of issues people complain about with ORMs. programming is just hard, we don't have to be such dogmatists about it.
I don't think the article being from 2006 (or 2004) makes it dated (though it does have a date). I think it addresses a fundamental issue with ORMs that will always be there, which makes it a bit of a classic.
David Hang's Django ORM article has a bullet-point section on ORM cons, including "difficult debugging", "performance", "hides underlying SQL". Whereas Neward's article goes into depth on each of the following topics:
The Object-Relational Impedence Mismatch
The Object-to-Table Mapping Problem
The Schema-Ownership Conflict
The Dual-Schema Problem
Entity Identity Issues
The Data Retrieval Mechansim Concern
The Partial-Object Problem and the Load-Time Paradox
If you haven't read the original article, I highly recommend it. You'll learn a bit more about "Vietnam" (from a US perspective), and be better equipped to discuss and make decisions about ORMs afterward.
[My own opinion is that ORMs can be useful (I wouldn't say never use one), but that a programmer should be grounded in SQL and the relational model[2] first, so that they'll know when to, and when not to, use one.]
[1] https://web.archive.org/web/20220823105749/http://blogs.tedn...
> This eliminates the problem quite neatly, because if there are no objects, there is no impedance mismatch.
But:
> Integration of relational concepts into the languages. Developers simply accept that this is a problem that should be solved by the language, not by a library or framework.
> Over the last several years, however, interest in "scripting" languages with far stronger set and list support, like Ruby, has sparked the idea that perhaps another solution is appropriate: bring relational concepts (which, at heart, are set-based) into mainstream programming languages, making it easier to bridge the gap between "sets" and "objects".
I agree that bringing relations into your language is a good thing, but many languages are strong enough to do that as libraries, there's no need for baking support into your language. Just as hash-tables can be implemented as a library in most languages.
We have relations as a library in a Haskell dialect I was working with in a previous job. Worked like a charm, and no language support was needed. Python would also be flexible enough to use relations as a user-defined datatype. And so would many other languages.
Databases are the bottleneck your classic website. We choose to query these databases in a extremely high level language called SQL. This language is so high level that you need to come up with tricks and query analyzers in order to hack the high level query into something performant.
A better abstraction would be one that's a bit more similar to a standard programming language with an std that has query related operations/optimizers that can be composed so programmers can compose and choose query operations and avoid optimization issues that are hidden by high level languages like SQL.
We are unfortunately, sort of stuck with SQL (there are other options, but SQL remains popular because years of development has made it pretty good in spite of the fact that it's a poor initial design choice). This is already a historical mistake that we have to live with. Same with javascript (which has been paved over with typescript), same with CSS, etc. The web is full of this stuff. It's fine. My main problem is the ORM.
The ORM is just another layer of indirection. You already have a high level language you're dealing with, now you want to put Another High level language on top of it? ORMs are basically high level languages that compile into SQL.
What is the point? The ORM isn't actually making things easier because SQL is pretty much as high level of a language you can get outside of having an LLM translating direct english.
The point is OCD. Programmers find it jarring to work with SQL strings inside their beautiful language so they want to chop up a SQL string into web app language primitives that they can compose together. Query builders operate on the same concept but are better because they aren't as high level.
This is basically the main reason why Many programmers experience the strange counter intuitive phenomena about why ORMs actually makes things harder. You have to Hack SQL to optimize it. Now you have to hack another ORM on top of it in order to get it to compile it into the hacked query.
So having an ORM in place that is tightly integrated in Python and Django lets even the junior developer fresh from the bootcamp make changes to an existing application.
It’s not a pretty story, but in my opinion the reason for staggering layers of complexity is the ability to move quickly even without experts on the team.
What? You have to understand a language to write performant code in it. That’s not a hack, that’s basic competence.
The poster is not referring to understanding the language, he/she is referring to having to guess at how to structure the query in a way that increases the chances that a good plan is within the constrained search space of plans (due to it being a combinatorial problem and the optimizer has limited time and information).
The garbage collector is something you should not think about when programming and most programmers don't even need to know it exists. You only think about it when you need to seriously optimize things.
For SQL it's a bad design choice because of what SQL is and what SQL is targeting.
SQL is a high level abstraction that automatically determines a query plan based off of high level input. The problem here is that it can choose a bad query plan. So you need to "hack" the query in order to trick the query planner into doing what you want.
It's also targeting the slowest part of the stack: Non volatile memory and IO. The slowest part of the stack should be targeted with a zero cost abstractions to maintain speed while the fastest part of the stack you can use a language like python for your web app it's fine because databases are magnitudes slower.
Then I tried the new generation of typescript SQL query builders that automatically infer types from your database and provide end-to-end type-safe query results, even for highly complex queries. And since then I became convinced the answer isn’t trying to shoehorn classes on top of relational tables, but having a sufficiently integrated and capable query builder that provides properly typed results.
And yet using it was such a shit experience I switched permanently to writing SQL.
I see zero advantage to using an ORM over SQL and in fact see many downsides.
Don’t use an ORM just learn SQL. It’s faster, more direct, more powerful, easier to understand, allows you to make full use of the power of the database and your knowledge isn’t suddenly valueless when you go to a project with a different ORM.
Good ORMs make sure your data model is always aligned with the DB, and there is no way in hell a runtime error can occur. What's more, it automates the mindless data class/table matching for 99% of your use-cases.
Once you got a more sophisticated query, please use SQL. But for simple stuff, use ORM.
I’ve read a lot of criticisms of ORMs, as I’m sure everyone else has. Some of them are certainly valid criticisms that are immovable and just inherent in what an ORM tries to do. Some of them just seem to be caused by not very many ORMs being good, and the writer not having used one of the better ones.
Although over the years my code trends more and more towards `.rawSql` or whatever equivalent exists. Even for the simple stuff. It’s just so much easier than first thinking up my query then bending over backwards three times to make it fit into the ORM’s pet syntax.
Plus raw sql is so much easier to copypasta between different tools when you’re debugging.
And before you say “but sql injection!” – that’s what prepared statements/parametrized queries are for.
Once you get sufficiently familiar with some paradigm the training wheels can come off.
“Raw” SQL is already an abstraction. Over time all the implicit magic will get on your nerves. Trying to shoehorn two completely different worlds into one abstraction is not worth it: you get to learn today’s untransferable funky ORM syntax and idiosyncrasies while losing sight of the actual skill that matters long term which is SQL itself.
I concede however that handling of SQL, the field names, the relations, is annoying. But it’s core to the problem you are probably solving (some form of CRUD). Plumbing is annoying but as a plumber I’d say get used to it instead of wishing to be dancer.
I notice this in other aspects of my work as well. When I switched away trom desktop environment to terminal I had the same feeling. It’s easier, less hassle, less wonky abstractions, more direct. Completely counter to what popular culture is telling me.
But the team chose EF due to it supposedly being easier to integrate with whatever database the customer might be using and that seems like valid reasoning.
I started using Django on I think version 1.2 or 1.3 in 2011, back when it didn't have database migrations and you had to use a library like South for it. Even then, as an ORM/query language it was apparently better than what other languages have now.
That branch merged on May 1st 2006: https://www.djangoproject.com/weblog/2006/may/01/magicremova...
I've long found Django's commitment to not breaking too much at once inspiring. The release notes and upgrade guides are really solid.
If you started from Rails and ActiveRecord, you're probably not be very appreciative of Django's ORM.
Define the object, hook it to a view, use our custom permission class, done.
GET, POST, PATCH, DELETE all urls, filtering, pagination, search, ordering, complex multi-org multi-user object permissions handled automatically. Need custom logic? Just overwrite method.
It’s a productivity superpower, sure there’s a lot to learn but that’s the price you pay for the tools that make you hyper productive
SQLAlchemy is leagues above and beyond Django ORM. I can’t say I have nightmares from dealing with it, but it certainly was not pleasure. A bit too much magic here and there, not flexible enough and provokes too much bad practices. Re-defining save() methods to do a lot of work, anyone?
The best “orm” that I’ve ever used was not an ORM but a query builder — HoneySQL from Clojure. That one is fantastic. No ORM bullshit, no extra data loaded from DB, no accidental N+1 queries. Not reading docs on how to do a GROUP BY, everything’s just easy to write.
Frankly, we often use SQLAlchemy as just a query builder too. Makes life so much easier.
The lack of support until very recently (and it's still lacking) for views is the main knock.
I'm rewriting a large Django project in Java (quarkus + jooq), because it's at the point where I need a type system now, but it still has a place in my heart.
I had the "pleasure" to use Doctrine once. Never again !
All you need is the django toolbar so you can check their efficiency, then keep telling it to make them more efficient.
It's been particularly helpful for aggregations, subqueries etc. You may not think LLMs are there yet but my project is proof. Try it yourself.
Imagine an electrical engineer saying, “I don’t really know what a bridge rectifier is, but I plugged these things into the breadboard where ChatGPT told me, and the output signal looks correct.”
