https://github.com/charliermarsh/ruff
It’s literally 100 times faster, with comparable coverage to Flake8 plus dozens of plugins, automatic fixes, and very active development.
relevant section from my pyproject.toml
[tool.ruff]
line-length = 88
# pyflakes, pycodestyle, isort
select = ["F", "E", "W", "I001"]What do you mean by this? Are you indenting Python with tabs?
Though their `v0.0.X` versioning is very funny to me (https://0ver.org/).
replaced both flake8 and isort across all my projects
I would nitpick this. You build images not containers and since files are not copied by default there is more nuance here that the .dockerignore file makes builds faster by not including them in the build context.
That does ultimately prevent COPY directives from using them but it is these sorts of brief, slightly inaccurate summaries that mislead folks as they build understanding.
> slightly inaccurate Not entirely, I'm not sure the author even wanted to stress on this in the article. People won't learn docker from a python article about the same.
I absolutely let Black change code and see the value in Black that it does that so the devs do not have to spend time on manually formatting code.
Black shouldn't break anything (and hasn't broken anything for me in the years I used it) but in the unlikely case it does it, there's still pytests/unittests after that that should catch problems...
Even while it won’t break anything you want CI to be your safety net, flagging a local setup as being wrong is more valuable than magically autocorrecting it.
CI/CD has no business changing your code; it builds stuff using it, exactly as if commit such-and-such.
I liked black, though I was never satisfied with the fact that there was no way to normalize quotes to be single quotes: '. Shift keys are hard on your hands, so avoiding " makes a lot of sense to me. But there's the -S option that simply doesn't normalize quotes so it has never been a real issue.
However, this new project has a lot of typer functions with fairly long parameter lists (which correspond to command line arguments so they can't be broken up).
black reformats these into these weird blocks of uneven code that are very hard to read, particularly if you have comments.
Everyone is a fan of black; no one liked the result. :-/
I have a key in my editor to blacken individual files, but we don't have it as part of our CI. Perhaps next project again.
100% this. I also let Black auto-format code in the CI and commit these formats.
A lot of developers, intentionally or not, don't have commit hooks properly setup. If Black doesn't change the code in CI they need to spend another cycle manually fixing the issues that Black could have just fixed for them.
You're saying that there's a risk that Black could break your code when formatting? Well, so could developers and I'd trust a machine to be less error-prone.
There is nothing more frustrating than coming back from a coffee break only to find out that you have to rerun your CI check because of a trivial formatting issue.
Let black format code before it is checked in. Code should not be reformatted for CI or production, and bad formatting should either ALWAYS throw errors (no known defects allowed) or NEVER throw errors (if it passes tests & runs ship it). Consistency is the key.
Also, mypy has gotten really good in recent years and I can vouch that on projects that have typing I catch bugs much much sooner. Previously I would only catch bugs when unit testing, now they are much more commonly type errors.
The other thing typing does is allow for refactoring code. If anything, high code quality relates to the ability to refactor code confidently and typing helps this. Therefore I would put it at the top of the list above all the tooling presented (exception I agree with ci/cd)
There's zero harm in using list in private interfaces: I know I'm the only one passing the value, I know it is always a list.
As an argument type, Iterable is compatible with list, so it's benefits are minimal (with rare exceptions).
Lists are easier to inspect in a debugging session.
Iterable can be useful as return type, because it limits the interface.
Iterable is useful if you are actually making use of generators because of memory implications, but in this case you already know to use it, because your interfaces are incompatible with lists.
I can count on fingers of my hands when using Iterable instead of list actually made a difference.
Iterable is not compatible with list, but list is compatible with iterable. As the more general type, Iterable is better as an argument type unless you have a reason to force consumers to use lists. Even in private interfaces, I tend to prefer it, because I often end up wanting to pass something constructed on the fly, and creating an extra list for that rather than using a genexp just seems wasteful.
`list` might be but `List` isn't. Are you not defining the type of the contents of the list?
No. What allows you confident refactoring code are automated tests. I honestly can't understand why people are so obsessed about types, especially in languages like Python or Javascript.
By depending on interfaces/abstractions instead of specific cases you can refactor the interface and not break clients. It's very difficult to do this unless you have types.
This is something that Go is really good at and encourages but can be done with python/js on top of their type systems.
Types in Python feel like an added layer of confidence that my code is structured the way I expect it to be. PyCharm frequently catches incorrect argument types and other mistakes I've made while coding that would likely result in more time spent debugging. If you don't use any tools that leverage types you won't see any benefit.
It's a very powerful sanity check that lets me write correct code faster, avoiding stupid bugs that the unit tests will also, eventually, find.
And, to me, reading the code is much much nicer. Types provide additional context to what's going on, at first glance, so I don't have to try to guess what something is, based on its name:
results: list[SomeAPIResult] = some_api.get_results()
Typing facilitates automated testing; e.g., hypothesis can infer test strategies for type-annotated code.
[0]: https://github.com/agronholm/typeguard/
[1]: https://typeguard.readthedocs.io/en/latest/userguide.html#us...
These statements contradict themselves? List is too specific, and Sequence[item] is preferred. Sometimes you are dealing with a tuple, or a generator, and so it makes more sense to annotate that it is a generic iterable versus a concrete list.
> For example, you basically never care whether something is exactly of type list, you care about things like whether you can iterate over it or index into it. Yet the Python type-annotation ecosystem was strongly oriented around nominal typing (i.e., caring that something is exactly a list) from the beginning.
I'm saying that this quote is a straw man and that contrary to what is claimed in the quote, instead, the ecosystem would go with/recommend Iterable[Item] or Sequence[Item] and not List[Item] if applicable.
I think we both agree, not sure which part of my comment you think is contradictory.
As an argument type, Iterable is permissive (generic).
As a return type, Iterable is restrictive (specific).
This is an odd complaint. typing.Sequence[T] has been there since the first iteration of typing (3.5), for exactly that use case, along with many related collection types.
https://docs.python.org/3/library/typing.html
mypy isn’t perfect, but it’s sure better than making things up without any checks; you’re going to want it for all but the smallest projects.
Dynamically typed code is 1/3rd the size of statically typed code, that means that one developer who is using dynamic typing is equivalent to 3 developers using statically typed code via MyPy.
Since the code is 1/3rd of the size it contains 1/3rd of the bugs.
This is confirmed by all the studies that have been done on the topic.
If you use a static type checking with Python, you have increased your development time by 3 and your bug count by 3.
Static typing's advantage is that the code runs a lot faster but that's only true if the language itself is statically typed. So with Python you have just screwed up.
This is absolutely not true.
> Since the code is 1/3rd of the size it contains 1/3rd of the bugs.
That is made up and contrary to all empirical evidence I've ever collected.
I'd be curious if you have a source, but I doubt it.
You should use it where it makes sense, and not where it doesn’t. I haven’t used any of Ruby’s type checkers, but Python makes this easy enough; make what has a reason to be dynamic dynamic, and have static safety rails everywhere else.
(This is true with many “statically typed” languages that have dynamic escape hatches, too, not just traditionally “scripting” languages.)
Still it's a good low bar for testing. It's easy and rises code quality. I have very good results with coverage driving colleagues to write tests. And on code review we can discuss how to make tests more useful and robust and how to decrease number of mocks, etc.
Depending on the language and the particular project, my sweet spot for test coverage is between 30-70%, testing the tricky bits.
I've seen 100% code coverage with tests for all the getters and setters. These tests were not only 100% useless, they actively hindered any changes to the system.
You can have bad unittests which make the system worse and you would be better of without them. You can also have useless unittests with 100% coverage, which is pretty much the same as bad tests because more code means more bugs and more work. Unittests are also code after all.
The only thing you can say about a very low coverage is that you probably don't have good tests. That's not a very useful metric, since you likely already know that.
The metric 'coverage' is almost useless. Code coverage starts to be useful once you let go of it as a goal and ignore the total percentage number. I found it is very useful though if you can generate detailed reports on each line of code or better yet, each branch in the code, indicating whether that line or branch is tested. Eyeball all the lines which don't have tests and ask yourself: would it be useful to add a test exercising this codepath? How do I make sure it works and what cases can I think of that could go wrong? This doesn't automatically lead to good tests, but it helps you spot where you should focus your testing efforts.
Code coverage is a good tool to help think of test cases, as a metric for the total codebase it is nearly useless.
It's a red flag to blame high coverage for fragile tests. Use narrow public component interfaces to reach code parts and you simultaneously gain robust tests which can be used during refactoring and you can be guided by coverage to generate test cases. Bob Martin has a great article: https://blog.cleancoder.com/uncle-bob/2017/10/03/TestContrav...
Absolutely not. This leads to testing being invasive and driving the design of your software, usually at the cost of something else (like readability). Testing is a tool, you can't let it turn into a goal.
> Testing is a tool, you can't let it turn into a goal.
Yep, and I use testing as a tool to be sure we ship quality code. It's 2x important for our case, we don't have control on hosts where our product is run and 100% coverage was a salvation. We even start to ship new versions without any manual QA.
> This is the first in hopefully a series of posts I intend to write about how to build/manage/deploy/etc. Python applications in as boring a way as possible.
It's a riff on Boring Technology, see https://boringtechnology.club/
Terrible advice not to use type hints and this reason makes no sense. There's already pretty good support for Sequence and Iterable and so on, and if you run into a place where you really can't write down the types (e.g. kwargs, which a lot of Python programmers abuse), then you can use Any.
Blows my mind how allergic Python programmers are to static typing despite the huge and obvious benefits.
It's true that Python's static typing does suck balls compared to most languages, but they're still a gazillion times better than nothing, and most of the reason they suck so much is that so many Python developers don't use them!
Black formats things differently depending on the version. So a project with 2 developers, one running arch and one running ubuntu, will get formatted back and forth.
isort's completely random… For example the latest version I tried decided to alphabetically sort all the imports, regardless if they are part of standard library or 3rd party. This is a big change of behaviour from what it was doing before.
All those big changes introduce commits that make git bisect generally slower. Which might be awful if you also have some C code to recompile at every step of bisecting.
Then add black as part of your environment with an specific version...
Reformatting the whole code every version isn't so good. It's also very slow.
The further you get away from the project folder the more likely each developer is to have a different environment.
Why? It is expected for the thing to run on different python versions and different setups… what's the point of forcing developers to a uniformity that will not exist?
It's actually better to NOT have this uniformity, so issues can get fixed before the end users complain about them.
Any team of developers who aren't using the exact same environment are going to run into conflicts.
At the very least, there must be a CI job that runs quality gates in a single environment in a PR and refuses to merge until the code is correct. The simplest way is to just fail the build if the job results in modified code, which leaves it to the dev to "get things right". Or you could have the job do the rewriting for simplicity. Just assuming the devs did things the right way before shipping their code is literally problems waiting to happen.
To avoid CI being a bottleneck, the devs should be developing using the same environment as the CI qualify gates (or just running them locally before pushing) with the same environment. The two simple ways to do this are a Docker image or a VM. People who hate that ("kids today and their Docker! get off my lawn!!") could theoretically use pyenv or poetry to install exact versions of all the Python stuff, but different system deps would still lead to problems.
You've never done any open source development I guess?
Do you think all the kernel developers run the same distribution, the same IDE, the same compiler version? LOL.
Same applies for most open source projects.
Bisection search is log2(n) so doubling the number of commits should only add one more bisection step, yes?
> Which might be awful if you also have some C code to recompile at every step of bisecting.
That reminds me, I've got to try out ccache (https://ccache.dev/ ) for my project. My full compile is one minute, but the three files that take longest to compile rarely change.
And testing 1 extra step could only add a 1 hour build more, yes?
This is not isort! isort has never done that. And it has a formatting guarantee across the major versions that it actively tests against projects online that use it on every single commit to the repository: https://pycqa.github.io/isort/docs/major_releases/release_po...
You should never develop using the system Python interpreter. I recommend pyenv [0] to manage the installed interpreters, with a virtual environment for the actual dependencies.
Yes yes… never ever make the software run in a realistic scenario! You might end up finding some bugs and that would be bad! (I'm being sarcastic)
use pre-commit https://pre-commit.com/ so that everyone is on the same version for commits.
Not using a formatter at all is clearly worse than either option.
why?
Do you hate terse diffs in git?
https://github.com/cjolowicz/cookiecutter-hypermodern-python
I would go so far as to say that the hypermodern template, nomenclature aside, is strictly better than the recommendations that the OP put forward both here and in the previous essay on dependency management. Poetry and ruff, for instance, are both very good tools — and I can understand _not_ recommending them for one reason or another but to not even mention them strikes me as worrisome.
My concern is a) It needs to be reliable (don't wanna spend a ton of time chasing bugs later on) b) How can I write the actual code better? I see what pro devs write and they use smarter language features or better organization of the code itself that makes it faster and reliable, I wish I could learn that explicitly somewhere.
I mean, just the 2.7->3.0 jump was big for me because since I don't code regularly that meant googling errors a lot basically. Even now, I dread new python versions because some dependency would start using those features and that means I have to use venv to get that small script to work and then figure out how to troubleshoot bugs in that other lib's code with the new feature so I can do a PR for them.
I love python but this is exactly why I prioritize languages that don't churn out new drastic features quickly. Those are just not suitable for people whose day job is not coding and migrating to new versions, supporting code bases, messing with build systems, unit tests, qa,ci,etc... coding is a tool for me, not the centerpiece of all I do. But python is still great despite all that.
What do you mean by "drastic" features "quickly"? Python releases new version once a year these days, and upgrading our Django-based source code with 150 dependencies from 3.4 to 3.11 literally meant switching out the python version in our CI configuration and README.rst every once in a while, no code changes were necessary for any of those jumps...
Our developer README also contains a guide how to set-up and use pyenv and it's virtualenv plugin which makes installing new python versions and managing virtualenvs easy, just pyenv install, pyenv virtualenv, pyenv local, and your shell automatically uses the correct virtualenv whenever you're anywhere inside your project folder...
jumping to python3 was big, but you had plenty of time to prepare for that and plenty of good utilities to make the jump easier (2to3, six, ...). python2.7 itself was released 18 months after python3.0, and by the time python2.7's support ended, python3.8 was already out...
Second, yes, all you have to do is switch out the python version to upgrade but let's say you start using f-strings that means all of your users (doesn't apply to django since it is server software) have to upgrade to the right python version including all the deps. But what if your project is a library? That means all other libraries need to use the same or greater python version but what if your distro doesn't yet support the very latesr python version? It's such a nightmare.
New versions should come out no more often than every 3-4 years imho and even then every effort should be made to have those features backward compatible like have a tool that will degrade scripts to be usable on a previous language version.
If a dependency breaks compatibility with earlier Python versions because the author wants to use a fancy new feature is not really the fault of Python, is it? Library authors should target the earliest supported Python version they can.
Being backwards compatible (at which Python has been doing a good job since the 2->3 fiasco) is one thing, but trying to be forwards compatible is something else.
Are you suggesting that Python developers should only ship bug fixes so that Python 3.0 can still run code written for Python 3.11?
In 3.8 someone decided that they didn't like the way people were excepting the Exception for cancelled asycnio tasks. So they changed the cancelled task exception to inherit from base exception instead of exception. This meant a bunch of well used libraries immediately had a load of subtle bugs that in normal operation just didn't happen. I can't remember the exact details but I think when the bug did happen the task queue would just continue to grow until we ran out of memory.
This change wasn't a bug fix, more an optimization or an attempt to get people to code a certain way.
I'm all in favour of bug fixes, but Devs shouldn't have to worry about minor upgrades breaking everything.
I have a library… most downloaded version is 3 years old. The newer versions are massively faster but nobody uses them.
About typings: I agree the eco-system is not mature enough, especially for some frameworks such as Django, but the effort is still valuable and in many cases the static analysis provided by mypy is more useful than not using it at all. So I would suggest to try do your best to make it work.
When python converges on consistent typing across its extended numpy and pandas ecosystem, I believe we will be able to move towards a fully JIT'd language.
Unless they actually go ahead with the deferred evaluation of types (PEP 563), make all types strings at runtime and make it impossible to know which type they actually are. :)
But they will probably not: https://discuss.python.org/t/type-annotations-pep-649-and-pe...
But it could be a breaking change in the language. As it is, I can run this "a: str = 3" and it will work.
On Ubuntu and Windows I use Poetry [0], and it works, although it has (had?) some quirks during the installation on Windows. I liked its portability and lockfile format though.
A few years ago I used conda [1], which was nice because it came batteries included especially for Deep Learning stuff. I switched because it felt way to heavy for porting scripts and small applications to constrained devices like a Raspberry Pi.
And then there are also Docker Images, which I use if I want to give an application to somebody that "just works".
What's your method of choice?
Agreed. I like docker images for smallish portable scripts. At home I can develop on my Mac and port it to a Raspberry PI or another x86 Windows/Linux box.
Planning on running a docker swarm with a few Pi’s to see how it works.