What's New in Python 3.11? (opens in new tab)

Every other major programming language is cargo culting as much "functional programming" as it can bolt on to its chassis. Which is then WAY overused by a small number of self-styled "10x-ers", to dump a bunch of unmaintainable write-only code into your codebase before jumping ship and abandoning its support a few months later.

Go and Python code may have significant difference. But there is spiritual overlap in the sense that Go really encourages a straightforward imperative programming style, which makes its code more accessible to a wider range of developers.

Never understood this. I've been doing coding for a quarter century. There is a reason we moved away from manual error handling to exceptions. It's easier to reason about, it's less code, it's better.

The Go fans keep insisting that this error handling in Go is new and fresh. I looked at my Go code, I saw that 2/3 of it is basically checking if a function returned an error, and I moved on. I've done it this way - I am not going there again.

And no, it doesn't force you to handle errors. You can simply not check for errors, just as you would swallow an exception.

My impression of the Go culture is that of the hip kids discovering vinyl.

As my former boss used to say - "there is an easy way, and there is the cool way".

> And the error handling... Shudder.

As an anecdotal data point... I like Python and prefer it to Go for the most part. However, in the case of error handling, Go seems to get it right and Python does it very wrong. Exceptions are extremely confusing, I just can't wrap my head around them. I don't even accept, philosophically speaking, the concept of exception: there are no "exceptions" when running a program, only conditions that you dislike.

> I never understood how people consider go an alternative to python.

I work in both Python and Go. I love both languages. Here are a few reasons why I chose Go over Python for a lot of things these days:

Alot faster, self-contained executables instead of virtualenv, scales natively, no GIL, linear code in goroutines instead of asyncio.

Same here. This baffles me.

Especially considered the obligatory compilation step and lack of IPython.

The new type system goes pretty much against the idea of duck typing, I'd consider that a pretty big fundamental change. And it's pretty half baked on top (e.g. `a: int = "foo"` goes through the python interpreter without even a warning).

And while there it certainly does provide a lot of benefits, it does make the language look more and more like a huge pile of patchwork and afterthoughts. It neither feels simple nor consistent and the tooling is a mess as well, both overly complex (e.g. half a dozens tools to do basic type checking) and lacking really basic important features (e.g. easy way to compile ship able binaries).

Especially after all the pain of the python3 transition, it feels just frustrating how much of a mess it all is. The "only one way to do it" philosophy seems to have been thrown overboard a long while ago.

Yep. Glad to see Python evolve, become more expressive and permit stronger static analysis.

Reason? This time of the day? Localized entirely into this thread?

For me, type annotations has been the single biggest ergonomics improvement to come to python. It's a very simple type system compared to many other languages (definitely simpler than rust, haskell, scala, java, and c++, I find it easier than go's generics but I am not practiced). I don't see how they add complexity to the language. In fact I find it limiting if anything.

Walrus is...fine. I don't use it much, but I defo see how it's one more thing.

Python's biggest complexity, imho, is its almost unrivaled dynamicism. There's just too many degrees of freedom at times.

The import system continues to be a nuclear footgun. I consider myself a python expert with 15 years under my belt and today I was still fighting with imports in a pytest suite. Like wtf.

Dictionary merge is a weird one to single out to me, it's an existing operator (|) and functions the same way as a set union over the keys (which uses that operator) in concept, it's a very natural thing that doesn't feel like an expansion of the language, just behaviour you'd expect from the datatype.

I love types. But as first class citizens. I started in Pascal and C. And they’re one of the reasons I’m enjoying Go for the kind of work I did with Python. Usually, if it compiles, its right.

I’d be cool with a statically typed flavor that wouldn’t allow for the type gymnastics.

But that’s gross with the None situation where you’d have to fall back to magic values vs ‘nil’ or similar concepts. None was one of my favorite things back when other languages didn’t address it.

And a lot of my opinion is probably based on the fact that I came to the language in the late 90s from a Pascal/C background. Probably just getting to old :).

For me Python feels like an accelerating train, as long as you are already on and don't get off then its great but it gets harder and harder to catch up for everyone else.

I love stuff like 'match', the walrus operator, Typing, etc. but it definitely feels like a lot to grok and it will be increasingly hard for new programmers to read arbitrary python code without grokking all of the new stuff.

Go is for people that want to build that sort of stuff though. It makes it feel like you’re being productive and scratches that itch as an engineer.

But then yea, 1 line of python using an inbuilt function solves a 10 line go routine in a hackerrank.

How does that make the language half-baked? Go didn't have generics until recently which made utility functions like that complicated. C doesn't have them either.

Exactly.

Which is why making an already complex language even more difficult a very big mistake.

What modern mainstream language is smaller than Python? Ruby? Javascript? Go is surely smaller but then your library surface area as well as code since increase correspondingly.

They did a lot of improvements on recent releases. Now the reference implementation even includes a JIT compiler.

Web development in Python is a lot of fun nowadays and has scaled just fine (we serve a few billion requests/ month). FastAPI makes writing APIs easy as well (it powers https://internetdb.shodan.io) and for deployment check out uWSGI or uvicorn.

Actually my primary use is CLIs, integrations, and such. It used to be a great systems language (1.6-1.8, 2.x). But now I can’t rely on an OS having a reasonable version, or of libraries, so there’s the distribution problem.

It’s really only good for apps or things I (or other Python people) are going to run.

Part of it is that I don’t do analytics work. Now that PERL is niche, my use cases are no longer the target.

As somebody who's comfortable with Python and C++... ugh. Types are frankly kinda silly. They don't do anything, the language doesn't even ship a blessed typechecker, they don't speed up your code, they don't make it safe, they just add useless notation that some mutually-incompatible third-party tools grok. And the mutually-incompatible thing is what really rubs me wrong. They didn't sit down and define a bombproof type system; they just added an optional notation and let the community figure it out (like packaging). Years later, and the community has not figured it out (like packaging).

Compared to C++, where you can do incredible zero-overhead magic with the type system; for example duck-typing, once The Way, is now an absolute nightmare in Python's type system. Python typing is little but baggage and warts unless you've drunk the cool-aid and then you're horrified that people like me haven't.

(and, all that said, I'm quite excited about py3.11, it's great on performance and ergonomics)

> If you don't like the complexity of newer libraries and features, they're still optional aren't they?

Chances are you're not programming in a vacuum. These features will be used by third party libraries and you will have to read/understand this code.

Even in your in house codebase, there will always be co-workers that want to use these features.

The optionality of language features is not real. If they exist, they will be used, you will have to understand them, they will leak from other libraries, etc.

It’s not as much my code as reading my dependencies, troubleshooting others’ code, and the ecosystem sprawl. I keep up to date and pick and choose what I want.

A lot of these issues just aren’t there in C++, and you probably wouldn’t even try to share a codebase for something with someone who’s not another expert. …and you’re most likely working on something big.

The big thing that’s plagued Python since 3.x is the distribution problem. You can’t just hand someone a script and maybe another one to user-install some requirements. You need to BYO Python, dependencies, etc.

Things like types, library management, etc. are being handled by external tools. If I want to onboard someone to collaborate with me they really have to have my version of Python (so probably pyenv), Pipenv or Poetry, mypy or whatever, pytest or whatever, black, and have the same config for some of them.

The verbosity overhead of Go (or Nim, etc.) is worth the trade off to me for most of my uses of Python. I can just hand someone a binary and the dev just needs the language installed and maybe like goimports.

I’ve never been married to one language. I like to use them for their wheelhouse. I’ll still use Python for 1-offs or some things I can ship in a container, etc. that I’m only working by myself or with experienced Python people on.

A lot of this is that it’s really turning into more of an Analytics language. The output isn’t expected to be shared and everyone around you is a Python dev.

And I’m old. I started when Python was so pretty to my Pascal/C self compared to PERL (shudder). The documentation is still amazing for the language and standard library. (No Google, I want the actual docs.) And you could onboard almost anyone to work on it if they had any programming background because it was such a simple language.

The 2->3 shift was really only bad for complex legacy codebases and when you primarily deal with bytes vs Unicode. The latter is really painful anywhere. It didn’t bother me much. I usually just had to fix a few small things or it gave me a good excuse to kill something that should have been retired anyway.

> As someone who mainly sits in the C++ and not the Python space - why?

For the same reasons that you can't "just" use C++11 and ignore C++20.

The ecosystem moves forward, and you can't just sit behind no matter how much you would like to.

Yep! Ruby’s typing is wayyy more cumbersome than Python’s unfortunately

if it weren't

Sounds like you need type aliases, or simpler abstractions.

This is my problem. I’ve gotten to where the extra work to use a language that generates a binary is lower than the distribution problem with Python.

I’ve rewritten 3 CLIs because it’s too much of a pain for people to get them set up.

I still use it for myself to whip through a CSV, collate data from APIs. Sometimes a tiny API or web app (I really don’t like webdev) where I can deploy a container or pipeline deployment.

And I don’t have to use all the new bells and whistles. But so many do.

The type hinting system in particular is so much more painful to read (to me) than in a language with actual types.

even if I don't, I probably need to be able to understand and recognise them when other people do, in libraries/snippets etc that I depend on.

Not specifically a new feature, but I remember wasting a good bit of time when coming across a line that was essentially:

    return headers.get('x-foo') == settings.FOO_KEY is not None

and being extremely confused until discovering it was a use of comparison chaining[1] and was actually doing the right thing.

[1] https://docs.python.org/3.10/reference/expressions.html#comp...

I'd prefer that they had done something like unexec to allow loading some arbitrary set of modules into a bootstrapping image, then dumping a runnable executable that would launch with all those modules already present. The old Emacs approaches are now mostly busted, but there are alternatives.

Great, can't wait for some jupyterlab speedups

For computation, sure. But isn't the wasm/js to DOM boundary always going to be a bottleneck?

Maybe I'm being too negative and a faster browser API will be developed.

> No way to fully type args and kwargs

I think "parameter Specification Variables"[0] come pretty close to solving that no? I'd love to hear your take if you've used them before.

> Type semantics aren't defined so there are multiple competing interpretations

> The most popular type checker - MyPy - is full of bugs and deliberate unsoundness.

100% agreed. I hope Pyright "wins" as the dominant type checker, because like you said, it's MUCH better in every aspect.

> A depressingly huge part of the Python community doesn't get static typing so a ton of your dependencies won't have types.

100%. I have so many half hearted attempts at typing my dependencies![1]

> I definitely wouldn't choose Python if I had a choice.

At this point, I'd have to agree, except of course for data science/ML things.

[0] https://peps.python.org/pep-0612/

[1] https://github.com/davidatbu?tab=repositories&q=stubs&type=&...

> I genuinely think I like types in python better than TS!

That's a step further than I'm willing to go :D. For example, the other day, I was able to do

    settingToToggle: KeysWithTypeOf<SomeObject, boolean>

which (after properly defining `KeysWithTypeOf`) allowed me to specify that `settingToToggle` should be a string literal that is a key on `SomeObject` whose corresponding value is a boolean.

Typescript comes closest to fully letting me express my intentions statically and curtly, and while I hope Python's static typing will grow to be as expressive as Typescript, I think that day is still afar off.

What do you use for type-generated CLIs? I've been using Pydantic for config parsing and Cleo for CLI, but I don't yet have a good solution for hoisting/injection of CLI arguments alongside env/file configs.

I really want a declarative solution like Rust's Clap library, but I haven't yet found anything like it out there.

Thanks for linking this! And I (though not OP) totally agree with your points!

I hope that these things could be solved with future iterations. For example:

    Variadic protocols don't exist.

Hopefully they are added sometime.

    the syntax is awful and verbose.

Hopefully we can settle on allowing `1` instead of `Literal[1]` as a type (and other similar improvements).

    many operations like stacking are inexpressible.

These could be expressed if things like "multiplying"/"adding" literal numeric types would be supported.

The variadic generics PEP was partly motivated by the ML use-case (and took input from maintainers of numpy ..etc), so I hope future iterations will also improve the usage in the ML space.

Sure, there are of course runtime checks you can do (and I now do after having been bitten by this before), but that’s not necessarily better than having a static analyzer which can guarantee correctness without executing a single line of code.

Yes, the example @beisner gave here (the val loop crashing after the training is done) is kind of a bad example, and some frameworks (like Pytorch lightning) do do a "full workflow check" before going into training, but his overall point that variadic generics could massively impact dev/researcher productivity stands, I think.

Yes that’s what I thought everyone did…

It's an optional typechecking tool. Speaking as an ML researcher who has authored my fair share of linear algebra bugs due to broadcasting, being able to optionally typecheck specific critical regions of the codebase would be a killer feature.

I think they want Python to fit into every niche, like C++, which means a lot of knobs to turn

This is what you get when you missed a chance to namespace standard library many years ago. Should have been std.toml.

To avoid breaking exiting code that imports "toml", which is more common than "tomllib".

toml was already taken and no one could contact the author.

PEP 518 lists some reasons not to use configparser and I agree with them wholeheartedly.[1] Personally I don't want to use ill-defined formats, even for simple projects.

[1] https://peps.python.org/pep-0518/#configparser

Newer packaging-related PEPs require TOML, and many developer tools use TOML by choice, but a fresh install of Python couldn't actually read it. That required workarounds from tool and library maintainers, making it more complicated to support as many users' preferences and ways of working as possible.

However you prefer to work, your tools may now be easier to test and maintain. This change is good for everyone.

Ini files aren’t a standard though, so every language can handle them differently. As a result they often bite people using them in multiple languages in the foot

HCL is also oodles more complex than TOML since it has expression evaluation, whereas TOML is just a static format. It really depends on your use case and I think there's a lot of cases where TOML makes a lot more sense than HCL (e.g. metadata formats).

CUE is much better than HCL. I've been converting my more complex TF to CUE and then exporting as json for TF to consume.

What's HCL?

I only do this for configurations that aren't meant to be used by non-programmers. I don't want to have to explain for the billionth time that something like the path "C:\New Folder" isn't going to work as-written in a string literal, or that the name "type" is already taken.

Yeah. It's easy to write, easy to read. It takes zero effort to parse. Serializing as "python file" might be a bit awkward.

I wouldn't quite say they're gamechangers, but they're exactly what I happened to need.

Variadic generics will (I hope) be very nice for typing pre-2.0 SQLAlchemy code. `Query[*Ts]` or similar would be a natural type for many values from the ORM API, and hopefully allow expressing types for query transformations like adding or removing columns, joining to specific query shapes, adding arbitrary subqueries, etc.

The self-type might unblock a testing tool I haven't shipped because the DX isn't where I want it. IIRC: normal TypeVar binding plus the TypeVar hack for self-types, interacting with Mypy's particular way of detecting the type of a descriptor, interacting with the way the project uses mixins, sometimes produce types that technically aren't wrong but lead to spurious type errors in user-defined data fixtures. So I'm probably the only one on earth with this particular problem.

I don't know, it's been a while. Maybe it won't work. But the self-type hack is noisy anyway and getting rid of it is nice.

Shipping TOML in the standard library means I can drop a bunch of Tox gymnastics in projects that support pyproject.toml or TOML-based configuration in addition to the older standards, and now I'm not stuck either forcing certain TOML libraries on people or working out a way to plug in their choice. It's not that hard, but complexity multiplies, and now it's one less thing to deal with.

Why not just use multi processing?

It's implemented in 3.11. Documentation still hasn't been written for it yet though and that's why the issue is still open, https://github.com/python/cpython/issues/90908.

Actually, it doesn't. In C, you can use indentation as you see fit; braces allow you to define code blocks and scope as necessary. That gives you both worlds. Python apparently doesn't.

But stay smug.

Actually, it isn't. I looked up the significance of whitespace in Python and found examples of how it crippled the language: https://medium.com/star-gazers/pythons-significant-whitespac...