Metaprogramming in Python (opens in new tab)

I've run across people in three different companies who decided to use them for no good reason at all.

I was only around for one of them while he did it but it seemed pretty clear that he was insecure and trying to prove that he was a more experienced programmer by using the most advanced features.

I call it "the advanced beginner metaclass trap".

I think you mean `__init_subclass__`

I personally consider code generation an anti-pattern in Python. With its dynamic nature and the two-step execution model, Python is essentially its own macro language, and any generation can be done at runtime.

How do you deal with changes over time?

Like let's say you want to add a new required property to every type of something... like you want every encoding to have an accepts_tainted_value property. You don't want a default because you want to be forced to think through every case.

Can you regenerate the code? If so, are you keeping the "real" object specification in some other structure? Or do you just change a superclass to make the property required then fix the generated source based on the errors you get?

Yes, also for anything else than subclass registration, there are things like the descriptor protocol or class decorators that are likely to be easier to use, debug and understand.

And metaclasses can be any callable, not just class. It's usually better to use a simple function.

Python is easy, not because it prevent people from doing complicated things, but because with only a few simple things, you are productive.

Most python programmer use 10% of the language, the same 10% that is described in most tutorial, because that's enough, and they don't even know the rest exist.

This makes for a very smooth, but long learning curve which allow you to enjoy python in the early years, yet keep getting a kick out of it after 15 years.

I hear a lot of people raising the same kind of concern you do in comments, but in the field, I never do with people actually using the language.

> It is shocking how much vodoo magic there is in a programming language marketed as "easy to learn and use".

But that's exactly what that expression means?

A car with an automatic transmission is "easy to learn and use", but an automatic transmission is way more complicated to construct than a manual one.

Ran into this face-first, just yesterday. I'm converting a rather old, rather big API (~400 endpoints) from Python 2.7 into 3, and apparently in Python 2, _hasattr(something, someattribute)_ just returns False if attribute access throws an exception! Specifically, if you have

  class A():
    @property
    def a(self): 
      raise Exception('go away')

  a = A()
 

then hasattr(a, 'a') will return False in Python 2.7 (and throw as expected in Python3)

A true WTF moment, the tests and the API have been slightly broken for years, without anyone noticing.

What an odd thing to say. This article is not for beginner Python programmers, and you don’t need to know anything in this article to use Python. I know people who refuse to use decorators because they seem difficult, and yet they are able to make very useful applications.

I’ve been writing Python professionally for over 20 years, and I’ve needed to use the things here once. It was highly encapsulated inside one module of an ORM-type project, where other code wanted to use it without having to know anything about its implementation details.

You can do these things in the extremely rare cases that you must, but other than that you shouldn’t.

For a language that is actually "easy to learn and use", look at Lua[1].

It's so easy to learn & use that children manage to be productive with it (the Roblox community).

It's still powerful though. With few enough primitives to remain simple & understandable, it has just enough primitives to build anything. People have written web frameworks, window managers and video games in Lua.

It's got less pitfalls, "voodoo magic" and advanced weirdness than any other language I know. You can learn 90% of the language from its Wikipedia page. Every time I encounter some of Python's internal weirdness, I wish Lua had been the scripting language of the 2000s instead. Unfortunately Python has a vast collection of libraries available that keep us using it instead.

[1]https://www.lua.org/pil/contents.html

That's not uncommon, when you make something easy* to learn and use you obfuscate certain things that are more "advanced" to streamline the onboarding experience, and then when you want to break out of the streamlining you gotta learn to circumvent those "stops" somewhat, which often end up being pretty non ergonomic. It compounds as well because the ones developing the language will have a bias towards what it currently offers and the "unorthodox" side keeps getting set aside.

*and by easy there I mean streamlined and more intuitive on most expected "common" tasks than the alternatives

Most Python programmers never need to worry about the "voodoo magic". But for writing tools and libraries that can nicely encapsulate a desired behavior and make it simple for other programmers, Python's "voodoo magic" is great.

You consider python full of vodoo magic? Go learn guile for a week and let me know what you think then :-)

Thanks for pointing it out

This is why any CS curriculum worth its salt should provide a summary of / introduction to Lisp and Smalltalk at or very near the start, so that in the second and third year of study or shortly after graduation, any encounters with so-called "voodoo" will elicit an "ah yes, sounds like CLOS MOP" response or something similar, rather than wide-eyed excitement leading to uninformed evangelism.

(Mine didn't have it, and I've been ruing it, and compensating for it, ever since.)

Small typos everywhere. Init functions without the dunders, move_ahead in an abstract class then moveahead later.

The example shows that django models uses metaclasses heavily. So I'm assuming if you needed to extend django models, you'd have to do it via metaclasses.

Pydantic uses metaprogramming to generate runtime validations according to the field type annotation.

The article however, does not show anything comparable to lispy metaprogramming.

I do not see a single new keyword being defined. I do not see anything changing the order of evaluation. Only to give 2 examples.

Recently I implemented a new kind of "define" in a Scheme, which allows for specifying contracts for a function. I don't think such a thing is possible using the tools shown in the article. As such, the things shown in the article are not really that meta, but rather parts of the Python language already. It is not like it is adding anything to Python, which was not there. It is just usage of some concepts Python already has. This is conceptually different from transforming source code to other source code, which then creates a new concept in the language itself. It is not like I can mold Python into whatever I want, but rather stay in the corset of the language facilities. Which of course is not as nice as lispy languages with good macro system, when it gets to creating DSLs and other conveniences.

So overall I am not so blown away by the article.

> Python’s way might even be better

Python recently got a form of structural pattern matching.

Can that be removed from Python, and then reimplemented only in Python? I mean other than by writing a .py file to .py file text filter?

Can it be backported to a prior version of Python?

Python's release history is full of "can't use this syntax if you're not on at least x.y.z version"; it continuously proves that it has it doesn't have metaprogramming on a level that could be used to develop the language. When the purveyors of Python decide to add some new syntax and semantics, they eschew Python metaprogramming and dive straight into C.