HRT's Python fork: Leveraging PEP 690 for faster imports (opens in new tab)

But yes, like you I had a great experience

I was quite impressed by the interviews, mostly for their pragmatism and skill-fitting. The programming interview wasn’t LC, it was “can you use a language (preferably Python) to parse a CSV and get useful information out of it,” because that’s the skill level the team needs. On the other hand, the Linux and DB interviews were quite in-depth, because again, the team needs those skills.

10/10 would interview again if I’m ever near an office again.

Its a finance firm - i.e scam firm. "We have a fancy trading algorithm that statistically is never going to outperform just buying VOO and holding it, but the thing is if you get lucky, it could".

Scammers are not tech people. And its pretty from their post.

> In Python, imports occur at runtime. For each imported name, the interpreter must find, load, and evaluate the contents of a corresponding module. This process gets dramatically slower for large modules, modules on distributed file systems, modules with slow side-effects (code that runs during evaluation), modules with many transitive imports, and C/C++ extension modules with many library dependencies.

As they should.

The idea that when you type something in the code and then the interpreter just doesn't execute it is how you end up with Java like services, where you have dependency injection chains that are so massive that when the first time everything has to get lazily injected the code takes a massive amount of time to run. Then you have to go figure out where is the initialization code that slows everything down, and start figuring out how to modify your code to make that load first, which leads to a mess.

If your python module takes a long time to load, this is a module problem. There is a reason why you can import submodules of modules directly, and overall the __init__.py in the module shouldn't import all the submodules by default. Structure your modules so they don't do massive initialization routines and problem solved.

Furthermore, because of pythons dynamic nature, you can do run time imports, including imports in functions. In use, whether you import something up at the top and it gets lazily loaded or you import something right when you have to use it has absolutely no difference other than code syntax, and the latter is actually better because you can see what is going on rather than the lazy loading being hidden away in the interpreter.

Or if you really care, you can implement lazy work process inside the modules, so when you import them and use them the first time it works exactly like lazy imports.

To basically spend time building a new interpreter with lazy loading just to be able to have all your import statements up at the top just screams that those devs prefer ideology over practicality.

Exactly this. There must be zero side effects at module import time, not just for load times, but because the order of such effects is 1) undefined, 2) heavily dependent on a import protocol implementation, and 3) poses safety and security nightmares that Python devs don't seem to care much about until bad things happen at the most inconvenient time possible.

> Of course that doesn't solve the overhead of finding the modules, but that could be optimized without lazy import, for example by having a way to pre-compute the module locations at install time.

1) opt for https://docs.python.org/3/reference/import.html#replacing-th...

2) pre-compute everything in CI by using a solution from (1) and doing universal toplevel import of the entire Python monorepo (safe, given no side effects).

3) This step can be used to scan all toplevel definitions too, to gather extra code meta useful for various dynamic dispatch at runtime without complex lookups. See for example: https://docs.pylonsproject.org/projects/venusian/en/latest/i...

3) put the result of (2) and (3) as a machine-readable dump, read by (1) as the alternative optimised loading branch.

4) deploy (3) together with your program.

    import argparse
    parser = argparse.ArgumentParser()
    parser.parse_args()
    import requests

The lazy import approach was pioneered in Mercurial I believe, where it cut down startup times by 3x.

It's dirty, but speeds things up vs putting all imports at the top.

Import-time side effects are definitely nasty though and I wonder what the implications on all downstream code would be. Perhaps a lazy import keyword is a better way forward.

A PEP is very much welcome, but using lazy import libraries is a fairly common, very old, method of speeding things up. My pre PEP 690 code looks like this:

    import typing
    from lazy import LazyImport

    member = LazyImport('my_module.subpackage', 'member')
    member1, member2, = LazyImport('my_module', 'member1', 'member2')
    
    if typing.TYPE_CHECKING:
        # normal import, for linter/IDE/navigation. 
        from my_module.subpackage import member
        from my_module import member1, member2