If you think having something like pipenv or gcc in your image is a "perfect way" of doing things, you're sorely mistaken.
Neither of those are needed. Use multi stage docker files to avoid such unnecessary and potentially dangerous dependencies.
In addition, why do you need to disable byte code writing? You're unnecessarily disabling optimizations done by python.