Jenkins is not the ideal place. We take some scripts and create pipelines that just invoke them. Some of these are configured to be scheduled jobs. We have separate declarative pipelines for each one.
But this isn’t good, we have Airflow and that’s where these things should be. Their stay inside Jenkins is technical debt, but at least is much better than not having it automated. The advantage is that you made an ad-hoc process less unstructured, you can add post-failure alarms, you don’t have to worry about long running scripts being interrupted by shabby notebooks, you don’t need to spread production credentials to every developer and so on.
