Systems like these should have an escape hatch of some sort. The key part is that it needs to be auditable.

Anything you do in CI should be possible outside of CI, at least by some subset of users.

This is why I personally like to use none of the CI features, and mostly use it like a shell script executor. Images? Stick to OS images only so that you can easily spin them up with `docker run` locally. Artifacts? Read and write them into S3 buckets and avoid the native artifact features.

This is obviously more difficult in the Github actions ecosystem, but I have mostly used Gitlab CI so far. My CI pipelines mostly look like this:

    image: ubuntu:24.04
    before_script:
        - apt-get install ...
    script:
        - ./ci/build-project.sh
    after_script:
        - ./ci/upload-build-artifacts.sh

I've run into a scenario where one of our rarely used environments needed a hotfix and the GitHub action we used to deploy there was broken. Was easy enough to translate GitHub action steps to shell scripting for a quick, manual deployment.