S3 as a Git remote and LFS server (opens in new tab)

> there are SO MANY s3 storage implementations

I suppose given this is under the AWS Labs org, they don’t really care about non-AWS S3 implementations.

Do you mean boto (the python SDK for AWS)?

EDIT: They probably do not, I'm guessing they mean https://docs.getmoto.org/en/latest/index.html ?

Unfortunately there's been a few vulnerability since that old Minio release. For something you expose to users, it's a problem.

Client side is, while interesting, of limited use as every CI and similar tool won't work this. This seems like a sort of automation of wormhole which I guess is neat https://github.com/cxw42/git-tools/blob/master/wormhole

That 128KB only applies to non-standard S3 storage tiers (glacier, infrequent access, one zone, etc)

S3 standard, which is likely what people would use for git storage, doesn't have that minimum file size charge.

See the asterisk sections in https://aws.amazon.com/s3/pricing/

Looks like it uses bundles rather than raw files: https://github.com/awslabs/git-remote-s3?tab=readme-ov-file#...

also the puts are 5x as expensive as the get operations

There is also git-annex, which supports S3 as well as a bunch of other storage backends (and it is very easy to implement your own, it just has to loosely resemble a key-value store). Git-annex can use any of its special remotes as git remotes, like what the presented tool does for just S3.

Also worth checking out https://github.com/jasonwhite/rudolfs

Been using it to store datasets via lfs. Written in rust and has been very reliable.

Dvc is great tool!

S3 recently got conditional writes and you can use do locking entirely in S3 - I don't think they are using this though. Must be too recent an addition.

Google Cloud Storage is good enough to implement locks all by itself: https://reddit.com/r/golang/comments/t52d4f/gmutex_a_global_...

Doesn't S3 provide primitives to do the same? At least since moving to strong read-after-write consistency?

PS: I wrote the above package. Happy to answer questions about it.

I thank goodness I have access to a non-stupid Terraform state provider[1] so I've never tried that S3+dynamodb setup but, if I understand the situation correctly, introducing Yet Another AWS Service ™ into this mix would mandate that callers also be given a `dynamo:WriteSomething` IAM perm, which is actually different from S3 in that in S3 one can -- at their discretion -- set the policies on the bucket such that it would work without any explicit caller IAM

1: https://docs.gitlab.com/ee/user/infrastructure/iac/terraform...

In what use case it can be interesting alternativd?

Limited access control (e.g. CI pass required), so not very useful for end users. For machine-to-machine it's an additional layer of abstraction when a regular tarball is fine.

Looks like it uses bundles, which handle incremental changes: https://github.com/awslabs/git-remote-s3?tab=readme-ov-file#...

I think this is more about storing the entire repository on s3, not just large files as git-lfs and git-annex are usually concerned with. But coincidentally, git-annex somewhat recently got the feature to use any of its special remotes as normal git remotes (https://git-annex.branchable.com/git-remote-annex/), including s3, webdav, anything that rclone supports, and a few more.

FWIW, their helpers make things pretty cheap to create new Formula by yourself

    $ brew create --python --set-license Apache-2 https://github.com/awslabs/git-remote-s3/archive/refs/tags/v0.1.19.tar.gz
    Formula name [git-remote-s3]:
    ==> Downloading https://github.com/awslabs/git-remote-s3/archive/refs/tags/v0.1.19.tar.gz
    ==> Downloading from https://codeload.github.com/awslabs/git-remote-s3/tar.gz/refs/tags/v0.1.19
    ##O=-#   #
    Warning: Cannot verify integrity of '84b0a9a6936ebc07a39f123a3e85cd23d7458c876ac5f42e9f3ffb027dcb3a0f--git-remote-s3-0.1.19.tar.gz'.
    No checksum was provided.
    For your reference, the checksum is:
      sha256 "3faa1f9534c4ef2ec130fac2df61428d4f0a525efb88ebe074db712b8fd2063b"
    ==> Retrieving PyPI dependencies for "https://github.com/awslabs/git-remote-s3/archive/refs/tags/v0.1.19.tar.gz"...
    ==> Retrieving PyPI dependencies for excluded ""...
    ==> Getting PyPI info for "boto3==1.35.44"
    ==> Getting PyPI info for "botocore==1.35.44"
    ==> Excluding "git-remote-s3==0.1.19"
    ==> Getting PyPI info for "jmespath==1.0.1"
    ==> Getting PyPI info for "python-dateutil==2.9.0.post0"
    ==> Getting PyPI info for "s3transfer==0.10.3"
    ==> Getting PyPI info for "six==1.16.0"
    ==> Getting PyPI info for "urllib3==2.2.3"
    ==> Updating resource blocks
    Please run the following command before submitting:
      HOMEBREW_NO_INSTALL_FROM_API=1 brew audit --new git-remote-s3
    Editing /usr/local/Homebrew/Library/Taps/homebrew/homebrew-core/Formula/g/git-remote-s3.rb

They also support building from git directly, if you want to track non-tagged releases (see the "--head" option to create)