undefined | Better HN

0 pointsschmooser5y ago0 comments

I tried it, doesn’t work - Lambda resource needs hash sum to compare against previous deployment to trigger updates of the source bundle, and Terraform needs a file to be present during plan stage. With null_resource the file is created after plan, during apply. To workaround this I tried to provide something else to bundle hash sum (like hash sum of source files, not bundle), but the value that TF keeps in the state is the one returned from AWS Lambda API, not the one you supply it; so it causes resource update on every apply, this is not what I wanted.

Your comment made me think of trying to skip bothering with Lambda’ hash sums and use custom refresh triggers instead, initiated by null_resource. Will do after holidays.

0 comments

talideon5y ago

It can work. I wrote a script in Python to build stable zip files. The main thing is to ensure any timestamps are reset to zero and the files are added in a fixed order. It also filters out any unwanted files and directories to minimise the size of the bundle. I have a wrapper around it that generates the hash and uses pip3 (as I use Python for my lambdas) to download the dependencies and build the directory hierarchy for the lambda/layer, runs the stable zip script, and returns the path of the bundle and the hash.

This didn't take long to write, and reduced the amount of churn we had with our deploys. We had massive problems with one particular set of lambdas due to the sheer amount of code (mostly unavoidable dependencies, but shared, so they could go in a layer), and our deployment times plummeted to practically nothing after I knocked this together.

I'm not sure I can share the code as it's something I wrote for work, but it ought to be simple to recreate from the description above.

felixhuttmann5y ago

Yes. If we use a null_resource that has the hashes of the source code files as a trigger, then in the `local-exec` provisioner of the null_resource, we can run the build. The build can also be run remotely (we use google cloud build) to be independent of the developer's machine architecture and operating system, which is important for native dependencies. Terraform will not re-run the null resource provisioner so long as the source code does not change, there is no need for a reproducible build.

talideon5y ago

For various reasons (mainly auditing purposes, but it also reduces any incidental infrastructure churn, and makes it easier to guarantee a rollback happened as expected), we need to ensure reproducibility, so it's a bit more important for us that we guarantee the artifacts produced are exactly what we expect.

nijave5y ago

Instead of using source_code_hash you can push your code to s3 with the hash as the filename and update the lambda to point at the new file

Terraform can manage uploading objects to s3

It also seems a bit strange to have Terraform do the packaging. We do that in CI for most of our lambdas to ensure the test suite runs, linting, etc then it creates a zip at the end and pushes to S3

The only ones Terraform deploys directly are fairly trivial Python API "glue" lambdas

smetj5y ago

After reading this comment I decided to write a small blog how you could tackle this nevertheless https://smetj.net/deploying_aws_lambdas_with_terraform.html

j / k navigate · click thread line to collapse

0 pointsschmooser5y ago0 comments

Your comment made me think of trying to skip bothering with Lambda’ hash sums and use custom refresh triggers instead, initiated by null_resource. Will do after holidays.

0 comments

talideon5y ago

I'm not sure I can share the code as it's something I wrote for work, but it ought to be simple to recreate from the description above.

felixhuttmann5y ago

talideon5y ago

nijave5y ago

Instead of using source_code_hash you can push your code to s3 with the hash as the filename and update the lambda to point at the new file

Terraform can manage uploading objects to s3

It also seems a bit strange to have Terraform do the packaging. We do that in CI for most of our lambdas to ensure the test suite runs, linting, etc then it creates a zip at the end and pushes to S3

The only ones Terraform deploys directly are fairly trivial Python API "glue" lambdas

smetj5y ago

After reading this comment I decided to write a small blog how you could tackle this nevertheless https://smetj.net/deploying_aws_lambdas_with_terraform.html

j / k navigate · click thread line to collapse