-
Notifications
You must be signed in to change notification settings - Fork 4.3k
cloudformation package is always generating a new zip #3131
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Comments
Could you elaborate a little more on why it is an issue that a new zip is formed for every |
Imagine a scenario where you have a cloudformation file with more than one Lambda declared. I build both of them, which generates two binaries fn1 and fn2. I run cloudformation package, and it generates 2 zip files and send them to S3. One week later, I change the fn1 function, but not the fn2. My CI builds both of them, but only the first has a different MD5 (the second has the same MD5 as before). The problem here is the package command will generate a new zip for the second one too, even if the file did not changed, which causes all my cloudformation declared functions to be deployed. |
I'm having the same issue with Python code. Every time I run |
@rizidoro Can you download the zip files from S3, unzip them locally and diff them? Turns out I had one file which was actually different, because it included a "generated at" date which was being updated everytime I built the CloudFormation script |
You also need to check for timestamp differences amongst the files |
That is exactly the issue. If timestamps on files are different in the zip file, even if they are the same contents, the md5 is different. In the case of scripting languages, this is probably not an issue. However with Go, each time you run This is especially troublesome if you are trying to use CodePipeline and CodeBuild (see https://docs.aws.amazon.com/lambda/latest/dg/automating-deployment.html) because no matter what, Perhaps |
@jmassara exactly the problem I'm facing right now. The final binary |
@rizidoro Yes. This is a bug with |
I have the same issue. Have a CodeCommit repo with a sam.yml containing multiple lambdas. When from my VM i use aws cli on the promt 2 times after each other, the frist will upload a .ip for every lambda. The seconds one does nothing because nothing changed = correct. But... doing exactly the same from CodePipeline , CodeBuild (aws cloudform package bla bla) it does not work. You can trigger the pipeline with "Release Change" without needing a commit which will trigger the pipeline. It start a aws cli docker for CodeBuild, gets the input sources from S3 and unzips them. Calls cloudformation package which DOES reupload unchanged code for every lambda causing redeployment in next steps.
Does anyone know a workaround and when this bug will be fixed? |
I am having the same issue. I'm finding reviewing CloudFormation change sets painful because they are polluted with changes to Lambda resources that didn't materially change. |
I'm seeing the same problem as @jmassara reported above with node. This one is painful for us as we are trying to use a CodePipeline to deploy Lambda@Edge functions with the CDN in the stack - even if we don't touch the functions, the CLI during packaging thinks the files changed resulting in a CDN update (wait 15 min) even if we didn't change anything in the function code. It is far more than just an unnecessary version publish in the change set - slows the entire CD process down unnecessarily because of how slow CloudFront updates are. |
Hi, Is there any progress with this feature-request? |
We are also facing this exact issue. |
@rmmeans I have exactly same issue. This not only slows down the deployment, but also the rollbacks. |
Guys, my question is not 100% related to this particular bug (I bypassed it by having different and separated lambdas) but there is smth I really cant bypass and I am giving up on it.. I would really appreciate any help/suggestions - please take a look at this error that package command fails when i have too many deps added to my package.json, and unfortunately, do to the nature of the lambda, there is no way to decrease files amount.. so, is there any way, to , actually, run it with zip64 support ? please help.. I have already given up on this... |
The solution may depend on the programming language (and therefore, potentially not possible for some). We solved it in the λ# CLI as follows: .NET Core has a deterministic build system, which means that if the source files and nuget packages have not changed, then the resulting compiled binaries remain identical as well. During the build phase of the package, the CLI creates a checksum of the file contents and filenames instead of the ZIP file itself. The latter contains date & timestamps that would cause the checksum to change with every build. The result is a package filename that only changes when the underlying code changes, which in-turn, only updates Lambda functions--or Lambda layers--when required. |
Any updates on this issue? |
I'm facing the exact same problem |
I've also been suffering this issue. I am using the As per the rest of the conversation in this issue, the md5 of the zip is different each time. Here is a demonstration: ~/C/t/test ❯❯❯ mkdir out
~/C/t/test ❯❯❯ touch out/test
~/C/t/test ❯❯❯ echo "Hello world" > out/test
~/C/t/test ❯❯❯
~/C/t/test ❯❯❯ md5 out/test
MD5 (out/test) = f0ef7081e1539ac00ef5b761b4fb01b3
~/C/t/test ❯❯❯ zip -rqX out.zip out
~/C/t/test ❯❯❯ md5 out.zip
MD5 (out.zip) = 5f28021c0b6fc266abbfb1b36870fa1d
~/C/t/test ❯❯❯
~/C/t/test ❯❯❯ zip -rqX out2.zip out
~/C/t/test ❯❯❯ md5 out2.zip
MD5 (out2.zip) = 5f28021c0b6fc266abbfb1b36870fa1d
~/C/t/test ❯❯❯ # Same md5!
~/C/t/test ❯❯❯ echo "Hello world" > out/test
~/C/t/test ❯❯❯ md5 out/test
MD5 (out/test) = f0ef7081e1539ac00ef5b761b4fb01b3
~/C/t/test ❯❯❯ # Same md5 for file!
~/C/t/test ❯❯❯ zip -rqX out3.zip out
~/C/t/test ❯❯❯ md5 out3.zip
MD5 (out3.zip) = 1a8ec423697ce9c657b6f1c12c51476f
~/C/t/test ❯❯❯ # Different zip file md5! Digging into the source code for the zipping + uploading functionality you can see that the code walks the file tree and adds each file to the zipfile: aws-cli/awscli/customizations/cloudformation/artifact_exporter.py Lines 183 to 196 in 384ae0a
My proposal would be that in this step it also I've tested locally on a lambda with a small 😛 sized ~/C/g/a/.s/Api ❯❯❯ time find . -type f -exec md5 \{\} >> ../out.md5 \;
10.51 real 3.18 user 6.76 sys
~/C/g/a/.s/Api ❯❯❯ md5 ../out.md5
MD5 (../out.mdf) = 6e6584c968e3974b60ba7b4e244a84b5 This was for 3098 files. |
Yes, that's close to how it's done in λ# for the .NET zip packages. Make sure to sort the files by their full path first, then MD5 the file contents and the file path. If you omit the latter, the MD5 doesn't change when you change capitalization of a file! |
@stealthycoin would there be any appetite for a PR implementing this? |
@stealthycoin any update on this? I'd be happy to take a crack at a PR to implement the behaviour discussed. |
hello guys, any updates please :) ? im facing the same issue, i have a multiple lambdas in monorepo |
Hi all, I've created a pull request which seems to solve the issue we were facing, where basically we compute the checksum on the entire function content (after installing all requirements) rather than computing it on the resulting ZIP file (the current behavior). It would be great if this pull gets accepted and merged. |
@gpiccinni I implemented a similar solution to yours in September here #4526, but unfortunately nothing ever came of it. |
@wmonk many thanks for pointing this out, by looking at your pull request I realized that in my case checksum is not changing when filenames change (which in my opinion should), whereas in your code you already addressed this ! I'll look into other libraries such as dirhash where the filename and path is included in the checksum and eventually change my pull request. Thanks |
@gpiccinni , awesome !!! and thanks ! i hope that your PR can be merged quickly ! this can fix a lot of pipelines.., |
@jmassara This problem exists for scripting languages also. I am facing same problem with Node.js lambdas. Looks like it is due to zip headers. Have a look at this stackoverflow discussion. |
Well the CDK team does not have this problem? Find out what they are doing and do the same |
After being frustrated at this issue for a while, i've fixed this in my own deploy scripts. Hopefully this can help some other, and maybe get some optimisations! I'm not sure if this is the "right" way to do it, but it's been working fine for us. One big benefit i've found is that I can make config changes without having to redeploy every function that relies on code (that hadn't changed). find src -type f -exec md5sum {} \; > tmp-md5
find node_modules -type f -exec md5sum {} \; >> tmp-md5
CODE_MD5=$(md5sum tmp-md5 | cut -c 1-32)
if [ ! -f "$CODE_MD5" ]; then
zip -q -r $CODE_MD5 src node_modules # more files here
fi
aws s3 ls s3://bucket-name/$CODE_MD5 || aws s3 cp $CODE_MD5.zip s3://bucket-name/$CODE_MD5
sam deploy --parameter-overrides CodeUriKey=$CODE_MD5 Parameters:
CodeUriKey:
Type: String
NoEcho: true
Lambda:
Type: AWS::Serverless::Function
Properties:
CodeUri:
Bucket: bucket-name
Key: !Ref CodeUriKey |
I have found another workaround (may be easier for those who have many lambda functions in one pipeline) to this issue. Key to this workaround was to find out what contributes to different md5 of a zip even if contents of files within zip have not changed. I found 'Modified Timestamp' of files to be culprit. So idea is; if we can have consistent 'Modified Timestamp' on all files just before 'aws cloudformation package' or 'sam package' command is run, produced zip files will have consistent md5 across build executions.
Above trick has worked for me so far. |
does not work for me |
I have similar issue but with lambda layers instead. I have my template on codecommit and I created a codepipeline with a codebuild that will automate the cloudformation package and deploy process. However, everytime if there is any changes to the codecommit even the lambda layer did not change, it will still create a new lambda layer. Anyone here has got any alternatives? |
I have this same issue. While the suggestion from @rsodha does work to prevent most duplicate packages from being uploaded by the If there were a way we could manually set the |
After many attempts, I still could not prevent a new Lambda Layer from being generated during each CodePipeline run. I tried the following:
I've downloaded a couple of Lambda Layer .zip files that didn't change between CodePipeline runs and checked their MD5 hash values and they are indeed different for some reason. The size of the files are different too (for example, 16,461,107 bytes vs. 16,461,114 bytes), but I can't figure out what the differences are between these two, as I've unzipped them and performed a directory comparison using the comparison tool Meld and it doesn't report any file differences. So, I'm out of ideas as to why a new Lambda Layer is always generated and how to stop this from happening. Any other ideas out there? Thanks. |
@ryancabanas the file dates are probably different. Different values also means different compression level. I had to solve this problem for LambdaSharp.Net as well. You have to MD5 only the file paths and file contents in the ZIP file to make it an idempotent process. |
@bjorg Thanks for helping! I am using the suggestion above from @rsodha and resetting the modified date for all the files, so they are consistent in that respect from build to build. Any suggestions on how to go about determining what else could be different between the files from build to build? Thanks! |
@ryancabanas not sure, but isn't there a modified and a created timestamp on files? Could that be it? Do folders have timestamps? Does the zip file itself have an internal timestamp? I'd recommend you write a little app that opens both zips and compare the metadata of all entries. If the files are the same, it's must be the metadata. Most zip libraries are pretty easy to use. It's almost identical to comparing two folders. This might be frustrating, but so is guessing blindly. Sorry I couldn't be of more assistance. |
@bjorg Okay. I'll dig further in the ways you've mentioned. Thanks! |
@ryancabanas did you try aws-cdk? It looks like it generates same hash for same contents each time. |
CDK fanboy here. They don't have this problem, the cdk-assets does things like normalize file dates and line endings before zipping. But @ryancabanas what you are describing the Some packages embed the absolute path in the package.json after installation and then because The solution is to either remove the package.json from every node_module/ package so that when the hash is calculated, they are excluded. The better solution is to use bundling, a tool like ES Build treeshakes and bundles all your code into a single .js file. This is the only file in the zip then, so no package.json anywhere. |
@rehanvdm Thanks for your article! Yes, what you said about the I'm new to development and AWS, so I haven't used CDK before, or bundling. I will have to look into these. Thanks for the help! |
Got it! So I used the folder-hash package that @rehanvdm mentioned in his article and this helped reveal differences between my Lambda Layer assets. I had already taken care of the |
For me my issue was the I was creating the bundled zip using linux's zip command. I needed to use the -X option so it didn't add all the extra attributes to the created zip. I also included deleting the .pyc files and setting the last modified date for all the files to be the same in the solution so I'm not positive which combination of them is needed. |
I've also encountered this issue when using CodeBuild to package a lambda functions and layers in a CloudFormation template. As a workaround, the |
I still have the same problem with The generated zip file always has a different checksum. What I would like to have is: When running cloudformation package, cloudformation deploy on the same source files then cloudformation must not re-deploy unchanged resources. Are you able to implement that? |
Uh oh!
There was an error while loading. Please reload this page.
I have a Golang lambda with the following template:
Even when the code didn't change (Go build generates the same compiled code),
aws cloudformation package
command generates a new zip file.The text was updated successfully, but these errors were encountered: