Update Cross-Stack AWS Lambda Layers

If you don’t already know what AWS Lambda Layers are, let me explain a bit. They’re a way of segregating the dependencies from an AWS Lambda function, which enables us to share those dependencies between multiple Lambda functions. However, I’m not going to explain how cool they are as you can already read them everywhere. But let me explain one of the biggest flaws of AWS Lambda Layers.

Many blogs, tutorials show how to use them in multiple lambda functions when all of them (lambda(s) and the layer) is defined in the same CloudFormation stack. But all hell lose when we define them in separate stacks.

What’s the Problem?

Sometimes, you need to segregate some dependencies (either third-party libs or even your own code) into a custom library so that it can be used in multiple Lambda functions. So, you put all these dependencies into a Lambda Layer and import them (using Exports and Imports) to your Lambda functions. And usually, we define all these in a single CloudFormation stack, and then it works fine.

However, if you have a Lambda Layer defined in a separate CloudFormation stack, you CAN NOT re-deploy the stack containing the Lambda Layer when it’s been used in a different stack.

As of now, the only straight-forward solution is to delete the stack with the Lambda functions, then update the stack with the Lambda layer, and finally redeploying the stack with Lambda function.

That’s insane!

But Why?

How we usually import a Lambda Layer to a Lambda function is using the Layers property in the Lambda function (in the CloudFormation Template).

...
Resources:
  MyAwesomeLambdaLayer:
    Type: AWS::Serverless::LayerVersion
    Properties:
      LayerName: !Sub
        - ${StackName}--layer
        - { StackName: !Ref "AWS::StackName" }
      Description: Lambda Layer containing my dependencies
      ContentUri: ../_build
      CompatibleRuntimes:
        - python3.6
        - python3.7
      RetentionPolicy: Retain
...
Outputs:
  MyAwesomeLambdaLayer:
    Value: !Ref MyAwesomeLambdaLayer
    Export:
      Name: !Sub my-awesome-lambda-layer-export

And I can use it in my other stack like this.

Resources:
  MyAwesomeLamdaFunction:
    Type: AWS::Serverless::Function
    Properties:
      FunctionName: !Sub
        - ${StackName}--functions
        - { StackName: !Ref "AWS::StackName" }
      CodeUri: ../_build
      Layers:
        - Fn::ImportValue: !Sub my-awesome-lambda-layer-export
      ...
      Runtime: python3.7
      Handler: my.awesome.lambda.handler
      ...

So, what’s the problem?

The issue is with the export value from the CloudFormation stack. The ARN of the Lambda Layer contains its version.

arn:aws:lambda:eu-west-1:account-id:layer:dependencies-layer:3

(Now you should sense the issue, I will explain it anyway.)

Let’s assume we deploy the Lambda layer for the first time. So it’s ARN will have :1 at the end denoting the Layer Version. And the export value from the dependencies.yaml will have that version.

arn:aws:lambda:eu-west-1:account-id:layer:dependencies-layer:1

Then we can deploy our Lambda function by using that exported value in its CloudFormation stack (using the Fn::ImportValue directive).

Now, let’s try to re-deploy the Lambda layer. Unlike other resource types, this will cause the ARN of the Lambda Layer to change because it contains the layer version. So, the new layer version should be incremented by one.

arn:aws:lambda:eu-west-1:account-id:layer:dependencies-layer:2

Because the ARN changes and it was exported, CloudFormation should delete the existing export and create a new export with the new ARN. (That’s how it’s implemented.)

But we already know it’s not possible! We cannot delete an export when it’s already used somewhere else! (Hope now you understand the issue. If not, there is no point in reading it further. JK 😛 )

So, the most straight-forward fix is to delete the lambda function (stacks), re-deploy the layer, and re-deploy the lambda stack again. (And no, don’t be stupid and manually delete the layer from the web-console. You will create a CloudFormation stack drift and it will be a huge pain to resolve it.)

Okay, so how to solve it?

Are you ready to fix this?

I did some research and many people suggested various solutions, including using AWS Systems Manager Parameter Store to store the ARN of the (latest) Lambda Layer, and programmatically injecting it into the Lambda function. But that sounded like a bit of an over-work for me.

Since we’re already using a Makefile to deploy the CloudFormation stacks, and I also fancy Makefiles and Shell Scripts, I thought of trying an unorthodox approach.

The first step is to avoid using the export! That’s the link between the two stacks and we MUST break it to proceed. But, then how can we pass the ARN to the Lambda function?

Then I added a Parameter to the Lambda function CloudFormation stack, which I should use as the Layer ARN.

...
Parameters:
  MyLambdaLayerARN:
    Type: String
    Description: ARN of the Lambda Layer (to avoid exports)
...

Resources:
  MyAwesomeLamdaFunction:
    Type: AWS::Serverless::Function
    Properties:
      ...
      Layers:
        - !Ref MyLambdaLayerARN
      ...

Problem (almost) solved!

If you need to keep the old lambda layer version for your Lambda functions, you can manually pass the ARN of the Lambda Layer when you call aws cloudformation deploy command.

But Praneeth, if I need to update my Lambda function, how we can extract the (latest) lambda layer ARN?

Good question! That’s why we need to use the good old awscli commands. The below command can list Lambda Layers.

aws lambda list-layer-versions --layer-name $(LAYER_NAME)

But we need to get the latest version of the Lambda Layer to be used in the Lambda Function. For that, we can use --query parameter on the above command and it will look like this.

aws lambda list-layer-versions --layer-name $(LAYER_NAME) --query 'max_by(LayerVersions, &Version).LayerVersionArn'  --output text

This command will output the ARN of the latest version of the Lambda Layer! Hold on, we’re almost there.

Note: --output text will produce None in case there is no Lambda Layer with the given name, instead of null.

Here is my make recipe (syntax) for deploying the Lambda function.

LAYER_ARN=dependencies-layer

deploy-lambda:
	@$(eval LAYER_ARN:=$(shell aws lambda list-layer-versions \
	  --layer-name $(LAYER_NAME) \
	  --query 'max_by(LayerVersions, &Version).LayerVersionArn'  \
	  --output text))
	@echo "Lambda Layer ARN = \"$(LAYER_ARN)\""

	@if [ $(LAYER_ARN) == "None" ]; then \
		echo "No Lambda Layer found. Please deploy the Lambda Layer stack first" 1>&2 ;\
		exit 1 ; \
	fi

	aws cloudformation deploy \
		--template-file _build/packaged-lambda-function.yaml \
		--stack-name lambda-function \
		...
		--parameter-overrides \
			...
			MyLambdaLayerARN=$(LAYER_ARN)

Yeah yeah, I used Shell Scripts inside a Makefile, but it’s not a sin. So don’t judge me! 😀

If the make recipe is not clear, let me quickly explain it.

It first executes an awscli command to extract the ARN of the latest Lambda Layer version. If it doesn’t exist, the recipe breaks.

Then it simply uses this ARN as a parameter to the lambda-function.yaml CloudFormation Stack. This is caught by the MyLambdaLayerARN parameter and it’s used as a value to the Layers property in the Lambda function definition.

Now, you can peacefully use Lambda Layer in your Lambda function without any restrictions on updating/redeploying the layer.

Oh, by the way

Here are a few thing you need to remember.

1. You MUST deploy the Lambda Layer stack before the Lambda function stack. (Duh!)
2. You MUST set RetentionPolicy property of the Lambda Layer to Retain. Otherwise, it will try to delete the old versions of the layer (when you update the layer) and fail since the old version is still being used by the lambda function.
3. Do NOT specify a Default value for the MyLambdaLayerARN parameter in the Lambda function CloudFormation template. Otherwise, it will keep on using it even when you pass a new one next time.

In short?

AWS being AWS, there are several loopholes and limitations when it comes to CloudFormation Templates. Among many, not allowing to redeploy a Lambda Layer when it’s been used by a Lambda function. How smart!

If we have the Lambda Layer defined in the same CloudFormation Template as all the Lambda functions that use it, everything fine.

But if you have the Lambda Layer defined in a separate CloudFormation Template, you can NOT update the Lambda Layer if you use import-export mechanism (in the CloudFormation template).

To fix this, you can remove the import-export link and pass the ARN as a parameter to the CloudFormation template containing the Lambda function. You can use aws lambda list-layer-versions command from awscli to get the ARN of the latest version of the Lambda Layer to use for the above parameter.

Well, it’s not straight-forward, but it’s way better than having to delete all you lambda functions just to re-deploy your Lambda Layer!

Where can I read more?

• Layers property in Lambda Function
  https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-lambda-function.html#cfn-lambda-function-layers
• Lambda Layers
  https://docs.aws.amazon.com/lambda/latest/dg/configuration-layers.html
• List Layer Versions
  https://docs.aws.amazon.com/cli/latest/reference/lambda/list-layer-versions.html
• Shell Functions
  https://www.gnu.org/software/make/manual/html_node/Shell-Function.html