Creating an Image Service using AWS API Gateway with Proxy Resources + CloudFormation

So, you already read the title and I don’t need to give boring explanations. But, one important thing is that I’m focusing on creating this API using CloudFormation because creating an API using the web console is not even a piece of cake! But when you do more “organizational” work, you need CloudFormation (IaaC – Infrastructure and Code).

So, here we go.

Contents

• Problem Definition
• What would the API look like?
  • Proxy Resources
  • Authentication
• CloudFormation Templates
  • Global Stack
    • Add IAM Role to the Account (optional)
  • Creating the RestAPI
  • Lambda Function
• How to use the API
• Summary

Problem Definition

We are using an external system to deliver images to the front end and other tools. This service provides more sophisticated features like image transformations and manipulations, apart from serving images. And of course, they charge us to provide this service.

We also have internal projects where we need to process (almost) all of our images, which is a huuuuge amount! And if we are to use this external service, it would cost us a LOT! And they are using our S3 buckets to read our images and use them for their processing, so we technically have the images we need to use for our projects.

So, we wanted to create a secured API Endpoint to serve images for our internal projects which kind of replicates the external service, without directly reading from S3 of course (because it’s ugly!).

What would the API look like?

The API Endpoint will expect a URL like below (which is similar to the external service) and it should deliver the correct image to the caller.

https://abc.execute-api.eu-west-1.amazonaws.com/dev/partnerimages/67/28/67286576.jpeg

We’re passing the image we need to retrieve as part of the URL itself, and NOT inside the body of the request. So, you might already have realized we should use a proxy resource for this. If you haven’t because you don’t know what the hell a proxy resource is, let me explain.

Proxy Resources

When you call an API endpoint such as https://api.dev.com/users/, users is the resource we’re trying to work with. So we can use different methods such as GET, POST, PUT, DELETE, etc. to manipulate this users resource. However, then we need to define different resources we’re going to serve from our API beforehand. Then you can trigger different services/actions for different resources from the API Gateway. But, if you have a single back end (i.e. a Lambda function) to handle all the resources, it doesn’t make sense to create different resources because all the requests will go to the same target anyway. So, it’s better to create a dummy resource, pass all the parameters to the back end, and let the back end decode the URL and the request. These dummy resources are called proxy resources.

So, we need an endpoint with a proxy resource forwarding all the requests to the same back end, which will be a lambda function in our case.

Authentication

We wanted to have a key for our API not just to restrict access, but also to track usage of it. And we will have one prod stack, dev stack, and multiple stacks for our engineers. And we wanted to have only two API keys for prod and all the dev stacks respectively. This means, the stacks of engineers will still use the dev API key, to make things easier to use, but it makes the CloudFormation a bit complex. Because these API key values should be created outside of the API level, we wanted to create two global stacks for prod and dev APIs.

So, we decided to use AWS Secrets Manager to hold the API key values and refer them from the API Key in the API Gateway. Let me show you how it looks.

Another issue we had to tackle is to have proper permission to read the images from our lambda functions because they are stored in a different AWS account than our engineering account. So, we also need two global IAM roles for all the lambda functions on prod and dev stacks.

CloudFormation Templates

Global Stack

First, we need to define the API Keys and the IAM role as global resources.

global.yaml
Resources:
  #  Will be used to call the images bucket.
  APILambdaRole:
    Type: AWS::IAM::Role
    Properties:
      RoleName: !Sub
        - ${StackName}--lambda-role
        - { StackName: !Ref "AWS::StackName" }
      ManagedPolicyArns:
        - arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole
      AssumeRolePolicyDocument:
        Version: 2012-10-17
        Statement:
          - Effect: Allow
            Action:
              - sts:AssumeRole
            Principal:
              Service:
                - lambda.amazonaws.com
      Policies:
      - PolicyName: S3Policy
        PolicyDocument:
          Version: 2012-10-17
          Statement:
            - Effect: Allow
              Action:
                - s3:Get*
                - s3:List*
              Resource:
                - !Sub "arn:aws:s3:::${ImagesBucketName}"
                - !Sub "arn:aws:s3:::${ImagesBucketName}/*"

  APIKey:
    Type: AWS::SecretsManager::Secret
    Properties:
      Description: API key definition for Imaginary API
      Name: !Sub
        - ${StackName}--api-key
        - { StackName: !Ref "AWS::StackName" }
      GenerateSecretString:
        SecretStringTemplate: "{}"
        GenerateStringKey: "api_key"
        PasswordLength: 30
        ExcludeCharacters: '!"#$%&()*+,-./:;<=>?@[\]^_`{|}~'

Outputs:
  APILambdaRole:
    Value: !GetAtt APILambdaRole.Arn
    Export:
      Name: !Sub global--api-role--${Environment}

  APIKeySecret:
    Value: !Ref APIKey
    Export:
      Name: !Sub global--api-key--${Environment}

NOTES:

• Make sure you give correct permission for your IAM role to be able to read from the S3 bucket.
• Remember how you have injected the secret. Here, we’re creating a simple JSON object with the api_key as the secret we generate.

Add IAM Role to the Account (optional)

As I explained earlier, our images are stored in a different AWS account, so we need to let this IAM role access it. So, we need to add this IAM role to the Bucket Policy of the S3 bucket. If your bucket is in the same account, then you only need to allow the Lambda role to access that bucket.

Bucket Policy
{
   "Version": "2012-10-17",
   "Statement": [
     {
       "Sid": "S3AccessImagesAccount",
       "Effect": "Allow",
       "Principal": {
         "AWS": [
           ...
           "arn:aws:iam::account_id:role/global--prod--lambda-role",
           "arn:aws:iam::account_id:role/global--dev--lambda-role",
           ...
         ]
       },
       "Action": [
         "s3:Get",
         "s3:List"
       ],
       "Resource": [
         "arn:aws:s3:::images-bucket",
         "arn:aws:s3:::images-bucket/*"
       ]
     },
     ...
   ]
 }

Creating the RestAPI

This is where things got interesting! We all know the AWS documentation is awesome and it’s super easy to find what we need. So, I had to spend a couple of days finding out how to create a RestAPI with a proxy resource using CloudFormation! Well, it took only a few minutes to do it using the web console, and we all know it’s no way close when we do it with good old CloudFormation. Here is my Stack Overflow question, and also the answer I put myself.

I will explain all the issues I had later, but here is the CloudFormation template you’re looking for.

api.yaml
Resources:
  Lambda:
    Type: AWS::Serverless::Function
    Properties:
      FunctionName: !Sub
        - ${StackName}-lambda
        - { StackName: !Ref "AWS::StackName" }
      Handler: api.lambda.handler
      Description: Handles API requests
      Role: {"Fn::ImportValue" : {"Fn::Sub" : "global--api-role--${Environment}"}}
      Runtime: python3.8
      Timeout: 300
      MemorySize: 1536
      CodeUri: ../_build/
      Environment:
        Variables:
          ENVIRONMENT: !Ref Environment
          LOG_LEVEL: !Ref LogLevel
          IMAGES_BUCKET: !Ref ImagesBucketName
          IMAGES_PREFIX: !Ref ImagesBucketPrefix

  RestApi:
    Type: AWS::ApiGateway::RestApi
    Properties:
      Name: !Sub
        - ${StackName}-api
        - { StackName: !Ref "AWS::StackName" }
      Description: Rest API for Images
      BinaryMediaTypes:
        - "*/*"  # <-- This allows the API send binary data (images)
      EndpointConfiguration:
        Types:
          - REGIONAL
      ApiKeySourceType: HEADER  # <-- This is important

  ProxyResource:
    Type: AWS::ApiGateway::Resource
    DependsOn:
      RestApi
    Properties:
      RestApiId: !Ref RestApi
      ParentId: !GetAtt
        - RestApi
        - RootResourceId
      PathPart: '{proxy+}'  # <-- This is important, denotes this is a Proxy Resource

  ProxyResourceANY:
    Type: AWS::ApiGateway::Method
    DependsOn:
      ProxyResource
    Properties:
      RestApiId: !Ref RestApi
      ResourceId: !Ref ProxyResource
      HttpMethod: ANY  # Triggers the lambda for all methods (GET, POST, etc.)
      AuthorizationType: NONE  # No authorization is required
      ApiKeyRequired: true  # <-- Enforces to expect the API Key
      RequestParameters:
        method.request.path.proxy: true  # <-- This is VERY important
      MethodResponses:
        - StatusCode: 200
      Integration:
        Type: AWS_PROXY  # <-- This is SUPER important
        IntegrationHttpMethod: POST
        ContentHandling: CONVERT_TO_TEXT  # <-- Important, depending on your usecase
        PassthroughBehavior: WHEN_NO_MATCH  # <-- This is important
        Uri: !Sub arn:aws:apigateway:${AWS::Region}:lambda:path/2015-03-31/functions/${Lambda.Arn}/invocations
        Credentials: !GetAtt ApiGatewayIamRole.Arn

  RestAPIDeployment:
    Type: AWS::ApiGateway::Deployment
    DependsOn:
      - ProxyResourceANY
    Properties:
      RestApiId: !Ref RestApi
      StageName: !Ref Environment

  ApiKey:
    Type: AWS::ApiGateway::ApiKey
    DependsOn:
      - RestAPIDeployment
    Properties:
      Description: API Key for Images API
      Enabled: 'true'
      StageKeys:
        - RestApiId: !Ref RestApi
          StageName: !Ref Environment
      # We're importing the key value from the Secrets Manager
      Value: !Sub
        - "{{resolve:secretsmanager:${SecretID}:SecretString:${SecretKey}}}"
        - {SecretID: {"Fn::ImportValue" : {"Fn::Sub" : "global--api-key--${Environment}"}}, SecretKey: "api_key"}

  UsagePlan:
    Type: AWS::ApiGateway::UsagePlan
    DependsOn:
      - ApiKey
      - RestAPIDeployment
    Properties:
      ApiStages:
        - ApiId: !Ref RestApi
          Stage: !Ref Environment
      Description: Usage plan for the Images API with the API Key
      UsagePlanName: !Sub
        - ${StackName}-usage-plan
        - { StackName: !Ref "AWS::StackName" }

  UsagePlanKey:
    Type: AWS::ApiGateway::UsagePlanKey
    DependsOn:
      - UsagePlan
    Properties:
      KeyId: !Ref ApiKey 
      KeyType: API_KEY   # <-- This is important
      UsagePlanId: !Ref UsagePlan

  Invoke:
    Type: AWS::Lambda::Permission
    Properties:
      Action: lambda:InvokeFunction
      FunctionName: !GetAtt Lambda.Arn
      Principal: apigateway.amazonaws.com
      SourceArn: !Sub
        - arn:aws:execute-api:${Region}:${AccountId}:${API}/${Env}
        - {Region: !Ref AWS::Region, AccountId: !Ref AWS::AccountId, API: !Ref RestApi, Env: !Ref Environment}

  # Add the IAM role for the API here

Well, that’s quite something, isn’t it! I have marked the important parameters inside the YAML itself, so take a look to see what we could’ve been missed.

NOTES:

• Make sure you have Depends on segments to make sure you have inter-dependent resources created correctly. Otherwise, you’ll get horrible error messages when you deploy the template.
• When we create the same API from the web console, it shows as LAMBDA integration. But it actually is a LAMBDA_PROXY integration. I found this out (from a tip from Mukul Vashishtha) by comparing the two Swagger definitions helped me a lot comparing the YAMLs taken out from Export as Swagger + API Gateway Extensions option.
• According to AWS Support, there are some parameters automatically getting initialized when you’re creating things from the web console, and not when you’re using CloudFormation.

Lambda Function

This is a very simple lambda function to extract the path from the event, retrieve the image, and return it as base64 content.

lambda.py
import json

def handler(event, context):
    # 'path': 'partnerimages/67/28/67286576.jpeg' 
    path = event.get('path')

    if not path:
        return {
            'statusCode': 400,
            'body': json.dumps('No path is provided!')
        }

    # use some dark magic to get the file prefix for the image
    image_prefix = some_dark_magic(path)

    try:
        response = client.get_object(
            Bucket=IMAGES_BUCKET,
            Key=image_prefix
        )

        image = response['Body'].read()

        return {
            'headers': {"Content-Type": "image/jpeg"},
            'statusCode': 200,
            'body': base64.b64encode(image).decode('utf-8'),
            'isBase64Encoded': True
        }

    except Exception as e:
        return {
            'statusCode': 404,
            'body': json.dumps(f'Invalid image path: {path}')
        }

How to use the API

We need two values to use this API.

1. API Endpoint URL
   This can be extracted from the Stages section of your deployed API.
2. API Key
   This can be extracted either from the API Keys section in the API Gateway window or from the Secrets Manager window.

Because we’re using the API_KEY as the Authentication Key Type, we MUST add X-API-KEY to the header of the request.

And also, you need to set Accept and Content-Type parameters in the request header to image/jpeg or whatever the image code you’re using for the images.

Here is a sample Python code calling our API.

import requests

headers = {
    'Content-Type': 'image/jpeg',
    'Accept': 'image/jpeg',
    'X-API-KEY': 'you_api_key'
}

image_path = 'partnerimages/67/28/67286576.jpeg'
file_name = image_path.split('/')[-1]
url = f" https://abc.execute-api.eu-west-1.amazonaws.com/dev/{image_path}"

response = requests.get(url, headers=headers)

if response.status_code == 200:
    print('Success!')
    image = response.content  # base64 encoded image content

    with open(file_name, 'wb') as img:
        img.write(image)
    
elif response.status_code == 404:
    print('Not Found.')

Tadaaa!!! Now you have your own API to serve images from an S3 bucket.

Summary

Creating even a simple API using CloudFormation is a pain in the assets, but we need do it! The documentation doesn’t quite explain how to use different parameters with each other, so we should do some trial an error, or talk to Support.

However, this is a very simple API to serve images residing in an S3 bucket (in a different AWS account) with API Key authentication. Two special things about this implementation are that it uses Proxy Resources and shared API Keys.

So, what do you think? Let me know! Until then, here is a cookie for you!