Introduction to Amazon Web Services and Terraform

Introduction to Amazon Web Services and Terraform

API Gateway, Lambda Function and S3

Starting to learn about Amazon Web Services (AWS) and Infrastructure as Code (IaC) is a daunting task. But as any engineer knows, to solve a complex problem, one must divide it into smaller tasks, so I have created a small project that simulates a car sending data to AWS to be processed by a Lambda function and stored in an S3 bucket.

The sample project is stored in this GitHub repository.

Project Description

diagram

The car is simulated by a Spring Boot app that uses REST to send 2 kinds of data. Data is captured by an API Gateway, which sends it to a Lambda to format data properly to be saved on an S3 bucket. During all processes, data received by API Gateway and Lambda (input data and errors) is sent to ClowdWatch.

Car Simulator

The app sends 2 types of fake data: LIDAR (Wikipedia contributors, 2023) and ADAS (Wikipedia contributors, 2022)

Here is a data sample:

LIDAR:

{ "dataType": "LIDAR", "vin": "xxxxxxxxx", "geohash": "qqT1DSaOu", "dateTime": [ 2023, 1, 22, 17, 13, 12, 194770400 ], "x": 0.6050206, "y": 0.880925, "z": 0.310677, "lightIntensity": 0.95324767 }

ADAS:

{ "dataType": "ADAS", "vin": "xxxxxxxxx", "geohash": "IeO3kanb5", "dateTime": [2023, 1, 22, 17, 28, 50, 510061100], "engineTemp": -378673827, "engineRotations": -1445087171, "speed": 0.23065627 }

As can be seen, what is being sent, is just two JSON objects with not any kind of authentication. As the intention is to learn the basics, there is no need to add any security layer.

IAM Roles and Policies

Any resource used in AWS has to have its own permissions in order to work with other resources. This is provided by IAM (Identity and Access Management). IAM is a centralized web service that helps to control access to AWS resources (What Is IAM? - AWS Identity and Access Management, n.d.).

The main concept explored here is Roles and Policies.

Role

A Role is an IAM identity that you can create in your account that has specific permissions, instead of being uniquely associated with one person, a role is intended to be assumable by anyone who needs it (Roles Terms and Concepts - AWS Identity and Access Management), this can be a user, a web service offered by AW:

  • An IAM user in the same AWS account as the role

  • An IAM user in a different AWS account than the role

  • A web service offered by AWS

  • An external user authenticated by an external identity provider

Policies

A policy is an object in AWS that, when associated with an identity or resource, defines its permissions. AWS evaluates these policies when an IAM principal (user or role) makes a request. Permissions in the policies determine whether the request is allowed or denied. (Policies and Permissions in IAM - AWS Identity and Access Management).

Policies are stored in AWS as JSON documents like this:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "ListObjectsInBucket",
            "Effect": "Allow",
            "Action": ["s3:ListBucket"],
            "Resource": ["arn:aws:s3:::bucket-name"]
        },
        {
            "Sid": "AllObjectActions",
            "Effect": "Allow",
            "Action": "s3:*Object",
            "Resource": ["arn:aws:s3:::bucket-name/*"]
        }
    ]
}

This policy grants Read and Write access to objects in a specific S3 bucket.

Roles and Policies in Terraform

In iam.tf it can be seen how a Policy is associated with a Role and how a Role is associated with a resource.

The next code sample is a sample for creating the proper Role and Policy for the Lambda named "lambda_systems_redirection"

  1. create the "aws_iam_role" to create the role

    resource "aws_iam_role" "lambda_systems_redirection" { 
      name = "lambda_systems_redirection_role" 
      assume_role_policy = <<EOF
        { 
          "Version": "2012-10-17", 
          "Statement": [
             { "Action": "sts:AssumeRole",
               "Principal": {
                  "Service": "lambda.amazonaws.com"
                },
                "Effect": "Allow", "Sid": ""
              } 
            ] 
         }
         EOF
      }
    
  2. create the policy document with all the statements needed to send logs, list s3 (needed to search for the correct bucket) and write the proper S3 bucket;

    data "aws_iam_policy_document" "lambda__systems_redirection"{
    
      statement {
        sid = "AllowToCreateLogStream"
        actions   = [ "logs:CreateLogGroup",
            "logs:CreateLogStream",
            "logs:DescribeLogGroups",
            "logs:DescribeLogStreams",
            "logs:PutLogEvents",
            "logs:GetLogEvents",
            "logs:FilterLogEvents"]
        resources = ["arn:aws:logs:*"]
        effect = "Allow"
      }
    
      statement {
        sid = "AllowListS3"
        actions   = ["s3:ListBucket"]
        resources = [aws_s3_bucket.car-data-repository.arn]
        effect = "Allow"
    
      }
    
      statement {
        sid = "AllowWriteToS3"
        actions   = ["s3:PutObject"]
        resources = ["${aws_s3_bucket.car-data-repository.arn}/*"] 
        effect = "Allow"
    
      }
    }
    
  3. transform the policy into JSON format

    resource "aws_iam_policy" "lambda__systems_redirection" {
      name        = "lambda__systems_redirection"
      path        = "/"
      description = "AWS IAM Policy for Lamda systems_redirection"
      policy      = data.aws_iam_policy_document.lambda__systems_redirection.json
    }
    
  4. then in the "aws_lambda_function" resource, associate the role

    resource "aws_lambda_function" "systems_redirection" {
      ....
      role             = aws_iam_role.lambda_systems_redirection.arn
      ....
    

Amazon API Gateway

Amazon API Gateway is an AWS service for creating, publishing, maintaining, monitoring, and securing REST, HTTP, and WebSocket APIs at any scale (What Is Amazon API Gateway? - Amazon API Gateway). This is the entry point to our AWS systems from the outside.

To set up an Amazon API Gateway it is necessary to understand 3 key concepts: OpenAPI, Stage and Integrations.

OpenAPI Specification is a standard that aims to describe all the endpoints provided by an HTTP API. The OpenAPI Specification was originally based on the Swagger Specification and is developed by the OpenAPI Initiative (OpenAPI Specification v3.1.0 | Introduction, Definitions, & More).

The Specification is provided by the file api_gw.yaml. In the sample below it can be seen the "adas" endpoint is described.

paths:
  /adas:
    post:
      responses:
        "200":
          description: "200 response"
          content:
            application/json:
              schema:
                $ref: "#/components/schemas/Empty"
      x-amazon-apigateway-integration:
        type: "aws"
        uri: "arn:aws:apigateway:eu-west-1:lambda:path/2015-03-31/functions/${get_lambda_arn}/invocations"
        httpMethod: "POST"
        responses:
          default:
            statusCode: "200"
        passthroughBehavior: "when_no_match"
        contentHandling: "CONVERT_TO_TEXT"

The following sample is taken from api_gateway.tf and shows how Terraform uses the OpenAPI Specification to create the API Gateway.

/*
Template file for API GW
*/
data "template_file" "api_gw_config_file" {
  template = file("${path.module}/files/api_gw.yaml")

  vars = {
    get_lambda_arn = "${aws_lambda_function.systems_redirection.arn}"
    get_lambda_execution_arn = "${aws_iam_policy.nem_egress__api_gw.arn}"
  }

}
/**
Create AWS API GATEWAY
*/
resource "aws_api_gateway_rest_api" "nem_egress_api" {
  name = "nem-egress-api"

  body = "${data.template_file.api_gw_config_file.rendered}"

  endpoint_configuration {
    types = ["EDGE"]
  }
}

An API stage is a logical reference to a lifecycle state of your API (for example, dev, prod, beta, or v2). In other words, this is how AWS exposes In the project it was created the Stage "production".

#Create API Stage
resource "aws_api_gateway_stage" "nem_egress_api" {
  deployment_id = aws_api_gateway_deployment.nem_egress_api.id
  rest_api_id   = aws_api_gateway_rest_api.nem_egress_api.id
  stage_name    = "production"
}

#Deploy API Gateway do Stage
resource "aws_api_gateway_deployment" "nem_egress_api" {
  rest_api_id = aws_api_gateway_rest_api.nem_egress_api.id

  triggers = {
    redeployment = sha1(jsonencode(aws_api_gateway_rest_api.nem_egress_api.body))
  }

  lifecycle {
    create_before_destroy = true
  }
}

When an API Gateway receives a request, it must send it somewhere in order to be processed. For that, we configured a Lambda Integration.

#Integrate API Gateway with Lambda
resource "aws_lambda_permission" "apigw" {
    action        = "lambda:InvokeFunction"
    function_name = aws_lambda_function.systems_redirection.arn
    principal     = "apigateway.amazonaws.com"

    source_arn = "${aws_api_gateway_rest_api.nem_egress_api.execution_arn}/*/*"
}

AWS Lamba

Lambda is a computing artefact for processing without the need for a server. Lambda runs your code in AWS's infrastructure, providing managing all the computing resources needed. You're only responsible for your code (What Is AWS Lambda? - AWS Lambda). File lambda_code.py implements the code used. The code is implemented in Python and the function "lambda_handler" is where all the processing happens.


def lambda_handler(event, context):
    """
    :param event: The event dict that contains the parameters sent when the function
                  is invoked.
    :param context: The context in which the function is called.
    :return: The result of the action.
    """
    # send request to log
    logger.info(event)

    if event['dataType'] == 'LIDAR':
        key = S3_PREFIX_KEY_LIDAR + "/" + event['vin'] + "/" + current_timestamp()
        save(key, event )
    elif event['dataType'] == 'ADAS': 
        key = S3_PREFIX_KEY_ADAS + "/" + event['vin'] + "/" + current_timestamp()
        save(key, event )
    else:
        return {
            'statusCode': 400,
            "headers": {
                "Content-Type": "application/json"
        },
        'body': json.dumps('Invalid dataType: ' + event['dataType'])
    }


    return {
        'statusCode': 200,
        "headers": {
            "Content-Type": "application/json"
        },
        'body': json.dumps('Data Received')
    }

Amazon S3

Amazon Simple Storage Service (Amazon S3) is an object storage service (What Is Amazon S3? - Amazon Simple Storage Service) managed by AWS. Data is stored in S3 in Key-Value like a map.

This is how the S3 bucket is created in this project:

resource "aws_s3_bucket" "car-data-repository" {
  bucket = "car-data-repository"

  tags = {
    Name        = "Car Data Repository"
    Environment = "Ireland"
  }
}

References

What is Amazon S3? - Amazon Simple Storage Service. (n.d.-b). Retrieved January 25, 2023, from https://docs.aws.amazon.com/AmazonS3/latest/userguide/Welcome.html

What is AWS Lambda? - AWS Lambda. (n.d.-b). Retrieved January 25, 2023, from https://docs.aws.amazon.com/lambda/latest/dg/welcome.html

Configuring integrations for HTTP APIs - Amazon API Gateway. (n.d.). Retrieved January 25, 2023, from https://docs.aws.amazon.com/apigateway/latest/developerguide/http-api-develop-integrations.html

Working with stages for HTTP APIs - Amazon API Gateway. (n.d.). Retrieved January 25, 2023, from https://docs.aws.amazon.com/apigateway/latest/developerguide/http-api-stages.html

OpenAPI Specification v3.1.0 | Introduction, Definitions, & More. (n.d.). Retrieved January 25, 2023, from https://spec.openapis.org/oas/latest.html

What is Amazon API Gateway? - Amazon API Gateway. (n.d.). Retrieved January 23, 2023, from https://docs.aws.amazon.com/apigateway/latest/developerguide/welcome.html

Policies and permissions in IAM - AWS Identity and Access Management. (n.d.). Retrieved January 22, 2023, from https://docs.aws.amazon.com/IAM/latest/UserGuide/access\_policies.html

Wikipedia contributors. (2022, December 21). Advanced driver-assistance system. Wikipedia. https://en.wikipedia.org/wiki/Advanced\_driver-assistance\_system

Wikipedia contributors. (2023, January 20). Lidar. Wikipedia. https://en.wikipedia.org/wiki/Lidar

What is IAM? - AWS Identity and Access Management. (n.d.). Retrieved January 22, 2023, from https://docs.aws.amazon.com/IAM/latest/UserGuide/introduction.html

Roles terms and concepts - AWS Identity and Access Management. (n.d.). Retrieved January 22, 2023, from https://docs.aws.amazon.com/IAM/latest/UserGuide/id\_roles\_terms-and-concepts.html