Introduction to Amazon Web Services and Terraform
API Gateway, Lambda Function and S3
Starting to learn about Amazon Web Services (AWS) and Infrastructure as Code (IaC) is a daunting task. But as any engineer knows, to solve a complex problem, one must divide it into smaller tasks, so I have created a small project that simulates a car sending data to AWS to be processed by a Lambda function and stored in an S3 bucket.
The sample project is stored in this GitHub repository.
Project Description
The car is simulated by a Spring Boot app that uses REST to send 2 kinds of data. Data is captured by an API Gateway, which sends it to a Lambda to format data properly to be saved on an S3 bucket. During all processes, data received by API Gateway and Lambda (input data and errors) is sent to ClowdWatch.
Car Simulator
The app sends 2 types of fake data: LIDAR (Wikipedia contributors, 2023) and ADAS (Wikipedia contributors, 2022)
Here is a data sample:
LIDAR:
{ "dataType": "LIDAR", "vin": "xxxxxxxxx", "geohash": "qqT1DSaOu", "dateTime": [ 2023, 1, 22, 17, 13, 12, 194770400 ], "x": 0.6050206, "y": 0.880925, "z": 0.310677, "lightIntensity": 0.95324767 }
ADAS:
{ "dataType": "ADAS", "vin": "xxxxxxxxx", "geohash": "IeO3kanb5", "dateTime": [2023, 1, 22, 17, 28, 50, 510061100], "engineTemp": -378673827, "engineRotations": -1445087171, "speed": 0.23065627 }
As can be seen, what is being sent, is just two JSON objects with not any kind of authentication. As the intention is to learn the basics, there is no need to add any security layer.
IAM Roles and Policies
Any resource used in AWS has to have its own permissions in order to work with other resources. This is provided by IAM (Identity and Access Management). IAM is a centralized web service that helps to control access to AWS resources (What Is IAM? - AWS Identity and Access Management, n.d.).
The main concept explored here is Roles and Policies.
Role
A Role is an IAM identity that you can create in your account that has specific permissions, instead of being uniquely associated with one person, a role is intended to be assumable by anyone who needs it (Roles Terms and Concepts - AWS Identity and Access Management), this can be a user, a web service offered by AW:
An IAM user in the same AWS account as the role
An IAM user in a different AWS account than the role
A web service offered by AWS
An external user authenticated by an external identity provider
Policies
A policy is an object in AWS that, when associated with an identity or resource, defines its permissions. AWS evaluates these policies when an IAM principal (user or role) makes a request. Permissions in the policies determine whether the request is allowed or denied. (Policies and Permissions in IAM - AWS Identity and Access Management).
Policies are stored in AWS as JSON documents like this:
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "ListObjectsInBucket",
"Effect": "Allow",
"Action": ["s3:ListBucket"],
"Resource": ["arn:aws:s3:::bucket-name"]
},
{
"Sid": "AllObjectActions",
"Effect": "Allow",
"Action": "s3:*Object",
"Resource": ["arn:aws:s3:::bucket-name/*"]
}
]
}
This policy grants Read and Write access to objects in a specific S3 bucket.
Roles and Policies in Terraform
In iam.tf it can be seen how a Policy is associated with a Role and how a Role is associated with a resource.
The next code sample is a sample for creating the proper Role and Policy for the Lambda named "lambda_systems_redirection"
create the "aws_iam_role" to create the role
resource "aws_iam_role" "lambda_systems_redirection" { name = "lambda_systems_redirection_role" assume_role_policy = <<EOF { "Version": "2012-10-17", "Statement": [ { "Action": "sts:AssumeRole", "Principal": { "Service": "lambda.amazonaws.com" }, "Effect": "Allow", "Sid": "" } ] } EOF }
create the policy document with all the statements needed to send logs, list s3 (needed to search for the correct bucket) and write the proper S3 bucket;
data "aws_iam_policy_document" "lambda__systems_redirection"{ statement { sid = "AllowToCreateLogStream" actions = [ "logs:CreateLogGroup", "logs:CreateLogStream", "logs:DescribeLogGroups", "logs:DescribeLogStreams", "logs:PutLogEvents", "logs:GetLogEvents", "logs:FilterLogEvents"] resources = ["arn:aws:logs:*"] effect = "Allow" } statement { sid = "AllowListS3" actions = ["s3:ListBucket"] resources = [aws_s3_bucket.car-data-repository.arn] effect = "Allow" } statement { sid = "AllowWriteToS3" actions = ["s3:PutObject"] resources = ["${aws_s3_bucket.car-data-repository.arn}/*"] effect = "Allow" } }
transform the policy into JSON format
resource "aws_iam_policy" "lambda__systems_redirection" { name = "lambda__systems_redirection" path = "/" description = "AWS IAM Policy for Lamda systems_redirection" policy = data.aws_iam_policy_document.lambda__systems_redirection.json }
then in the "aws_lambda_function" resource, associate the role
resource "aws_lambda_function" "systems_redirection" { .... role = aws_iam_role.lambda_systems_redirection.arn ....
Amazon API Gateway
Amazon API Gateway is an AWS service for creating, publishing, maintaining, monitoring, and securing REST, HTTP, and WebSocket APIs at any scale (What Is Amazon API Gateway? - Amazon API Gateway). This is the entry point to our AWS systems from the outside.
To set up an Amazon API Gateway it is necessary to understand 3 key concepts: OpenAPI, Stage and Integrations.
OpenAPI Specification is a standard that aims to describe all the endpoints provided by an HTTP API. The OpenAPI Specification was originally based on the Swagger Specification and is developed by the OpenAPI Initiative (OpenAPI Specification v3.1.0 | Introduction, Definitions, & More).
The Specification is provided by the file api_gw.yaml. In the sample below it can be seen the "adas" endpoint is described.
paths:
/adas:
post:
responses:
"200":
description: "200 response"
content:
application/json:
schema:
$ref: "#/components/schemas/Empty"
x-amazon-apigateway-integration:
type: "aws"
uri: "arn:aws:apigateway:eu-west-1:lambda:path/2015-03-31/functions/${get_lambda_arn}/invocations"
httpMethod: "POST"
responses:
default:
statusCode: "200"
passthroughBehavior: "when_no_match"
contentHandling: "CONVERT_TO_TEXT"
The following sample is taken from api_gateway.tf and shows how Terraform uses the OpenAPI Specification to create the API Gateway.
/*
Template file for API GW
*/
data "template_file" "api_gw_config_file" {
template = file("${path.module}/files/api_gw.yaml")
vars = {
get_lambda_arn = "${aws_lambda_function.systems_redirection.arn}"
get_lambda_execution_arn = "${aws_iam_policy.nem_egress__api_gw.arn}"
}
}
/**
Create AWS API GATEWAY
*/
resource "aws_api_gateway_rest_api" "nem_egress_api" {
name = "nem-egress-api"
body = "${data.template_file.api_gw_config_file.rendered}"
endpoint_configuration {
types = ["EDGE"]
}
}
An API stage is a logical reference to a lifecycle state of your API (for example, dev
, prod
, beta
, or v2
). In other words, this is how AWS exposes In the project it was created the Stage "production".
#Create API Stage
resource "aws_api_gateway_stage" "nem_egress_api" {
deployment_id = aws_api_gateway_deployment.nem_egress_api.id
rest_api_id = aws_api_gateway_rest_api.nem_egress_api.id
stage_name = "production"
}
#Deploy API Gateway do Stage
resource "aws_api_gateway_deployment" "nem_egress_api" {
rest_api_id = aws_api_gateway_rest_api.nem_egress_api.id
triggers = {
redeployment = sha1(jsonencode(aws_api_gateway_rest_api.nem_egress_api.body))
}
lifecycle {
create_before_destroy = true
}
}
When an API Gateway receives a request, it must send it somewhere in order to be processed. For that, we configured a Lambda Integration.
#Integrate API Gateway with Lambda
resource "aws_lambda_permission" "apigw" {
action = "lambda:InvokeFunction"
function_name = aws_lambda_function.systems_redirection.arn
principal = "apigateway.amazonaws.com"
source_arn = "${aws_api_gateway_rest_api.nem_egress_api.execution_arn}/*/*"
}
AWS Lamba
Lambda is a computing artefact for processing without the need for a server. Lambda runs your code in AWS's infrastructure, providing managing all the computing resources needed. You're only responsible for your code (What Is AWS Lambda? - AWS Lambda). File lambda_code.py implements the code used. The code is implemented in Python and the function "lambda_handler" is where all the processing happens.
def lambda_handler(event, context):
"""
:param event: The event dict that contains the parameters sent when the function
is invoked.
:param context: The context in which the function is called.
:return: The result of the action.
"""
# send request to log
logger.info(event)
if event['dataType'] == 'LIDAR':
key = S3_PREFIX_KEY_LIDAR + "/" + event['vin'] + "/" + current_timestamp()
save(key, event )
elif event['dataType'] == 'ADAS':
key = S3_PREFIX_KEY_ADAS + "/" + event['vin'] + "/" + current_timestamp()
save(key, event )
else:
return {
'statusCode': 400,
"headers": {
"Content-Type": "application/json"
},
'body': json.dumps('Invalid dataType: ' + event['dataType'])
}
return {
'statusCode': 200,
"headers": {
"Content-Type": "application/json"
},
'body': json.dumps('Data Received')
}
Amazon S3
Amazon Simple Storage Service (Amazon S3) is an object storage service (What Is Amazon S3? - Amazon Simple Storage Service) managed by AWS. Data is stored in S3 in Key-Value like a map.
This is how the S3 bucket is created in this project:
resource "aws_s3_bucket" "car-data-repository" {
bucket = "car-data-repository"
tags = {
Name = "Car Data Repository"
Environment = "Ireland"
}
}
References
What is Amazon S3? - Amazon Simple Storage Service. (n.d.-b). Retrieved January 25, 2023, from https://docs.aws.amazon.com/AmazonS3/latest/userguide/Welcome.html
What is AWS Lambda? - AWS Lambda. (n.d.-b). Retrieved January 25, 2023, from https://docs.aws.amazon.com/lambda/latest/dg/welcome.html
Configuring integrations for HTTP APIs - Amazon API Gateway. (n.d.). Retrieved January 25, 2023, from https://docs.aws.amazon.com/apigateway/latest/developerguide/http-api-develop-integrations.html
Working with stages for HTTP APIs - Amazon API Gateway. (n.d.). Retrieved January 25, 2023, from https://docs.aws.amazon.com/apigateway/latest/developerguide/http-api-stages.html
OpenAPI Specification v3.1.0 | Introduction, Definitions, & More. (n.d.). Retrieved January 25, 2023, from https://spec.openapis.org/oas/latest.html
What is Amazon API Gateway? - Amazon API Gateway. (n.d.). Retrieved January 23, 2023, from https://docs.aws.amazon.com/apigateway/latest/developerguide/welcome.html
Policies and permissions in IAM - AWS Identity and Access Management. (n.d.). Retrieved January 22, 2023, from https://docs.aws.amazon.com/IAM/latest/UserGuide/access\_policies.html
Wikipedia contributors. (2022, December 21). Advanced driver-assistance system. Wikipedia. https://en.wikipedia.org/wiki/Advanced\_driver-assistance\_system
Wikipedia contributors. (2023, January 20). Lidar. Wikipedia. https://en.wikipedia.org/wiki/Lidar
What is IAM? - AWS Identity and Access Management. (n.d.). Retrieved January 22, 2023, from https://docs.aws.amazon.com/IAM/latest/UserGuide/introduction.html
Roles terms and concepts - AWS Identity and Access Management. (n.d.). Retrieved January 22, 2023, from https://docs.aws.amazon.com/IAM/latest/UserGuide/id\_roles\_terms-and-concepts.html