Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Added possibility to deploy sample example with terraform #37

Merged
merged 8 commits into from
Oct 24, 2024
Merged
Show file tree
Hide file tree
Changes from 3 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions .env.example
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
LOCALSTACK_AUTH_TOKEN=YOUR_TOKEN
# LOCAL_RUN=false # Default is true
k-a-il marked this conversation as resolved.
Show resolved Hide resolved
27 changes: 26 additions & 1 deletion .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -59,4 +59,29 @@ volume/

# lambda packages
lambdas/*/package/
lambdas/*/lambda.zip
lambdas/*/lambda.zip
terraform/*/*.zip

# Local .terraform directories
**/.terraform/*

# .tfstate files
*.tfstate
*.tfstate.*

# Crash log files
crash.log
crash.*.log

*.tfvars
*.tfvars.json

override.tf
override.tf.json
*_override.tf
*_override.tf.json

.terraform.tfstate.lock.info

.terraformrc
terraform.rc
32 changes: 32 additions & 0 deletions Makefile
Original file line number Diff line number Diff line change
@@ -0,0 +1,32 @@
export AWS_ACCESS_KEY_ID ?= test
export AWS_SECRET_ACCESS_KEY ?= test
export LOCAL_RUN ?= true
SHELL := /bin/bash

include .env

build:
bin/build_lambdas.sh;

awslocal-setup:
bin/deploy.sh

terraform-setup:
$(MAKE) build
cd deployment/terraform; \
tflocal init; \
echo "Deploying Terraform configuration 🚀"; \
tflocal apply --auto-approve -var="local_run=${LOCAL_RUN}"; \
echo "Paste the function URLs above to the WebApp 🎉";

terraform-destroy:
cd deployment/terraform; \
tflocal destroy --auto-approve;

start:
@LOCALSTACK_AUTH_TOKEN=$(LOCALSTACK_AUTH_TOKEN) localstack start -d

stop:
localstack stop

.PHONY: build awslocal-setup terraform-setup terraform-destroy start stop
k-a-il marked this conversation as resolved.
Show resolved Hide resolved
47 changes: 31 additions & 16 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -69,22 +69,37 @@ source .venv/bin/activate
pip install -r requirements-dev.txt
```

## Instructions

You can set up and deploy the sample application on LocalStack by executing the commands in our Makefile. First, create a `.env` file using the provided `.env.example` file as a template, and include your LocalStack token in it. Then, run `make -s start` to initiate LocalStack on your machine.

Next, execute `make -s terraform-setup` to provision the infrastructure on LocalStack using Terraform CLI and its scripts. Alternatively, run `make -s awslocal-setup` to set up the infrastructure with the local AWS CLI.
alexrashed marked this conversation as resolved.
Show resolved Hide resolved

If you prefer, you can also follow these step-by-step instructions for a manual deployment.

### LocalStack

Start LocalStack Pro with the appropriate CORS configuration for the S3 Website:
Start LocalStack Pro with Auth Token:

```bash
LOCALSTACK_AUTH_TOKEN=... localstack start
LOCALSTACK_AUTH_TOKEN=... localstack start (-d)
```

## Instructions
### Terraform

To create the infrastructure using Terraform, run the following commands:

```shell
cd deployment/terraform
tflocal init
tflocal apply --auto-approve
```

You can create the AWS infrastructure on LocalStack by running `bin/deploy.sh`.
Make sure you have Python 3.11 activated before running the script.
We are using the `tflocal` wrapper to configure the local service endpoints, and send the API requests to LocalStack, instead of AWS.

Here are instructions to deploy it manually step-by-step.
### AWS CLI

### Create the buckets
#### Create the buckets

The names are completely configurable via SSM:

Expand All @@ -93,14 +108,14 @@ awslocal s3 mb s3://localstack-thumbnails-app-images
awslocal s3 mb s3://localstack-thumbnails-app-resized
```

### Put the bucket names into the parameter store
#### Put the bucket names into the parameter store

```bash
awslocal ssm put-parameter --name /localstack-thumbnail-app/buckets/images --type "String" --value "localstack-thumbnails-app-images"
awslocal ssm put-parameter --name /localstack-thumbnail-app/buckets/resized --type "String" --value "localstack-thumbnails-app-resized"
```

### Create the DLQ Topic for failed lambda invokes
#### Create the DLQ Topic for failed lambda invokes

```bash
awslocal sns create-topic --name failed-resize-topic
Expand All @@ -115,9 +130,9 @@ awslocal sns subscribe \
--notification-endpoint [email protected]
```

### Create the lambdas
#### Create the lambdas

#### S3 pre-signed POST URL generator
##### S3 pre-signed POST URL generator

This Lambda is responsible for generating pre-signed POST URLs to upload files to an S3 bucket.

Expand All @@ -143,7 +158,7 @@ awslocal lambda create-function-url-config \

Copy the `FunctionUrl` from the response, you will need it later to make the app work.

### Image lister lambda
#### Image lister lambda

The `list` Lambda is very similar:

Expand All @@ -166,7 +181,7 @@ awslocal lambda create-function-url-config \
--auth-type NONE
```

### Resizer Lambda
#### Resizer Lambda

```bash
(
Expand All @@ -189,23 +204,23 @@ awslocal lambda create-function \
--environment Variables="{STAGE=local}"
```

### Connect the S3 bucket to the resizer lambda
#### Connect the S3 bucket to the resizer lambda

```bash
awslocal s3api put-bucket-notification-configuration \
--bucket localstack-thumbnails-app-images \
--notification-configuration "{\"LambdaFunctionConfigurations\": [{\"LambdaFunctionArn\": \"$(awslocal lambda get-function --function-name resize | jq -r .Configuration.FunctionArn)\", \"Events\": [\"s3:ObjectCreated:*\"]}]}"
```

### Create the static s3 webapp
#### Create the static s3 webapp

```bash
awslocal s3 mb s3://webapp
awslocal s3 sync --delete ./website s3://webapp
awslocal s3 website s3://webapp --index-document index.html
```

### Using the application
#### Using the application

Once deployed, visit http://webapp.s3-website.localhost.localstack.cloud:4566

Expand Down
28 changes: 28 additions & 0 deletions bin/build_lambdas.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
#!/bin/bash

(cd lambdas/presign; rm -f lambda.zip; zip lambda.zip handler.py)

(cd lambdas/list; rm -f lambda.zip; zip lambda.zip handler.py)

os=$(uname -s)
if [ "$os" == "Darwin" ]; then
(
cd lambdas/resize
rm -rf libs lambda.zip
docker run --platform linux/x86_64 --rm -v "$PWD":/var/task "public.ecr.aws/sam/build-python3.11" /bin/sh -c "pip3 install -r requirements.txt -t libs; exit"

cd libs && zip -r ../lambda.zip . && cd ..
zip lambda.zip handler.py
rm -rf libs
)
else
(
cd lambdas/resize
rm -rf package lambda.zip
mkdir package
pip3 install -r requirements.txt --platform manylinux2014_x86_64 --only-binary=:all: -t package
zip lambda.zip handler.py
cd package
zip -r ../lambda.zip *;
)
fi
26 changes: 2 additions & 24 deletions bin/deploy.sh
k-a-il marked this conversation as resolved.
Show resolved Hide resolved
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,8 @@

export AWS_DEFAULT_REGION=us-east-1

bin/build_lambdas.sh
alexrashed marked this conversation as resolved.
Show resolved Hide resolved

awslocal s3 mb s3://localstack-thumbnails-app-images
awslocal s3 mb s3://localstack-thumbnails-app-resized

Expand All @@ -14,7 +16,6 @@ awslocal sns subscribe \
--protocol email \
--notification-endpoint [email protected]

(cd lambdas/presign; rm -f lambda.zip; zip lambda.zip handler.py)
awslocal lambda create-function \
--function-name presign \
--runtime python3.11 \
Expand All @@ -30,7 +31,6 @@ awslocal lambda create-function-url-config \
--function-name presign \
--auth-type NONE

(cd lambdas/list; rm -f lambda.zip; zip lambda.zip handler.py)
awslocal lambda create-function \
--function-name list \
--runtime python3.11 \
Expand All @@ -46,28 +46,6 @@ awslocal lambda create-function-url-config \
--function-name list \
--auth-type NONE

os=$(uname -s)
if [ "$os" == "Darwin" ]; then
(
cd lambdas/resize
rm -rf libs lambda.zip
docker run --platform linux/x86_64 --rm -v "$PWD":/var/task "public.ecr.aws/sam/build-python3.11" /bin/sh -c "pip3 install -r requirements.txt -t libs; exit"

cd libs && zip -r ../lambda.zip . && cd ..
zip lambda.zip handler.py
rm -rf libs
)
else
(
cd lambdas/resize
rm -rf package lambda.zip
mkdir package
pip3 install -r requirements.txt --platform manylinux2014_x86_64 --only-binary=:all: -t package
zip lambda.zip handler.py
cd package
zip -r ../lambda.zip *;
)
fi

awslocal lambda create-function \
--function-name resize \
Expand Down
12 changes: 12 additions & 0 deletions deployment/policies/lambda.json
k-a-il marked this conversation as resolved.
Show resolved Hide resolved
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
{
"Version": "2012-10-17",
"Statement": [
{
"Action": "sts:AssumeRole",
"Effect": "Allow",
"Principal": {
"Service": "lambda.amazonaws.com"
}
}
]
}
k-a-il marked this conversation as resolved.
Show resolved Hide resolved
14 changes: 14 additions & 0 deletions deployment/policies/lambda_ssm.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"ssm:GetParameters"
],
"Resource": [
"arn:aws:ssm:::parameter/localstack-thumbnail-app/*"
]
}
]
}
18 changes: 18 additions & 0 deletions deployment/policies/list_lambda_s3_buckets.json.tpl
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"s3:ListBucket",
"s3:GetObject"
],
"Resource": [
"arn:aws:s3:::${images_bucket}",
"arn:aws:s3:::${images_bucket}/*",
"arn:aws:s3:::${images_resized_bucket}",
"arn:aws:s3:::${images_resized_bucket}/*"
]
}
]
}
17 changes: 17 additions & 0 deletions deployment/policies/presign_lambda_s3_buckets.json.tpl
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"s3:ListBucket",
"s3:GetObject",
"s3:PutObject"
],
"Resource": [
"arn:aws:s3:::${images_bucket}",
"arn:aws:s3:::${images_bucket}/*"
]
}
]
}
17 changes: 17 additions & 0 deletions deployment/policies/resize_lambda_s3_buckets.json.tpl
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"s3:ListBucket",
"s3:GetObject",
"s3:PutObject"
],
"Resource": [
"arn:aws:s3:::${images_resized_bucket}",
"arn:aws:s3:::${images_resized_bucket}/*"
]
}
]
}
17 changes: 17 additions & 0 deletions deployment/policies/resize_lambda_sns.json.tpl
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
{
"Version": "2012-10-17",
"Statement": [
{
"Action": [
"sns:Publish"
],
"Effect": "Allow",
"Resource": "${failure_notifications_topic_arn}"
},
{
"Action": "lambda:InvokeFunction",
"Effect": "Allow",
"Resource": "${resize_lambda_arn}"
}
]
}
13 changes: 13 additions & 0 deletions deployment/policies/website_s3_bucket.json.tpl
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"AWS": "${cdn_identity_arn}"
},
"Action": "s3:GetObject",
"Resource": "${website_bucket_arn}/*"
}
]
}
Loading
Loading