CloudFormation

Overview

A CloudFormation test environment is currently being implemented. This test environment mirrors the current AWS setup to enable safe testing, validation, and iteration of infrastructure changes without impacting production workloads.

The purpose of this environment is to:

• Replicate the ETL workflow architecture using isolated, prefixed test resources (e.g., test- S3 buckets, Lambda functions, and Step Functions).
• Trial updates and new configurations to AWS services such as IAM roles, EventBridge rules, and Lambda environment variables.
• Enable infrastructure as code (IaC) development through version-controlled CloudFormation templates.
• Support CI/CD pipeline testing, ensuring deployments are automated and repeatable before being applied to production.
• Facilitate troubleshooting and optimisation of individual components in the ETL pipeline in a low-risk environment.

All resources in the test environment are provisioned and managed via CloudFormation templates, and should be tagged accordingly (e.g., Environment=Test) for clear cost attribution and easier cleanup.

Templates

We currently have 8 CloudFormation template files stored in the s3://kpt-cloudformation/ S3 bucket.

It’s important to create the stacks in the correct order, as some templates depend on resources created by others.

The templates and their recommended order are as follows:

1. EC2.yml
2. SecretsManager.yaml
3. s3.yaml
4. DynamoDB.yaml
5. Lambda.yaml
6. Eventbridge.yaml
7. statemachine.yml
8. RDS.yml

Design

TBD