Lambda

Background

AWS Lambda is used to automate various components of the K'aute data pipeline.

Initially focused on managing Indici data, Lambda is now also embedded in Step Functions for both Indici and Kotahi workflows and is responsible for handling immunisation (TWO) file collection and ingestion.

Detail

Lambda functions are triggered via EventBridge (cron schedules or S3 PUT events) or within Step Functions. They handle tasks such as:

• Collecting data from secure SFTP endpoints (e.g. TWO immunisation data)
• Converting files (e.g. CSV to Parquet)
• Writing to PostgreSQL schemas (e.g. `imms`, `rpt`)
• Triggering deduplication or transformation functions
• Supporting Step Functions that orchestrate multi-stage ELT processes for both Kotahi and Indici

References

• Indici data
• Kotahi data supply