Psycopg2 layer
Background[edit | edit source]
Postgres instance connectivity is a critical component of our current pipeline. This library however is not available by default in AWS Lambda which is quite a PITA.
Current version[edit | edit source]
As at August 2024, there are two psycopg layers available:[1]
| Layer name | Psycopg version | Python runtime |
|---|---|---|
| psycop9 | 1.3.8 | 3.9 |
| 20240828_psycopg2[2] | 3.2 | 3.12 |
Obviously layer psycop9 is more or less EOL and should not be implemented in new functions.[3]
Layer 20240828_psycopg2 is the most current available at the time of writing and should be used by default for any Postgres connection.
Updating/making new layers[edit | edit source]
For psycop specifically, but also generally, the following steps are necessary to make a functional layer for use in AWS Lambda:
- Create a new folder and navigate to it
- Create
requirements.txtspecifying anything necessary. For psycopg2:echo 'psycopg2-binary' > requirements.txt - Install required library locally using the following filepath exactly:
pip install -r requirements.txt -t python/lib/python3.12/site-packages/ --upgrade.[4]- The filepath /lib/python3.12/site-packages/ is super duper important!!
- Zip up the dir you just made:
zip -r amazing.zip python - $$$
Above instructions/examples for linux only - no idea how to do any of this on Windows or Mac.
TODO[edit | edit source]
- Figure out how to catch new psycopg releases and upgrade layers accordingly.[5]
References[edit | edit source]
- ↑ Note that
psycopg2is a legacy term that refers to major version 2 of the library. Version 3 was released in October 2021, and is called aspsycopginstead. Unfortunately this syntax is used a bit interchangeably for historic reasons, so double check to avoid any confusion. - ↑ There are multiple versions - this relates to the latest version only.
- ↑ Existing functions should also be transitioned from this as soon as feasible.
- ↑ Python version in this example should be amended depending on version available on your local machine AND the intended Lambda runtime.
- ↑ Probably just security or LTS releases but yeah nah.