How to install any Python binary dependency in AWS lambda

When you develop an AWS Lambda function in Python, you may require packages which include binary libraries. In this blog we will show you how to use the official Docker Python image to make sure you have a working Lambda.

If you are developing your code on a Mac or Windows operating system, it is pretty hard to install the correct Python packages that your code depends on. There are even github repositories with pre-compiled binaries, which blindly copy into your zip file. This is an unwanted process, as it deviates from the standard way of developing and installing Python software.

Luckily Docker provides us with an official Python image based upon 64-bit Linux, which matches the AWS Lambda runtime. By running the installation and compilation process inside Docker, we have a standard workflow that delivers an artifact matching our target runtime environment!

In this example, we have the following source directory structure:

├── requirements.txt
├── src
│   ├── my_postgres_function.py
├── target
│   └── content

in which we want to include the Python Postgresql package psycopg2. This dependency is declared in the file requirements.txt.

To create a ready-to-run zip file for AWS Lambda, we first clear the target directory:

rm -rf target/content
mkdir -p target/content

Then run a pip install inside the container based upon the official Python 2.7 image:

cp requirements.txt target/content
docker run --volume $PWD/target/content:/venv \
	   --workdir /venv python:2.7 \
	   pip install --quiet -t /venv -r requirements.txt

Next, we can add and compile all the sources:

cp -r src/* target/content
docker run --volume $PWD/target/content:/venv \
	   --workdir /venv python:2.7 \
	   python -m compileall -q -f .

Now all the sources are added and dependencies are installed, we need to ensure that all sources and directories are readable:

find target/content -type d | xargs  chmod ugo+rx
find target/content -type f | xargs  chmod ugo+r

Finally, we zip the result to upload to an s3 bucket for deployment.

cd target/content && zip --quiet -9r ../../target/$NAME-$VERSION.zip  *
chmod ugo+r target/$NAME-$VERSION.zip


By using the official Docker Python image, we can create functions which include the proper binaries for the AWS Lambda run-time.

Picture of Mark van Holsteijn
Mark van Holsteijn
Cloud Consultant