How to install any Python binary dependency in AWS lambda

When you develop an AWS Lambda function in Python, you may require packages which include binary libraries. In this blog we will show you how to use the official Docker Python image to make sure you have a working Lambda.

If you are developing your code on a Mac or Windows operating system, it is pretty hard to install the correct Python packages that your code depends on. There are even github repositories with pre-compiled binaries, which blindly copy into your zip file. This is an unwanted process, as it deviates from the standard way of developing and installing Python software.

Luckily Docker provides us with an official Python image based upon 64-bit Linux, which matches the AWS Lambda runtime. By running the installation and compilation process inside Docker, we have a standard workflow that delivers an artifact matching our target runtime environment!

In this example, we have the following source directory structure:

├── Dockerfile
├── requirements.txt
├── src
│   ├── my_postgres_function.py

in which we want to include the Python Postgresql package psycopg2. This dependency is declared in the file requirements.txt.

We can obtain the ready-to-deploy zip file for AWS Lambda, using the following standard Dockerfile:

FROM python:3.6
RUN apt-get update && apt-get install -y zip
WORKDIR /lambda

# Add the requiremts
ADD requirements.txt /tmp
RUN pip install --quiet -t /lambda -r /tmp/requirements.txt && \
    find /lambda -type d | xargs chmod ugo+rx && \
    find /lambda -type f | xargs chmod ugo+r

# Add your source code
ADD src/ /lambda/
RUN find /lambda -type d | xargs chmod ugo+rx && \
    find /lambda -type f | xargs chmod ugo+r

# compile the lot.
RUN python -m compileall -q /lambda

RUN zip --quiet -9r /lambda.zip .

FROM scratch
COPY --from=0 /lambda.zip /

To build your zip file, just type:

docker build -t my-lambda .

Getting the zip file out, takes a little bit of Docker foo:

ID=$(docker create my-lambda /bin/true)		# create a container from the image
docker cp $ID:/ .				# copy the file from the /


By using the official Docker Python image and the multi-stage build functionality, we have a standard Dockerfile that we can use to build any AWS Lambda written in Python that includes the property binary dependencies.

Picture of Mark van Holsteijn
Mark van Holsteijn