Lambda Circuit Breaker

on
nov 09, 2018
in
AWS

The Circuit Breaker Pattern (CBP) is a pattern where a function call is
wrapped by a circuit breaker (CB) object. When the function call generates an error, the CB trips and the
connection is open, which means the CB will throw an error and thus fail fast. After some time the CB closes and the
wrapped function will be called. If the function generates an error, the CB process starts again.

Bulkheads

When architecting a solution in AWS, it is common practise to create bulkheads in the design. Bulkheads isolate failures
in systems and prevent cascading failures. Introducing CB in the design creates bulkheads.

Lambda and Circuit Breakers

AWS Lambda does not provide the circuit breakers capability. Fortunately, there are circuit breaker implementations that
can be packaged with Lambda so the lambda itself can be the circuit breaker.

Python Circuit breaker

The open source project fabfuel/circuitbreaker by Fabian Fuelling
is a CBP implementation for Python 2 and 3. The library provides a configurable CB that is very easy to use and add to existing code bases.
The library provides the @circuit decorator that can be used to decorate existing functions.

@circuit(failure_threshold=2)
def send_request(url: str, user: str, pwd: str, req=None) -> (int, dict):
    resp = requests.get(url, auth=HTTPBasicAuth(user, pwd), verify=False, timeout=2)
    body = resp.json()
    code = resp.status_code
    if code == 500:
        raise ValueError('Service returned 500')
    return code, body

Example

The example repository shows how to create and build a lambda.
The lambda calls http://httpbin.org/status/500, which returns a HTTP 500 status code. The lambda then raises a
ValueError and the CB trips after two exceptions. The CB resets after 30 seconds and tries to call the service again.

Using the example

I assume you have a mac with python, pipenv, sceptre and docker installed. To run the example type make deploy.
To get the url of the api gateway, type make info. To remove the example type make delete.
After two invocations, the CB will trip. The next invocation will fail fast.

Conclusion

It is easy to add bulkheads to the solution design by wrapping functions that call subsystems by a CB. When the
subsystem fails, the CB acts as a bulkhead, isolating the subsystem.

Share this article: Tweet this post / Post on LinkedIn