How we created a serverless version of cfn-flip

The original version of cfn-flip is a command line tool that converts CloudFormation templates from JSON to YAML and vice versa. This tool has existed for some time and does the job magnificently. Most people find YAML more readable. I belong to the minority that finds JSON easier to read. It was ritten by Steve Engledow and you can find the original cfn-flip Github repository at https://github.com/awslabs/aws-cfn-template-flip
In my experience new idea’s aren’t born trying to think up what doesn’t yet exist, but rather by observing something lacking and building something around that. This was definitely the case for serverless cfn-flip.
The idea to create a serverless version of the tool that could do more than just convert the two formats was born when I noticed cfnfip.com was not registered.

Objective

Before I wrote a single line of code I thought of what I wanted to achieve and narrowed it down to the following:

As little as possible interaction with the AWS console!
Created fully in CloudFormation.
As I am a Ruby fanatic, it must output to CfnDsl as well.
Each conversion function in its own Lambda function.
With pretty colours (syntax highlighting)
Backed with API Gateway, of course.
Giving back to the community by open sourcing the entire project.

Architecture

CfnFlip consists of an S3 bucket, four Lambda functions, an API Gateway, a CloudFront distribution and Route53. I’ve crafted the architecture to provide you with a visual overview:

CloudFormation Dependency Stack

I really like how the serverless framework creates an S3 bucket on the fly. It can store your artifacts there without the need to perform any manual bucket creation in the console or with the CLI. Inspired by that simple yet eloquent awesomeness I decided that this project needed that kind of magic as well. This came in the form of a dependency stack where I create some immutable resources that subsequent stacks import. The dependency stack took this form:

AWSTemplateFormatVersion: '2010-09-09'
Parameters:
  DomainName:
    Type: String
    content: Domain name for the hosted zone
    Default: cfnflip.com
  ProjectName:
    Type: String
    content: Name of your project, e.g. cfnflip
    Default: CfnFlip

I’ve used parameters here so you can easily re-use this snippet for another project. It takes the domain name and project name as parameters. Then onto the resources:

  CloudformationBucket:
    Type: AWS::S3::Bucket

We let CloudFormation create a bucket with a dynamic name. The name will be outputted and exported. Then we let CloudFormation create our HostedZone.

  HostedZone:
    Type: AWS::Route53::HostedZone
    Properties:
      HostedZoneConfig:
        Comment: !Join
          - ''
          - - 'HostedZone for '
            - !Ref 'DomainName'
      Name: !Ref 'DomainName'

Next up is a Lambda function. I wanted to automatically configure the Route53 domain name (cfnflip.com) with the DNS servers of the dynamically created Route53 HostedZone, so I would have to have the least amount of interaction with the AWS console. A custom resource invokes this Lambda function and passes on the DNS servers and then proceeds to configure the domain name using the NodeJS AWS SDK:

  LambdaFunction:
    Type: AWS::Lambda::Function
    Properties:
      Handler: index.handler
      Role: !GetAtt 'LambdaFunctionRole.Arn'
      Code:
        ZipFile: !Join
          - "\n"
          - - var AWS = require('aws-sdk');
            - 'AWS.config.update({region: ''us-east-1''});'
            - var response = require('cfn-response');
            - exports.handler = function(event, context) {
            - '    var nameservers = event.ResourceProperties.Nameservers;'
            - '    var domainname  = event.ResourceProperties.HostedZone;'
            - ''
            - '    console.log(nameservers);'
            - '    console.log(domainname);'
            - '    if (event.RequestType == ''Delete'') {'
            - '        var responseData = { Value: ''Go ahead.'' };'
            - '        response.send(event, context, response.SUCCESS, responseData);'
            - '    }'
            - '    else'
            - '    {'
            - ''
            - '        var route53domains = new AWS.Route53Domains();'
            - '        var ServerList = [];'
            - '        nameservers.forEach(function(server){'
            - '          ServerList.push({'
            - '            Name: server'
            - '          })'
            - '        });'
            - '        route53domains.updateDomainNameservers('
            - '            {'
            - '                DomainName: domainname,'
            - '                Nameservers: ServerList'
            - '            }, function(err, data) {'
            - '                if (err) console.log(err, err.stack);'
            - '                else     console.log(data);'
            - '                var responseData = { Value: ''Hurray!'' };'
            - '                response.send(event, context, response.SUCCESS, responseData);'
            - '            }'
            - '        )'
            - '    }'
            - '};'
      Runtime: nodejs6.10
      MemorySize: '128'
      Timeout: '25'
    DependsOn: LambdaFunctionRole

Of course our Lambda function needs permissions. Our execution role permits the Lambda function to update the DNS servers and to log to CloudWatch:

  LambdaFunctionRole:
    Type: AWS::IAM::Role
    Properties:
      Path: /
      AssumeRolePolicyDocument:
        Version: '2012-10-17'
        Statement:
          - Effect: Allow
            Principal:
              Service: lambda.amazonaws.com
            Action:
              - sts:AssumeRole
      Policies:
        - PolicyDocument:
            Version: '2012-10-17'
            Statement:
              - Action:
                  - logs:CreateLogGroup
                  - logs:CreateLogStream
                  - logs:PutLogEvents
                Effect: Allow
                Resource: arn:aws:logs:*:*:*
              - Action:
                  - route53domains:UpdateDomainNameservers
                Effect: Allow
                Resource: '*'
          PolicyName: Route53LambdaUpdateRole

Finally, we define a custom resource and pass on the DNS servers to our Lambda function:

  UpdateRoute53:
    Type: Custom::UpdateRoute53
    Properties:
      HostedZone: !Ref 'DomainName'
      Nameservers: !GetAtt 'HostedZone.NameServers'
      ServiceToken: !GetAtt 'LambdaFunction.Arn'
    DependsOn:
      - LambdaFunction
      - HostedZone

We now have a HostedZone and an S3 bucket that should never change. We output and export these immutable values for our application stack to use:

Outputs:
  HostedZoneId:
    Value: !Ref 'HostedZone'
    Export:
      Name: !Join
        - ''
        - - !Ref 'ProjectName'
          - HostedZoneId
  HostedZoneName:
    Value: !Ref 'DomainName'
    Export:
      Name: !Join
        - ''
        - - !Ref 'ProjectName'
          - HostedZoneName
  CloudformationBucket:
    Value: !Ref 'CloudformationBucket'
    Export:
      Name: !Join
        - ''
        - - !Ref 'ProjectName'
          - CloudformationBucket

You can import an exported value in another stack by referring to it like this: !ImportValue ‘MyExportedValueName’

A word of caution: If you use an exported value in another stack you can’t modify the value of what you’ve exported until no stack refers to it anymore. In other words, use exports only for data that never ever changes.

The Lambda functions

cfn-flip is a Python script that ordinarily takes a flat file as argument and transforms it to either YAML or JSON, whereas cfn2dsl is a Ruby script that does something similar but instead outputs to Ruby code that the cfndsl gem uses and outputs a CloudFormation template when you feed it to a Ruby interpreter. That meant I had to create two custom Lambda deployment packages: One for cfn-flip and one for cfn2dsl.
Creating a deployment package for Python is a straightforward process. You can simply clone the cfn-flip repository and install the dependencies to your deployment package directory as follows:

pip install -r requirements.txt -t ~/your/deployment_package
On MacOS X, however, this doesn’t work out of the box. You will need to create (or add to) setup.cfg in the cloned directory with the following content:

[install]
prefix=

By default you invoke the cfn-flip tool with some arguments to specify which file you want to convert. In this case, I wanted to have cfn-flip to use the content provided to the Lambda function. So essentially: whatever code the user submitted on the website. Looking at the source code there was just a single function in cfn-flip I needed, called flip.
My Lambda function then looked like this:

import json
import sys
import cStringIO
import urlparse
from cfn_flip import flip


def lambda_handler(event, context):
    data = urlparse.parse_qsl(event["body"])
    for k, v in data:
        if k == 'code':
            plaintext = v
        else:
            plaintext = ""

    stdout_ = sys.stdout
    stream = cStringIO.StringIO()
    sys.stdout = stream
    hndlr = sys.stdout
    hndlr.write(flip(
        plaintext
    ))
    sys.stdout = stdout_
    data = stream.getvalue()

    return {
        'statusCode': 200,
        'body': json.loads(json.dumps(data)),
        'headers': {
                'Content-Type': 'plain/text',
                'Access-Control-Allow-Origin': '*',
                'Cache-control': 'private, max-age=0, no-cache'
            },
        }

The event body expects the template’s body being posted with code as key. The Lambda function then captures the output of cfn-flip’s flip function, and finally returns the flipped version.
Porting Ruby’s “cfn-dsl” was a different challenge altogether. I’ve used a portable version of Ruby 2.2.x called Traveling Ruby to prime my deployment package on a Linux environment. The cfn2dsl gem (rightfully) pinned to Ruby 2.3 as a minimum version, so I had to get creative.
I started with installing Ruby with an identical version of traveling ruby using rbenv. Then, I created a Gemfile with the dependencies of cfn2dsl, but without the versions pinned to a minimum version. This allowed me to get all of cfndsl’s gem dependencies that were compatible with Ruby 2.2. I then moved all the dependencies into the traveling ruby’s gem directory and created a cfn2dsl_lib directory and stuck the cfn2dsl source code there. With that out-of-the-way I copied over the deployment package to my trusty Macbook and started to work on the Lambda function.
Now, AWS does not support Ruby natively, so the deployment package consists of a NodeJS function to invoke the Ruby script, and the Ruby script to transform the template into CfnDsl code. The ruby code looked like this:

#!./bin/ruby

require "uri"
require_relative 'cfn2dsl_lib/cfn2dsl'

payload = URI.decode(JSON.parse(ARGV[0])["body"].tr('+', ' ')).gsub(/^code\=/,'')
template = JSON.parse(payload)
cfndsl   = CloudFormation.new(template)
dsl = Render.new(cfndsl).cfn_to_cfndsl

dsl = "require 'cfndsl'\n\n" \
      "# Generated by https://cfnflip.com/\n" \
      "# The 'cfndsl' gem is required. Type `gem install cfndsl` to install it.\n\n" \
      "#{dsl.gsub(/^CloudFormation\ do/, 'template = CloudFormation do')}\n" \
      "puts template.to_json"

puts dsl

There are more eloquent ways to parse POST data, but I didn’t include the required gems in the deployment package to avoid as much bloat as possible. Ruby can be notoriously slow. I broke down the cfn2dsl functionality to just rendering a template from a given string and outputting the result.
The NodeJS wrapper function looks like this:

const exec = require('child_process').exec;

exports.handler = function(event, context, callback) {
    const child = exec('./lambdaRuby.rb ' + '\'' + JSON.stringify(event) + '\'', function(error, stdout, stderr) {
        const response = {
          statusCode: 200,
          headers: {
            'Access-Control-Allow-Origin': '*',
            'Content-Type': 'text/html'
          },
          body: stdout
        };
        callback(null, response);
    });
    child.stdout.on('data', console.log);
    child.stderr.on('data', console.error);
}

The NodeJS function spawns the Ruby script as a child process, passing a JSON-encoded string as argument to the Ruby script, with a function that waits for the child process to terminate and fetch stdout. The output is then modified so it’s ready to run out of the box and returned.
Last but not least, I needed to create the front-end which consists of a static HTML page and a Lambda function that returns the content:

'use strict';

var fs = require('fs');

module.exports.frontPage = (event, context, callback) => {
  var html = fs.readFileSync("index.html", "utf8");
  const response = {
    statusCode: 200,
    headers: {
      'Access-Control-Allow-Origin': '*',
      'Content-Type': 'text/html',
    },
    body: html
  };

  callback(null, response);
};

The Lambda function reads the contents of index.html and returns it.

Anatomy of API Gateway

With the Lambda functions in place, it is time to build an API gateway that terminates different end-points to their corresponding Lambda functions. We have a Lambda for the main page, and two Lambda’s to convert the submitted templates to a different format. I will highlight some parts of the CloudFormation template that are key. If you’re interested in the full template you can find it on GitHub.
First we create our API Gateway:

  ApiGatewayRestApi:
    Type: AWS::ApiGateway::RestApi
    Properties:
      Name: cfnflip

We create our /cfn2dsl/ endpoint as follows:

  ApiGatewayResourceRuby:
    Type: AWS::ApiGateway::Resource
    Properties:
      ParentId: !GetAtt 'ApiGatewayRestApi.RootResourceId'
      PathPart: cfn2dsl
      RestApiId: !Ref 'ApiGatewayRestApi'

For each Lambda I’ve created a log group to log to a location that’s predictable and consistent. In this case the log group name is identical to the name of the Lambda function. In case of the Ruby Lambda function the log group resource looks like this:

  CfnFlipRubyLogGroup:
    Type: AWS::Logs::LogGroup
    Properties:
      RetentionInDays: 7
      LogGroupName: /aws/lambda/cfn2dsl

The LogGroupName determines where in CloudWatch you’ll be able to find back the log files, and we don’t want to keep them forever so we throw away the logs after 7 days. Remember the LogGroupName. We’ll get back to it when creating the IAM AssumeRole policy.
To deploy the Lambda function we need to create an AWS::Lambda::Function resource:

  CfnFlipRubyLambdaFunction:
    Type: AWS::Lambda::Function
    Properties:
      Code:
        S3Bucket: !ImportValue 'CfnFlipCloudformationBucket'
        S3Key: cfn2dslb538066ac5e095501bfe89eccc015a7f.zip
      Handler: ruby.handler
      FunctionName: cfn2dsl
      MemorySize: 256
      Role: !GetAtt 'IamRoleLambdaExecution.Arn'
      Runtime: nodejs6.10
      Timeout: 10
    DependsOn:
      - IamRoleLambdaExecution
      - CfnFlipRubyLogGroup

As you can see we import the bucket from the dependency stack and pass it to S3Bucket. In our deployment package we’ve created ruby.js, which we use as the function’s handler. MemorySize is all about balance. The higher the memory size, the more CPU the function has access to, so it’s all about balance between speed and cost. For our Ruby lambda function 256 MB of memory provided the best balance. When sizing Lambda functions you have to keep in mind that smaller sizing might mean more costs per invocation, and that sizing high might mean the increased speed doesn’t justify the additional costs.
Then, finally, we need to create a POST method for our endpoint:

  ApiGatewayMethodRubyPost:
    Type: AWS::ApiGateway::Method
    Properties:
      HttpMethod: POST
      RequestParameters: {}
      ResourceId: !Ref 'ApiGatewayResourceRuby'
      RestApiId: !Ref 'ApiGatewayRestApi'
      AuthorizationType: NONE
      Integration:
        RequestTemplates:
          application/x-www-form-urlencoded: "#set($allParams = $input.params())\n\
            {\n  \"params\" : {\n    #foreach($type in $allParams.keySet())\n    #set($params\
            \ = $allParams.get($type))\n    \"$type\" : {\n      #foreach($paramName\
            \ in $params.keySet())\n      \"$paramName\" : \"$util.escapeJavaScript($params.get($paramName))\"\
            \n      #if($foreach.hasNext),#end\n      #end\n    }\n    #if($foreach.hasNext),#end\n\
            \    #end\n  }\n}\n"
        IntegrationHttpMethod: POST
        Type: AWS_PROXY
        Uri: !Join
          - ''
          - - 'arn:aws:apigateway:'
            - !Ref 'AWS::Region'
            - :lambda:path/2015-03-31/functions/
            - !GetAtt 'CfnFlipRubyLambdaFunction.Arn'
            - /invocations
      MethodResponses: []

The RequestTemplate in the above snippet transforms the request into an event body that we can use in the function, and then we pass it on to the Lambda function.
So did you remember to LogGroupName ? It will come into play now. As you can see in the Lambda function snippet, there is a reference to the IamRoleLambdaExecution role. This role provides the Lambda function with relevant permissions to perform actions on your behalf. You want to limit these privileges to the least amount required for the function to work. Our Lambda’s fortunately don’t require a lot of permissions. It doesn’t interact with any AWS service nor does it require any VPC to talk to. So what does it need? It creates CloudWatch log streams and write to them! So let’s have a look:

  IamRoleLambdaExecution:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Statement:
          - Action:
              - sts:AssumeRole
            Effect: Allow
            Principal:
              Service:
                - lambda.amazonaws.com
        Version: '2012-10-17'
      Policies:
        - PolicyDocument:
            Statement:
              - Action:
                  - logs:CreateLogStream
                Effect: Allow
                Resource:
                  - !Sub 'arn:aws:logs:${AWS::Region}:${AWS::AccountId}:log-group:/aws/lambda/cfnflip:*'
                  - !Sub 'arn:aws:logs:${AWS::Region}:${AWS::AccountId}:log-group:/aws/lambda/pyflip:*'
                  - !Sub 'arn:aws:logs:${AWS::Region}:${AWS::AccountId}:log-group:/aws/lambda/cfn2dsl:*'
              - Action:
                  - logs:PutLogEvents
                Effect: Allow
                Resource:
                  - !Sub 'arn:aws:logs:${AWS::Region}:${AWS::AccountId}:log-group:/aws/lambda/cfnflip:*:*'
                  - !Sub 'arn:aws:logs:${AWS::Region}:${AWS::AccountId}:log-group:/aws/lambda/pyflip:*:*'
                  - !Sub 'arn:aws:logs:${AWS::Region}:${AWS::AccountId}:log-group:/aws/lambda/cfn2dsl:*:*'
            Version: '2012-10-17'
          PolicyName: !Join
            - '-'
            - - cfnflip
              - lambda
      Path: /
      RoleName: !Join
        - '-'
        - - cfnflip
          - eu-west-1
          - lambdaRole

We allow the Lambda function to assume this role with all the permissions listed in the PolicyDocument. We grant the CreateLogStream and the PutLogEvents privilege to the LogGroups we’ve created before. This is all our functions require.

Epilogue

It’s 2:27 AM and I’m typing this very epilogue. It took me longer to write this blog post than it took me to create the MVP for cfnflip.com. It was great fun to make and I’m already brewing some new ideas. My esteemed colleague Mark is toying with the thought of whipping up some Terraform transformer. If you have any cool ideas do let us know, or better yet: Fork the project and submit a PR for your improvements.