Analyzing API Gateway Access Logs with AWS Elasticsearch Service

In my blog AWS Elasticsearch Service with Firehose Delivery Stream we saw how easy it is to setup an Elasticsearch cluster, ingesting data data from Firehose and creating dashboards in Kibana. This time we use the same architecture to ingest access logs from AWS ApiGateway and analyze the data in Kibana.

API Gateway Account

In every AWS account there is a single API Gateway (APIGW) service. APIGW can hosts multiple RestApi instances. Each RestApi instance contains multiple stages like dev, test or prod. To prepare for logging, the single APIGW service instance has to have permissions to access CloudWatch logs. The configuration is easy. Create a role that the APIGW service can assume, and create an ‘AWS::ApiGateway::Account’ that points to the role.

<span style="color:#bbb">  </span><span style="font-weight:bold">CloudWatchLogRole</span>:<span style="color:#bbb">
</span><span style="color:#bbb"></span><span style="color:#bbb">    </span><span style="font-weight:bold">Type</span>:<span style="color:#bbb"> </span>AWS::IAM::Role<span style="color:#bbb">
</span><span style="color:#bbb">    </span><span style="font-weight:bold">Properties</span>:<span style="color:#bbb">
</span><span style="color:#bbb"></span><span style="color:#bbb">      </span><span style="font-weight:bold">AssumeRolePolicyDocument</span>:<span style="color:#bbb">
</span><span style="color:#bbb"></span><span style="color:#bbb">        </span><span style="font-weight:bold">Version</span>:<span style="color:#bbb"> </span><span style="color:#b84">'2012-10-17'</span><span style="color:#bbb">
</span><span style="color:#bbb">        </span><span style="font-weight:bold">Statement</span>:<span style="color:#bbb">
</span><span style="color:#bbb"></span><span style="color:#bbb">        </span>- <span style="font-weight:bold">Effect</span>:<span style="color:#bbb"> </span>Allow<span style="color:#bbb">
</span><span style="color:#bbb">          </span><span style="font-weight:bold">Principal</span>:<span style="color:#bbb">
</span><span style="color:#bbb"></span><span style="color:#bbb">            </span><span style="font-weight:bold">Service</span>:<span style="color:#bbb">
</span><span style="color:#bbb"></span><span style="color:#bbb">            </span>- apigateway.amazonaws.com<span style="color:#bbb">
</span><span style="color:#bbb">          </span><span style="font-weight:bold">Action</span>:<span style="color:#bbb"> </span>sts:AssumeRole<span style="color:#bbb">
</span><span style="color:#bbb">      </span><span style="font-weight:bold">Path</span>:<span style="color:#bbb"> </span>/<span style="color:#bbb">
</span><span style="color:#bbb">      </span><span style="font-weight:bold">ManagedPolicyArns</span>:<span style="color:#bbb">
</span><span style="color:#bbb"></span><span style="color:#bbb">      </span>- arn:aws:iam::aws:policy/service-role/AmazonAPIGatewayPushToCloudWatchLogs<span style="color:#bbb">
</span><span style="color:#bbb">
</span><span style="color:#bbb">  </span><span style="font-weight:bold">Account</span>:<span style="color:#bbb">
</span><span style="color:#bbb"></span><span style="color:#bbb">    </span><span style="font-weight:bold">Type</span>:<span style="color:#bbb"> </span>AWS::ApiGateway::Account<span style="color:#bbb">
</span><span style="color:#bbb">    </span><span style="font-weight:bold">Properties</span>:<span style="color:#bbb">
</span><span style="color:#bbb"></span><span style="color:#bbb">      </span><span style="font-weight:bold">CloudWatchRoleArn</span>:<span style="color:#bbb"> </span>!GetAtt<span style="color:#bbb"> </span><span style="color:#b84">'CloudWatchLogRole.Arn'</span><span style="color:#bbb">
</span>

API Gateway Deployment

A RestApi instance contains a Deployment that specifies, among others, logging and tracing configuration. To configure access logs, we have to configure the AccessLogs settings and define a log line Format. The RestApi will use this format to create access log lines in CloudWatch logs. There are several standard formats to choose from, but you can also specify a standard format like the JSON below.

<span style="color:#bbb">  </span><span style="font-weight:bold">ApiGatewayDeployment</span>:<span style="color:#bbb">
</span><span style="color:#bbb"></span><span style="color:#bbb">    </span><span style="font-weight:bold">Type</span>:<span style="color:#bbb"> </span>AWS::ApiGateway::Deployment<span style="color:#bbb">
</span><span style="color:#bbb">    </span><span style="font-weight:bold">Properties</span>:<span style="color:#bbb">
</span><span style="color:#bbb"></span><span style="color:#bbb">      </span><span style="font-weight:bold">RestApiId</span>:<span style="color:#bbb"> </span>!Ref<span style="color:#bbb"> </span>RestAPIv1<span style="color:#bbb">
</span><span style="color:#bbb">      </span><span style="font-weight:bold">StageName</span>:<span style="color:#bbb"> </span>dev<span style="color:#bbb">
</span><span style="color:#bbb">      </span><span style="font-weight:bold">Stagecontent</span>:<span style="color:#bbb">
</span><span style="color:#bbb"></span><span style="color:#bbb">        </span><span style="font-weight:bold">DataTraceEnabled</span>:<span style="color:#bbb"> </span><span style="font-weight:bold">true</span><span style="color:#bbb">
</span><span style="color:#bbb">        </span><span style="font-weight:bold">LoggingLevel</span>:<span style="color:#bbb"> </span>INFO<span style="color:#bbb">
</span><span style="color:#bbb">        </span><span style="font-weight:bold">MetricsEnabled</span>:<span style="color:#bbb"> </span><span style="font-weight:bold">true</span><span style="color:#bbb">
</span><span style="color:#bbb">        </span><span style="font-weight:bold">TracingEnabled</span>:<span style="color:#bbb"> </span><span style="font-weight:bold">true</span><span style="color:#bbb">
</span><span style="color:#bbb">        </span><span style="font-weight:bold">MethodSettings</span>:<span style="color:#bbb">
</span><span style="color:#bbb"></span><span style="color:#bbb">        </span>- <span style="font-weight:bold">LoggingLevel</span>:<span style="color:#bbb"> </span>INFO<span style="color:#bbb">
</span><span style="color:#bbb">          </span><span style="font-weight:bold">ResourcePath</span>:<span style="color:#bbb"> </span>/*<span style="color:#bbb">
</span><span style="color:#bbb">          </span><span style="font-weight:bold">HttpMethod</span>:<span style="color:#bbb"> </span><span style="color:#b84">'*'</span><span style="color:#bbb">
</span><span style="color:#bbb">        </span><span style="font-weight:bold">AccessLogSetting</span>:<span style="color:#bbb">
</span><span style="color:#bbb"></span><span style="color:#bbb">          </span><span style="font-weight:bold">DestinationArn</span>:<span style="color:#bbb"> </span>!GetAtt<span style="color:#bbb"> </span><span style="color:#b84">'CloudWatchAccessLogGroup.Arn'</span><span style="color:#bbb">
</span><span style="color:#bbb">          </span><span style="font-weight:bold">Format</span>:<span style="color:#bbb"> </span>><span style="color:#b84">-
</span><span style="color:#b84">           </span><span style="color:#b84"> </span><span style="color:#b84">{</span><span style="color:#bbb">
</span><span style="color:#bbb">            </span><span style="color:#b84">"requestId"</span>:<span style="color:#b84">"$context.requestId"</span>,<span style="color:#bbb">
</span><span style="color:#bbb">            </span><span style="font-weight:bold">"ip": </span><span style="color:#b84">"$context.identity.sourceIp"</span>,<span style="color:#bbb">
</span><span style="color:#bbb">            </span><span style="color:#b84">"caller"</span>:<span style="color:#b84">"$context.identity.caller"</span>,<span style="color:#bbb">
</span><span style="color:#bbb">            </span><span style="color:#b84">"user"</span>:<span style="color:#b84">"$context.identity.user"</span>,<span style="color:#bbb">
</span><span style="color:#bbb">            </span><span style="color:#b84">"requestTime"</span>:<span style="color:#b84">"$context.requestTime"</span>,<span style="color:#bbb">
</span><span style="color:#bbb">            </span><span style="color:#b84">"httpMethod"</span>:<span style="color:#b84">"$context.httpMethod"</span>,<span style="color:#bbb">
</span><span style="color:#bbb">            </span><span style="color:#b84">"resourcePath"</span>:<span style="color:#b84">"$context.resourcePath"</span>,<span style="color:#bbb">
</span><span style="color:#bbb">            </span><span style="color:#b84">"status"</span>:<span style="color:#b84">"$context.status"</span>,<span style="color:#bbb">
</span><span style="color:#bbb">            </span><span style="color:#b84">"protocol"</span>:<span style="color:#b84">"$context.protocol"</span>,<span style="color:#bbb">
</span><span style="color:#bbb">            </span><span style="color:#b84">"responseLength"</span>:<span style="color:#b84">"$context.responseLength"</span><span style="color:#bbb">
</span><span style="color:#bbb">            </span>}<span style="color:#bbb">
</span>

When the gateway is accessed by typing make hello or make error, the RestApi will be invoked and the following access logs will appear:

Error:

{
    <span style="color:#000080">"requestId"</span>: <span style="color:#b84">"3b864b4b-ea50-11e8-b859-9bd3a7d2b23c"</span>,
    <span style="color:#000080">"ip"</span>: <span style="color:#b84">"217.19.26.243"</span>,
    <span style="color:#000080">"caller"</span>: <span style="color:#b84">"-"</span>,
    <span style="color:#000080">"user"</span>: <span style="color:#b84">"-"</span>,
    <span style="color:#000080">"requestTime"</span>: <span style="color:#b84">"17/Nov/2018:10:04:55 +0000"</span>,
    <span style="color:#000080">"httpMethod"</span>: <span style="color:#b84">"GET"</span>,
    <span style="color:#000080">"resourcePath"</span>: <span style="color:#b84">"/error"</span>,
    <span style="color:#000080">"status"</span>: <span style="color:#b84">"502"</span>,
    <span style="color:#000080">"protocol"</span>: <span style="color:#b84">"HTTP/1.1"</span>,
    <span style="color:#000080">"responseLength"</span>: <span style="color:#b84">"36"</span>
}

Hello:

{
    <span style="color:#000080">"requestId"</span>: <span style="color:#b84">"3a71f4fa-ea50-11e8-b4f5-8116b21e9431"</span>,
    <span style="color:#000080">"ip"</span>: <span style="color:#b84">"217.19.26.243"</span>,
    <span style="color:#000080">"caller"</span>: <span style="color:#b84">"-"</span>,
    <span style="color:#000080">"user"</span>: <span style="color:#b84">"-"</span>,
    <span style="color:#000080">"requestTime"</span>: <span style="color:#b84">"17/Nov/2018:10:04:53 +0000"</span>,
    <span style="color:#000080">"httpMethod"</span>: <span style="color:#b84">"GET"</span>,
    <span style="color:#000080">"resourcePath"</span>: <span style="color:#b84">"/hello"</span>,
    <span style="color:#000080">"status"</span>: <span style="color:#b84">"200"</span>,
    <span style="color:#000080">"protocol"</span>: <span style="color:#b84">"HTTP/1.1"</span>,
    <span style="color:#000080">"responseLength"</span>: <span style="color:#b84">"14"</span>
}

CloudWatch Logs Subscription Filter

A CloudWatch Logs Subscription Filter (CLSF) sends log events to Kinesis stream, Kinesis Data Firehose delivery stream, or Lambda function. To access these resources, the CLSF has to have permissions. The configuration is easy. Create a role that CloudWatch can assume and configure that role on the ‘AWS::Logs::SubscriptionFilter’ resource. The filter is empty so all log lines will be processed.

<span style="color:#bbb"> </span><span style="font-weight:bold">CloudWatchLogSubscriptionRole</span>:<span style="color:#bbb">
</span><span style="color:#bbb"></span><span style="color:#bbb">    </span><span style="font-weight:bold">Type</span>:<span style="color:#bbb"> </span>AWS::IAM::Role<span style="color:#bbb">
</span><span style="color:#bbb">    </span><span style="font-weight:bold">Properties</span>:<span style="color:#bbb">
</span><span style="color:#bbb"></span><span style="color:#bbb">      </span><span style="font-weight:bold">AssumeRolePolicyDocument</span>:<span style="color:#bbb">
</span><span style="color:#bbb"></span><span style="color:#bbb">        </span><span style="font-weight:bold">Version</span>:<span style="color:#bbb"> </span><span style="color:#b84">'2012-10-17'</span><span style="color:#bbb">
</span><span style="color:#bbb">        </span><span style="font-weight:bold">Statement</span>:<span style="color:#bbb">
</span><span style="color:#bbb"></span><span style="color:#bbb">        </span>- <span style="font-weight:bold">Effect</span>:<span style="color:#bbb"> </span>Allow<span style="color:#bbb">
</span><span style="color:#bbb">          </span><span style="font-weight:bold">Principal</span>:<span style="color:#bbb">
</span><span style="color:#bbb"></span><span style="color:#bbb">            </span><span style="font-weight:bold">Service</span>:<span style="color:#bbb">
</span><span style="color:#bbb"></span><span style="color:#bbb">            </span>- logs.eu-west<span style="color:#099">-1.</span>amazonaws.com<span style="color:#bbb">
</span><span style="color:#bbb">          </span><span style="font-weight:bold">Action</span>:<span style="color:#bbb"> </span>sts:AssumeRole<span style="color:#bbb">
</span><span style="color:#bbb">      </span><span style="font-weight:bold">Path</span>:<span style="color:#bbb"> </span>/<span style="color:#bbb">
</span><span style="color:#bbb">      </span><span style="font-weight:bold">Policies</span>:<span style="color:#bbb">
</span><span style="color:#bbb"></span><span style="color:#bbb">      </span>- <span style="font-weight:bold">PolicyName</span>:<span style="color:#bbb"> </span>Allow<span style="color:#bbb">
</span><span style="color:#bbb">        </span><span style="font-weight:bold">PolicyDocument</span>:<span style="color:#bbb">
</span><span style="color:#bbb"></span><span style="color:#bbb">          </span><span style="font-weight:bold">Statement</span>:<span style="color:#bbb">
</span><span style="color:#bbb"></span><span style="color:#bbb">          </span>- <span style="font-weight:bold">Effect</span>:<span style="color:#bbb"> </span>Allow<span style="color:#bbb">
</span><span style="color:#bbb">            </span><span style="font-weight:bold">Action</span>:<span style="color:#bbb">
</span><span style="color:#bbb"></span><span style="color:#bbb">            </span>- firehose:*<span style="color:#bbb">
</span><span style="color:#bbb">            </span><span style="font-weight:bold">Resource</span>:<span style="color:#bbb">
</span><span style="color:#bbb"></span><span style="color:#bbb">            </span>- <span style="color:#b84">'*'</span><span style="color:#bbb">
</span><span style="color:#bbb">      </span><span style="font-weight:bold">ManagedPolicyArns</span>:<span style="color:#bbb">
</span><span style="color:#bbb"></span><span style="color:#bbb">      </span>- arn:aws:iam::aws:policy/service-role/AmazonAPIGatewayPushToCloudWatchLogs<span style="color:#bbb">
</span><span style="color:#bbb">
</span><span style="color:#bbb">  </span><span style="font-weight:bold">CloudWatchLogSubscription</span>:<span style="color:#bbb">
</span><span style="color:#bbb"></span><span style="color:#bbb">    </span><span style="font-weight:bold">Type</span>:<span style="color:#bbb"> </span>AWS::Logs::SubscriptionFilter<span style="color:#bbb">
</span><span style="color:#bbb">    </span><span style="font-weight:bold">Properties</span>:<span style="color:#bbb">
</span><span style="color:#bbb"></span><span style="color:#bbb">      </span><span style="font-weight:bold">DestinationArn</span>:<span style="color:#bbb"> </span>!GetAtt<span style="color:#bbb"> </span>Deliverystream.Arn<span style="color:#bbb">
</span><span style="color:#bbb">      </span><span style="font-weight:bold">FilterPattern</span>:<span style="color:#bbb"> </span><span style="color:#b84">''</span><span style="color:#bbb">
</span><span style="color:#bbb">      </span><span style="font-weight:bold">LogGroupName</span>:<span style="color:#bbb"> </span>!Ref<span style="color:#bbb"> </span>CloudWatchAccessLogGroup<span style="color:#bbb">
</span><span style="color:#bbb">      </span><span style="font-weight:bold">RoleArn</span>:<span style="color:#bbb"> </span>!GetAtt<span style="color:#bbb"> </span>CloudWatchLogSubscriptionRole.Arn<span style="color:#bbb">
</span>

Decompressing, Decoding and Selecting

When we look at a received CloudWatch event, we see that the logEvents field contain a single entry, which is the access log. The access log is stored in a JSON field as a string so we have to decode the field to JSON. When the message field has been decoded, we select only the first element of the logEvents field, which is the access log.

{
   <span style="color:#000080">"messageType"</span>: <span style="color:#b84">"DATA_MESSAGE"</span>,
   <span style="color:#000080">"owner"</span>: <span style="color:#b84">"612483924670"</span>,
   <span style="color:#000080">"logGroup"</span>: <span style="color:#b84">"api-gateway-access-logs-dev"</span>,
   <span style="color:#000080">"logStream"</span>: <span style="color:#b84">"8d5e957f297893487bd98fa830fa6413"</span>,
   <span style="color:#000080">"subscriptionFilters"</span>: [
       <span style="color:#b84">"blog-aws-elasticsearch-firehose-api-gw-example-elasticsearch-CloudWatchLogSubscription-95E57LT45IU0"</span>
   ],
   <span style="color:#000080">"logEvents"</span>: [
       {
           <span style="color:#000080">"id"</span>: <span style="color:#b84">"34397996112421660129451470773032598202872829922794405888"</span>,
           <span style="color:#000080">"timestamp"</span>: <span style="color:#099">1542459492102</span>,
           <span style="color:#000080">"message"</span>: <span style="color:#b84">"{ \"requestId\":\"70939afd-ea68-11e8-add9-17a2f8edb26e\", \"ip\": \"217.19.26.243\", \"caller\":\"-\", \"user\":\"-\", \"requestTime\":\"17/Nov/2018:12:58:12 +0000\", \"httpMethod\":\"GET\", \"resourcePath\":\"/error\", \"status\":\"502\", \"protocol\":\"HTTP/1.1\", \"responseLength\":\"36\" }"</span>
       }
   ]
}

Firehose Processor

In order to correctly index log lines we have to post-process the log lines before they are published to Elasticsearch. Firehose supports processing messages before delivery by means of AWS Lambda. CloudWatch events publishes the log lines in Gzip format to Elasticsearch. The cloudwatch event also contains information that we are not interested in, we are only interested in the fields of the access logs. We use the following lambda to process the logs:
processor:

<span style="font-weight:bold">from</span> <span style="color:#555">base64</span> <span style="font-weight:bold">import</span> b64encode, b64decode
<span style="font-weight:bold">import</span> <span style="color:#555">json</span>
<span style="font-weight:bold">import</span> <span style="color:#555">gzip</span>

<span style="font-weight:bold">def</span> <span style="color:#900;font-weight:bold">decompress</span>(data):
    <span style="font-weight:bold">return</span> gzip<span style="font-weight:bold">.</span>decompress(data)

<span style="font-weight:bold">def</span> <span style="color:#900;font-weight:bold">decode_record</span>(data: <span style="color:#999">dict</span>) <span style="font-weight:bold">-</span><span style="font-weight:bold">></span> <span style="color:#999">dict</span>:
    x <span style="font-weight:bold">=</span> decompress(b64decode(data[<span style="color:#b84"></span><span style="color:#b84">'</span><span style="color:#b84">data</span><span style="color:#b84">'</span>]))
    <span style="font-weight:bold">return</span> json<span style="font-weight:bold">.</span>loads(x<span style="font-weight:bold">.</span>decode(<span style="color:#b84"></span><span style="color:#b84">'</span><span style="color:#b84">utf8</span><span style="color:#b84">'</span>))

<span style="font-weight:bold">def</span> <span style="color:#900;font-weight:bold">handler</span>(event, context):
    records <span style="font-weight:bold">=</span> event[<span style="color:#b84"></span><span style="color:#b84">'</span><span style="color:#b84">records</span><span style="color:#b84">'</span>]
    <span style="font-weight:bold">for</span> record <span style="font-weight:bold">in</span> records:
        record<span style="font-weight:bold">.</span>pop(<span style="color:#b84"></span><span style="color:#b84">'</span><span style="color:#b84">approximateArrivalTimestamp</span><span style="color:#b84">'</span>, None)
        decoded <span style="font-weight:bold">=</span> decode_record(record)
        <span style="font-weight:bold">if</span> decoded[<span style="color:#b84"></span><span style="color:#b84">'</span><span style="color:#b84">messageType</span><span style="color:#b84">'</span>] <span style="font-weight:bold">==</span> <span style="color:#b84"></span><span style="color:#b84">"</span><span style="color:#b84">DATA_MESSAGE</span><span style="color:#b84">"</span>:
            event <span style="font-weight:bold">=</span> decoded[<span style="color:#b84"></span><span style="color:#b84">'</span><span style="color:#b84">logEvents</span><span style="color:#b84">'</span>][<span style="color:#099">0</span>]
            event<span style="font-weight:bold">.</span>update({<span style="color:#b84"></span><span style="color:#b84">'</span><span style="color:#b84">message</span><span style="color:#b84">'</span>: json<span style="font-weight:bold">.</span>loads(event[<span style="color:#b84"></span><span style="color:#b84">'</span><span style="color:#b84">message</span><span style="color:#b84">'</span>])})
            msg <span style="font-weight:bold">=</span> b64encode(<span style="color:#999">bytes</span>(json<span style="font-weight:bold">.</span>dumps(event), <span style="color:#b84"></span><span style="color:#b84">'</span><span style="color:#b84">utf-8</span><span style="color:#b84">'</span>))<span style="font-weight:bold">.</span>decode(<span style="color:#b84"></span><span style="color:#b84">'</span><span style="color:#b84">ascii</span><span style="color:#b84">'</span>)
            record<span style="font-weight:bold">.</span>update({<span style="color:#b84"></span><span style="color:#b84">'</span><span style="color:#b84">data</span><span style="color:#b84">'</span>: msg})
            record<span style="font-weight:bold">.</span>update({<span style="color:#b84"></span><span style="color:#b84">'</span><span style="color:#b84">result</span><span style="color:#b84">'</span>: <span style="color:#b84"></span><span style="color:#b84">'</span><span style="color:#b84">Ok</span><span style="color:#b84">'</span>}) <span style="color:#998;font-style:italic"># Ok, Dropped, ProcessingFailed</span>
        <span style="font-weight:bold">else</span>:
            record<span style="font-weight:bold">.</span>update({<span style="color:#b84"></span><span style="color:#b84">'</span><span style="color:#b84">result</span><span style="color:#b84">'</span>: <span style="color:#b84"></span><span style="color:#b84">'</span><span style="color:#b84">Dropped</span><span style="color:#b84">'</span>}) <span style="color:#998;font-style:italic"># Ok, Dropped, ProcessingFailed</span>

    <span style="font-weight:bold">return</span> {<span style="color:#b84"></span><span style="color:#b84">'</span><span style="color:#b84">records</span><span style="color:#b84">'</span>: records}

Example

The example project shows how to configure a project to create an elasticsearch cluster and to ingest API Gateway access logs. The example can be deployed with make merge-lambda && make merge-swagger && make deploy and removed with make delete. To publish access the API Gateway type make hello and make error to get some entries in the access logs.

Kibana

Log into the ‘AWS Console’, then the ‘Elasticsearch service dashboard’, and click on the Kibana URL. Once logged in, click on ‘discover’ and create a new index pattern with the name example-*. Click on ‘discover’ another time and you should see data. If not, type make hello and make error a couple of times in the console to have data available in ES. To search for data type message.status:50* or message.status:20* in the search bar.

Try to create a visualization and dashboards with the access logs you have available. If you have read my previous blog post about AWS Elasticsearch Service with Firehose Delivery Stream it shouldn’t be difficult

Conclusion

In this example we have deployed a Elasticsearch service, ingested access logs, and created a dashboard. Elasticsearch is perfect for analyzing access logs to get real time information about how an API is performing. Dashboards provide aggregated data and provide insights that can be used to make changes to the platform. Next time we’ll look at ingesting CloudTrail logs and get insights to who is doing what in the AWS account.

Share this article: Tweet this post / Post on LinkedIn