AWS CloudWatch logs is an useful logging system, but in practice we run into two quircks: It does not allow you
to set a default retention period for newly created log groups. Secondly it does not delete empty log streams
that are older than the retention period. In this blog we introduce a utility that will fix these two issues. It can
be installed as a CLI or deployed as an AWS Lambda.
This utility provides two functions:
- apply a retention policy on log groups that do not have one.
- remove empty log streams older than the retention period of the log group.
This utility can be used from the command line. You can also install it as an AWS Lambda function and have your
logs kept in order, NoOps style!
Install the utility
to install the utility, type:
pip install aws-cloudwatch-log-minder
Set retention period
to set the default retention period on log groups without one, type:
cwlog-minder --dry-run set-log-retention --days 30
This will show you log groups which do not have a retention period set. Remove the
Delete empty log streams
To delete empty log streams older than the retention period, type:
cwlog-minder --dry-run delete-empty-log-streams
This will show you which empty log streams will be deleted. Remove the
these streams will be deleted.
Deploy as Lambda
To deploy the utility as an AWS Lambda, type:
git clone https://github.com/binxio/aws-cloudwatch-log-minder.git cd aws-cloudwatch-log-minder aws cloudformation create-stack \ --capabilities CAPABILITY_IAM \ --stack-name aws-cloudwatch-log-minder \ --template-body file://./cloudformation/aws-cloudwatch-log-minder.yaml aws cloudformation wait stack-create-complete --stack-name aws-cloudwatch-log-minder
This will install the utility in your AWS account and run every hour. The added benefit
is that the maintenance of the log groups is automated and executed in parallel.
This utility provides a work-around for two quircks you will encounter when working with AWS CloudWatch.
it will set a retention policy on log groups created outside of control of the user, and it will
delete empty log streams older than the retention period. It would of course be cool if
AWS would fix the latter.