AWS CloudWatch logs is an useful logging system, but in practice we run into two quircks: It does not allow you to set a default retention period for newly created log groups. Secondly it does not delete empty log streams that are older than the retention period. In this blog we introduce a utility that will fix these two issues. It can be installed as a CLI or deployed as an AWS Lambda.
This utility provides two functions:
This utility can be used from the command line. You can also install it as an AWS Lambda function and have your logs kept in order, NoOps style!
to install the utility, type:
pip install aws-cloudwatch-log-minder
to set the default retention period on log groups without one, type:
cwlog-minder --dry-run set-log-retention --days 30
This will show you log groups which do not have a retention period set. Remove the
To delete empty log streams older than the retention period, type:
cwlog-minder --dry-run delete-empty-log-streams
This will show you which empty log streams will be deleted. Remove the
these streams will be deleted.
To deploy the utility as an AWS Lambda, type:
git clone https://github.com/binxio/aws-cloudwatch-log-minder.git cd aws-cloudwatch-log-minder aws cloudformation create-stack \ --capabilities CAPABILITY_IAM \ --stack-name aws-cloudwatch-log-minder \ --template-body file://./cloudformation/aws-cloudwatch-log-minder.yaml aws cloudformation wait stack-create-complete --stack-name aws-cloudwatch-log-minder
This will install the utility in your AWS account and run every hour. The added benefit is that the maintenance of the log groups is automated and executed in parallel.
This utility provides a work-around for two quircks you will encounter when working with AWS CloudWatch. it will set a retention policy on log groups created outside of control of the user, and it will delete empty log streams older than the retention period. It would of course be cool if AWS would fix the latter.