How to update an EC2 instance with volume attachments using CloudFormation

How do you update an EC2 instance with volume attachments using CloudFormation? When you have a stateful server with one more volumes attached to it in your infrastructure, the AWS::EC2::VolumeAttachment resource makes it impossible to update the instance. In this blog I will show you how to configure volume attachments that allow the instance to be updated using an auto scaling group and the EC2 Volume manager.

When you try to update an AWS::EC2::VolumeAttachment, CloudFormation will give the following error:

ERROR: Update to resource type AWS::EC2::VolumeAttachment is not supported.

This prevents you from updating the AMI, or any other property that requires a replacement of the instance.

I solved this problem by changing the resource definition of the stateful machine to a single instance auto scaling group and use the EC2 volume manager utility to dynamically attache volumes upon the start of instances.

To implement this, you have to:

  1. deploy the ec2 volume manager
  2. change from an instance to an auto scaling group
  3. enable rolling updates on the auto scaling group
  4. signal successful startup to CloudFormation
  5. attach ec2 volume manager tags to volumes and instances

Deploy the EC2 volume manager

Deploy the ec2-volume-manager using the following commands:

git clone https://github.com/binxio/ec2-volume-manager.git
cd ec2-ec2-volume-manager
aws cloudformation deploy \
        --capabilities CAPABILITY_IAM \
        --stack-name ec2-volume-manager \
        --template ./cloudformation/ec2-volume-manager.yaml

Change instance to auto scaling group

Change the definition of your persistence instance, from a AWS::EC2::Instance to an single instance auto scaling group. From:

StatefulServer:
  Type: AWS::EC2::Instance
  Properties:
    SubnetId: !Select [0, !Ref 'Subnets']
    LaunchTemplate:
      LaunchTemplateId: !Ref 'LaunchTemplate'
      Version: !GetAtt 'LaunchTemplate.LatestVersionNumber'

to:

AutoScalingGroup:
    Type: AWS::AutoScaling::AutoScalingGroup
    Properties:
      AutoScalingGroupName: !Ref AWS::StackName
      VPCZoneIdentifier:
        - !Select [0, !Ref 'Subnets']
      LaunchTemplate:
        LaunchTemplateId: !Ref 'LaunchTemplate'
        Version: !GetAtt 'LaunchTemplate.LatestVersionNumber'
      MinSize: '0'
      MaxSize: '1'
      DesiredCapacity: '1'

Enable rolling update

Instruct CloudFormation to perform a rolling update to replace the instances:

AutoScalingGroup:
  Type: AWS::AutoScaling::AutoScalingGroup
  Properties:
    ...
    UpdatePolicy:
      AutoScalingRollingUpdate:
        MinInstancesInService: 0
        MaxBatchSize: 1
        WaitOnResourceSignals: true

When the instance needs to be replaced, CloudFormation will update the auto scaling group by destroying the old instance first, followed by the creation of a new instance.

Signal successful startup

CloudFormation will wait until the instance reports it has succesfully started. This is done by the cfn-signal at the end of the boot commands in the launch template.

LaunchTemplate:
  Type: AWS::EC2::LaunchTemplate
  Properties:
    ...
    UserData: !Base64
      Fn::Sub: |
        bootcmd:
          ...
          -  /opt/aws/bin/cfn-signal --stack ${AWS::StackName} --resource AutoScalingGroup

Without this signal, the update is rolled back.

Attach tags on volumes and instance

The EC2 volume manager utility automatically attaches volumes to instances with the same tag values. When an instance with the tag ec2-volume-manager-attachments reaches the state running, it will attach all volumes with the same tag value. When the instance is stopped or terminated, all volumes with a tag ec2-volume-manager-attachments will be detached from it.

To get the volume manager to work, tag the volumes of the instance as follows:

  Disk1:
    Type: AWS::EC2::Volume
    Properties:
      AvailabilityZone: !Sub '${AWS::Region}a'
      Size: 8
      Tags:
        - Key: ec2-volume-manager-attachment
          Value: stateful-instance-1
        - Key: device-name
          Value: xvdf

Note that the volume manager also requires the tag device-name, referring to the device name of the volume for the operating system. Next, add the ec2-volume-manager-attachments to the auto scaling group:

AutoScalingGroup:
  Type: AWS::AutoScaling::AutoScalingGroup
  Properties:
    ...
    Tags:
      - Key: ec2-volume-manager-attachment
        Value: stateful-instance-1
        PropagateAtLaunch: true

That is all that is required to enable fully automated updates of a stateful server with attached volumes.

You can see all differences when you compare the CloudFormation template of the stateful server with the template for the ec2-volume-manager solution.

Deploy the demo

A demo is available. Deploy it with:

export VPC_ID=$(aws ec2  
                --output text \
                --query 'Vpcs[?IsDefault].VpcId' describe-vpcs)
export SUBNET_IDS=$(aws ec2 describe-subnets --output text \
                --filters Name=vpc-id,Values=$(VPC_ID) \
                          Name=default-for-az,Values=true \
                --query 'join(`,`,sort_by(Subnets[?MapPublicIpOnLaunch], &AvailabilityZone)[*].SubnetId)')

aws cloudformation deploy \
        --capabilities CAPABILITY_NAMED_IAM \
        --stack-name ec2-volume-manager-demo \
        --template ./cloudformation/demo-stack.yaml \
        --parameter-overrides VPC=$VPC_ID Subnets=$SUBNET_IDS

Conclusion

When you have a stateful server with one more volumes attached to it in your infrastructure, the AWS::EC2::VolumeAttachment resource makes it impossible to update the instance. But, when you use a single instance auto scaling group in combination with the EC2 Volume manager you can!

If you want to want you can also attach a static ip address to your stateful instance. Make sure to properly mount ebs volumes during boot.

Alternative solutions

Although, we always recommend to keep your EC2 instances stateless and use managed persistent services whenever possible, we have successfully used the EC2 volume manager for IBM MQ and Microsoft SQL Server instances.

If you do not like the magic of the EC2 volume manager, you can also attach the volumes in the boot script of the persistent instance.

Mark van Holsteijn is a senior software systems architect, and CTO of binx.io. He is passionate about removing waste in the software delivery process and keeping things clear and simple.
Share this article: Tweet this post / Post on LinkedIn