How to set up an SSH tunnel to private AWS RDS and EC2 instances

on
Apr 26, 2022
in

When debugging applications in the cloud, we sometimes need to set up an SSH tunnel from our local network in order to interact with them. However, we almost never want these machines to be publicly accessible! In this post, I will explain how to create SSH tunnels to private EC2 and RDS instances without exposing any public endpoints, using aws-ssh-tunnel and a single private EC2 instance.

Requirements

In order to set up an SSH tunnel, we are going to need three things: deploy an EC2 jump server, set up the right IAM permissions for our AWS role, and configure the aws-ssh-tunnel CLI.

In my next post, I will explain what exactly aws-ssh-tunnel is doing in the background. The gist of it is that we can make use of the AWS Systems Manager StartSession API in order to forward SSH traffic to a private EC2 instance. This instance acts as a jump server that tunnels our shell commands to a remote host, such as RDS.

Setting up the jump server instance

First, let’s set up the jump server instance. This can be any instance as long as it is on a private subnet and does not allow any inbound traffic. AWS CDK has a construct specifically meant for this purpose called BastionHostLinux. Below is a working example that you can import in your CDK application:

import {
    App,
    Stack,
    Tags,
    aws_ec2 as ec2,
} from "aws-cdk-lib";

export class JumpServerStack extends Stack {

    constructor(app: App, id: string, props ? : any) {
        super(app, id, props);

        Tags.of(this).add('application', 'jump_server')

        const vpc = ec2.Vpc.fromLookup(this, "JumpServerVpc", {
            tags: {
                "vpc_id": "jump_server_vpc"
            }
        });

        const jumpServer = new ec2.BastionHostLinux(this, 'JumpServer', {
            vpc,
            blockDevices: [{
                deviceName: '/dev/sdf',
                volume: ec2.BlockDeviceVolume.ebs(10, {
                    encrypted: true,
                }),
            }],
        });
    }
}

Creating the right IAM policy

Our user will also need to have the following IAM permissions in order to set up the tunnel:

ec2:DescribeInstances – used to identify the jump server.

ec2-instance-connect:SendSSHPublicKey – used to authenticate the SSH session with the jump server.

ssm:StartSession – used to start the SSM session that will act as a proxy for our SSH session.

In this example, we will restrict the scope of these permissions by adding a condition that limits our access to ec2 instances which have a tag with the key application and the value jump_server.

Add the following IAM policy to the AWS role that we want to assume (make sure to adjust the placeholder values):

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "ec2:DescribeInstances",
                "ec2-instance-connect:SendSSHPublicKey",
                "ssm:StartSession"
            ],
            "Resource": [
                "arn:aws:ec2:eu-west-1:123456789123:instance/*"
            ],
            "Condition": {
                "StringLike": {
                    "ec2:resourceTag/application": [
                        "jump_server"
                    ]
                }
            }
        }
    ]
}

Using aws-ssh-tunnel

Now that we have a jump server and added the right IAM permissions to our AWS role, we can start connecting to it using aws-ssh-tunnel. Let’s first configure it by calling aws-ssh-tunnel config.

We will be prompted for our aws region, aws profile, the jump server tag, and the jump server user. For EC2 instances running AWS AMIs, the default user is ec2-user.

Once we have configured our setup, we can connect to our target instance by passing the service endpoint and port to aws-ssh-tunnel run. For example, we can start a port forwarding session with an RDS Postgres database as follows:

aws-ssh-tunnel run --remote_host mydb.123456789012.eu-west-1.rds.amazonaws.com --port 5432

That’s it! We now have a tunneling session with our remote private database that we can locally connect to.

Conclusion

As we have seen, this tool makes it very simple to quickly set up an SSH tunnel with private instances in AWS. In addition, we don’t even need to expose any public bastion hosts to access our private subnet! This reduces the attack surface of our application, and makes our AWS environment much more secure. In my next post, I will go into more detail what aws-ssh-tunnel is doing in the background.

Daniel is a cloud and software engineer who specializes in security. He enjoys automating everything in the cloud, and likes to share security best practices.
Share this article: Tweet this post / Post on LinkedIn