SSM Session Manager - no bastion host necessary!

February 3, 2023

door

Motivation

The AWS Systems Manager’s (SSM) Session Manager has been out there for a few years now but we still see projects using some kind of bastion host (jump host) or VPN to access their EC2 and RDS instances.

While this works and can certainly be done securely, it comes with some drawbacks. Some examples are:

managing SSH keys - adding/removing coworkers keys, what about ensuring mandatory key rotation? etc.
running, securing, and maintaining reliable bastion host instances.
often direct internet access through port 22.
often inadequate auditing/logging that doesn’t align with existing SSO setups.

… we could go on but let’s leave it here.

SSM Session Manager leverages your existing IAM user management to allow you to get a shell on your EC2 instances by using your existing IAM credentials with the AWS CLI or AWS Console. This post explores how to implement some common use-cases for SSH based sessions by using the SSM Session Manager. The service also offers other helpful features which we won’t go into detail here, but it is definitely worth checking out the security improvements and auditing features that SSM Session Manager offers.

Prerequisites

In case you don’t use an AWS default AMI - make sure your EC2 instance has the SSM Agent installed.
Make sure to allow the EC2 instance to access the SSM Session Manager APIs e.g. through the InternetGateway or a VPC endpoint.
Enable Default Host Management Configuration to allow SSM to manage all EC2 instances in this account’s region.

Update: Since you now can enable Default Host Management Configuration to allow SSM to manage the EC2 instance there is no more need to alter the EC2 Instance-Profiles :-)

Here you can read more about Setting up Session Manager.

For additional configuration options you can enable Host Management through Systems Manager’s Quick Setup.

Finally, to use Session Manager through your AWS CLI, you need to install the Session Manager plugin.

No keys necessary

As already said SSM Session Manager allows you get a shell on your EC2 instance by using your existing IAM credentials with the AWS CLI:

aws ssm start-session -i i-xxxxxxxxxxxxxxxxx

or AWS Console:

Session Manager console

You got this running and that’s all you need? Then you are one of the lucky ones and from now on you can launch and run EC2 instances without requiring any SSH keys at all.

But I need port forwarding

You asked for it! Now we can also do SSH over the Session Manager connection, by utilizing EC2’s one-time SSH public key feature:

# first send your public key to ec2 (this is for one-time use only <3)
aws ec2-instance-connect send-ssh-public-key --instance-id i-xxxxxxxxxxxxxxxxx --instance-os-user ec2-user --ssh-public-key "$(<~/.ssh/id_rsa.pub)"
# start ssh session plus some additional options to make ssh cloud ready
ssh -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -o"proxycommand aws ssm start-session --target %h --document-name AWS-StartSSHSession --parameters portNumber=%p" ec2-user@i-xxxxxxxxxxxxxxxxx -i ~/.ssh/id_rsa

Amazon EC2 Instance Connect enables system administrators to publish one-time use SSH public keys to EC2, providing users a simple and secure way to connect to their instances.

and include the port forwarding to our example PostgreSQL RDS instance:

# yupp don't forget to send your public key to ec2
aws ec2-instance-connect send-ssh-public-key --instance-id i-xxxxxxxxxxxxxxxxx --instance-os-user ec2-user --ssh-public-key "$(<~/.ssh/id_rsa.pub)"
# start a ssh session including a tunnel to your RDS instances (don't forget the EC2 instances needs access to that RDS)
ssh -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -o"proxycommand aws ssm start-session --target %h --document-name AWS-StartSSHSession --parameters portNumber=%p" -L "5432:database-1.csddzeenrobs.eu-west-1.rds.amazonaws.com:5432" ec2-user@i-xxxxxxxxxxxxxxxxx -i ~/.ssh/id_rsa
# and let's go
psql -h localhost -p5432 -Upostgres

Yes, you can also create a tunnel in the background:

# yupp don't forget to send your public key to ec2
aws ec2-instance-connect send-ssh-public-key --instance-id i-xxxxxxxxxxxxxxxxx --instance-os-user ec2-user --ssh-public-key "$(<~/.ssh/id_rsa.pub)"

# start it without launching a terminal, see additional options
ssh -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -o"proxycommand aws ssm start-session --target %h --document-name AWS-StartSSHSession --parameters portNumber=%p" -M -S ssh-tunnel-ctrl-socket -fnNT -L "5432:database-1.csddzeenrobs.eu-west-1.rds.amazonaws.com:5432" ec2-user@i-xxxxxxxxxxxxxxxxx -i ~/.ssh/id_rsa

# stop it again
ssh -S ssh-tunnel-ctrl-socket -O exit ec2-user@i-xxxxxxxxxxxxxxxxx

Copying files

Using SCP:

# yupp don't forget to send your public key to ec2
aws ec2-instance-connect send-ssh-public-key --instance-id i-xxxxxxxxxxxxxxxxx --instance-os-user ec2-user --ssh-public-key "$(<~/.ssh/id_rsa.pub)"
# similar to before plus some additional options to make ssh cloud ready
scp -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -o"proxycommand aws ssm start-session --target %h --document-name AWS-StartSSHSession --parameters portNumber=%p" -i ~/.ssh/id_rsa static/images/blog/ssm-session-manager-or-no-bastion-host-necessary/door.jpg ec2-user@i-xxxxxxxxxxxxxxxxx:door.jpg

Using Rsync:

# yupp don't forget to send your public key to ec2
aws ec2-instance-connect send-ssh-public-key --instance-id i-xxxxxxxxxxxxxxxxx --instance-os-user ec2-user --ssh-public-key "$(<~/.ssh/id_rsa.pub)"
# similar to before plus some additional options to make ssh cloud ready
rsync -avz -e "ssh -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -o\"proxycommand aws ssm start-session --target %h --document-name AWS-StartSSHSession --parameters portNumber=%p\" -i ~/.ssh/id_rsa" static/images/blog/ssm-session-manager-or-no-bastion-host-necessary/door.jpg ec2-user@i-xxxxxxxxxxxxxxxxx:door.jpg

What else?

Yes, there’s an Ansible plugin for SSM Session Manager
There are even VPN implementations with DNS support on top of SSM Session Manager: https://aws.nz/projects/ssm-tunnel/
There are convenient wrappers for all these bash one-liners: https://github.com/mludvig/aws-ssm-tools
Check out the list of configuration and security options for SSM Session Manager
Check the Auditing session activity guide of SSM Session Manager
Check the Logging session activity guide of SSM Session Manager

And finally stop managing SSH keys and enjoy exploring!

Joscha is a Cloud Consultant at superluminar. After working as a software engineer for various start-ups, he continues to enjoy learning new things and sharing his knowledge and passion for Clojure, Typescript, automation and distributed systems. He left Twitter during the lockdown and now prefers to talk to his neighbours.