Tunnel

Exposing Homelab services using SSH

/ DevOps, Kubernetes, linux / By

I recently deployed a web application for Andile Mbele to my Kubernetes homelab cluster. Andile wanted to use his own domain to reach the application. I use Cloudflare Tunnels to expose my applications to the Internet so I thought I could use the same to expose his app. At first I thought that setting up his application’s domain and mapping it to an endpoint in my Kubernetes homelab cluster would be as simple as creating a CNAME DNS record pointing to my secure tunnel. I was wrong, Andile’s domain is managed by Cloudflare and Cloudflare does’t allow cross account redirects to tunnels on their free tier. So I had to come up with a different secure way to expose the application.

There are a number of ways to expose Kubernetes services running in a private cluster securely to the Internet. These include forwarding ports from your router to the Kubernetes Ingress, using a VPN, using tunneling services like Cloudflare Tunnels or tunneling using SSH. I used SSH tunneling because its free, all my servers have have SSH installed and, unlike using a third party service, I’m in full control of the tunnel.

In this post, I’ll show you how to expose services running in your server through an SSH tunnel without modifying firewall rules or setting up public ingress. For this set up, you’ll need a publicly accessible server or VPS on the Internet and your local Kubernetes cluster. On VPS, install and set up a reverse proxy so you can use the VPS as a gateway to the private cluster. I’ll refer to the Kubernetes server as local server and the VPS as the remote server in this article.

Create SSH key pairs in the local machine

The first step is to create SSH keys on the local server:

sudo mkdir -p /etc/sshtunnel
sudo ssh-keygen -t ed25519 -f /etc/sshtunnel/tunnel_key

Next, copy the contents of the public key to the remote server’s ~/.ssh/authorized_keys file to allow SSH access from the local server.

Create Reverse SSH Tunnel

Create a reverse SSH tunnel to the remote server by running this command. Substitute the IP addresses and user details with your own:

ssh -qnN -R 9091:172.16.0.8:80 user@remote-server -i /etc/sshtunnel/tunnel_key

This command creates a reverse SSH tunnel from the local server to the remote server. Any traffic sent to port 9091 on the remote server is automatically forwarded to the local server on port 80. Here’s a detailed explanation of the command:

Command explained:

  • N: No command execution, just forwarding
  • q – Quiet mode. Suppresses warnings and debug messages.
  • n – Redirects stdin to /dev/null. Prevents SSH from reading from standard input, useful when running it in the background.
  • -R 9091:172.16.0.8:80: Maps port 9091 on the VPS to 172.16.0.8:80 in the local environment
  • user@remote-server — Server user and address or hostname

To help me remember what this command does, I always read it this way: “Create a reverse SSH tunnel that forwards traffic from port 9091 to 172.16.0.8:80 using the following credentials”. Run the command and test if forwarding works by sending requests through the tunnel. If it works, great, you’re off to a good start. The problem with this however is if the server is switched off or there’s a network problem, the SSH connection will drop. So you need to keep the connection persistent. I’ll show you how to make the connection persistent shortly, let’s first configure a reverse proxy on the remote host.

Create Apache VirtualHost

To avoid exposing the SSH tunnel port directly, you can put it behind a reverse proxy like Nginx or Apache. In this example, I’ll use Apache to proxy requests between the application’s domain and the tunnel the local machine is connected to. Using a reverse proxy has the benefit of allowing us to use a human-friendly domain as opposed to an IP address:port combination. Create a new Apache virtual server and populate it with your app details:

<VirtualHost remote-server-ip:80>
    ProxyPreserveHost On
    ServerName your_app_domain_name
    ProxyPass "/" "http://localhost:9091/"
    ProxyPassReverse "/" "http://localhost:9091/"
</VirtualHost>

This instructs Apache to forward or proxy requests it receives on the VPS IP to the SSH tunnel.

Enable proxy and proxy_http modules if not already enabled:

a2enmod proxy proxy_http
systemctl restart apache2

Adjust DNS Records

Next, navigate to your provider’s DNS records and create a new A record pointing your app’s domain to the IP address of the VPS. This step allows you to connect to the app using its domain name and not via IP address.

Make the Tunnel persistent

With DNS out of the way, we can now make the SSH tunnel persistent. Most tutorials or guides you’ll see online direct you to use autossh to keep the SSH client process alive, but this is redundant in modern versions of OpenSSH; that functionality is now built in.

We’ll modify the SSH command we ran above with the following options to keep the connection alive:

  • ServerAliveInterval=30 — If no data sent through the tunnel, send a message to request a response from the server. Sends a hello, r u there? message.
  • ServerAliveCountMax=3 — sets the max number of server alive messages that should be sent before a connection is considered dead.
  • ExitOnForwardFailure — Use this setting to ensure that the tunnel is working. Ensures that SSH disconnects immediately if it fails to create a tunnel. Default action is to remain connected even when it can’t set up the tunnel.
    • Prevent the connection from hanging if forwarding fails
  • StrictHostKeyChecking=yes — Only connect to hosts in the known_hosts file, if host fingerprint changes, do not connect

In addition to adding the options above, we’ll run the command as a systemd service to allow starting the tunnel when the server boots up.

Create a Systemd Service

To ensure the SSH tunnel survives server reboots, create a systemd service for it. Replace tunnel-user, user and remote-server with the values in your system. I’m naming this file ssh-tunnel.service.

# ssh-tunnel.service

[Unit]
Description= Service to maintain an SSH reverse tunnel
Wants=network-online.target
After=network-online.target
StartLimitIntervalSec=0

[Service]
User=tunnel-user
Type=simple
ExecStart=/usr/bin/ssh -qNn \
    -o ServerAliveInterval=30 \
    -o ServerAliveCountMax=3 \
    -o ExitOnForwardFailure=yes \
    -o StrictHostKeyChecking=yes \
    -i /etc/sshtunnel/id_ed25519 \
    -R 9091:172.16.0.8:80 \
    user@remote-server 2>&1 | logger -t ssh-tunnel
Restart=always
RestartSec=10

[Install]
WantedBy=multi-user.target

This unit file runs and maintains an SSH reverse tunnel and ensures it runs reliably when the system boots. Here’s an explanation of some of the sections:

  • The service should be started after networking is enabled.
  • After= — starts after networking is fully active
  • StartLimitInterval — systemd limits how frequently a service can restart. This option disables rate limiting for this service.
  • Service to start at boot when multi user login support state has been reached

Enable the systemd unit

Validate that the service is defined correctly

sudo systemd-analyze verify ssh-tunnel.service

If there are no errors, copy the unit file to the systemd directory

sudo mv ssh-tunnel.service /etc/systemd/system

Instruct systemd to reload its configuration so its aware of the new service then enable the service:

sudo systemctl daemon-reload
sudo systemctl enable ssh-tunnel.service
sudo systemctl start ssh-tunnel.service

Running the above commands make systemd reload its configuration, schedule the service to start on boot and runs the SSH tunnel service.

Running the SSH tunnel this way using systemd is nice because it ensures that the tunnel gets started when the server turns on and the tunnel process can be managed and monitored easily. For example, to view the status of the tunnel run

sudo systemctl status ssh-tunnel

To view logs from the service run:

journalctl -u ssh-tunnel

Conclusion

SSH tunneling is a good option if you want full control over data flowing in and out of your environment, you’re not locked in to any vendor’s products and there aren’t any external dependencies to manage which is good from a security perspective. In this article, you saw how to securely set up a reverse SSH tunnel to a public VPS as an entry-point to a Kubernetes service (or any service really) running in a private cluster without opening ports in a firewall or port-forwarding in a router.

What do you think of this set up? Let me know in the comments.

Useful References

https://www.baeldung.com/linux/ssh-tunneling-and-proxying
https://dev.to/bulletmark/create-a-reverse-ssh-tunnel-for-remote-access-to-a-restricted-machine-1ma0