Getting out past the firewall

Ahhh, the joys of a IT departments that think everybody just uses Word/Excel/Outlook and just browses to Facebook at lunchtime.

Networks that transparently proxy HTTP/HTTPS (probably with man in the middle TLS CA certs deployed to all the machines, but that is an entirely different problem) and block everything else really do not work in the modern world where access to places like GitHub via SSH or devices connecting out via MQTT are needed.

One possible solution to the SSH problem is a bastion host. This is a machine that can be reached from the internal network but is also allowed to connect to the outside world. This allows you to use this machine as a jumping off point to reach services blocked by the firewall.

The simple way is to log into the bastion, and then from the shell connect on to your intended external host, this works for targets you want a shell on but not for things like cloning/updating git repositories. We also want to automate as much of this as possible.

The first step is to set up public/private key login for the bastion machine. To do this we first generate a key pair with the ssh-keygen command.

$ ssh-keygen -f ~/.ssh/bastion -t ecdsa -b 521
Generating public/private ecdsa key pair.
Enter passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved in ~/.ssh/bastion.
Your public key has been saved in ~/.ssh/bastion.pub.
The key fingerprint is:
SHA256:3Cfr60QRNbkRHDt6LEUUcemhKFmonqDlgEETgZl+H8A hardillb@tiefighter
The key's randomart image is:
+---[ECDSA 521]---+
|oB+      ..+X*.. |
|= .E    . .o++o  |
|.o  .  . o..+= . |
|....o...o..=o..  |
|  .=.o..S.* +    |
|  . ..o  . *     |
|          o      |
|         o       |
|         .+.     |
+----[SHA256]-----+

In this case we want to leave the passphrase blank because we want to use this key as part of automation of other steps, normally you should use a passphrase to protect access should the keys be compromised.

Once generated you can copy it to the `~/.ssh/authorized_keys` file on the bastion machine using the ssh-copy-id command

$ ssh-copy-id -i ~/.ssh/bastion user@bastion

Once that is in place we should be able to use the key to log straight into the bastion machine. We can now use the `-J` option to specify the bastion as a jump point to reach a remote machine.

$ ssh -J user@bastion user@remote.machine

We can also add this as an entry in the `.ssh/config` file which is more useful for things like git where it’s harder to get at the actual ssh command line.

Host bastion
 HostName bastion
 User user
 IdentityFile ~/.ssh/bastion

Host github
 Hostname github.com
 User git
 IdentityFile ~/.ssh/github
 ProxyCommand ssh -W %h:%p bastion

This config will proxy all git commands working with remote repositories on github.com via the bastion machine, using the bastion key to authenticate with the bastion machine and the github key to authenticate with github. This is all totally transparent to git.

ssh also supports a bunch of other useful ticks, such as port forwarding from either end of the connection.

It also can proxy other protocols using the Socks tunnelling protocol which means it can be used as poor man’s VPN in some situations. To enable Socks proxying you can use -D option to give a local port number or the DynamixProxy directive in the ~/.ssh/config file. This option is really useful with web browser that supports Socks proxies as it means you can point the browser at a local port and have it surf the web as if it was the remote machine.

All of this still works if you are using a bastion machine.

Custom hardware

Combining all this really useful SSH capability with a the Raspberry Pi gadgets makes it possible to carry a bastion host with you. Using a Raspberry Pi Zero W or even a full sized Pi 4 that can be configured to join a more open WiFi network (e.g. a visitor or testing network) you can have a device you just plug into a spare USB port that will give you a jumping off point to the outside world while still being connected to the more restricted internal network with all that provides. Just don’t tell IT security ;-).

This works well because even the most locked down machine normally still allows USB network adapters to be used.

Installing SSH Keybox

I’ve recently installed a Keybox on a Raspberry Pi attached to my home network. Keybox is a bastion service that acts as a hardened access point that a protected network sits behind. The idea being that a single locked externally facing machine is easier to defend than allowing access to the whole network. The usual approach is for that one machine to just run an SSH daemon configured to only allow access via a private key. SSH allows terminal access and file transfer via scp, it allows for tunnels to be set up, so a authorised user can with the right config not normally notice that the bastion machine is there.

Keybox extends this model a little, it provides web hosted terminals to access the machines that sit behind it. The upside to this is that users don’t need anything more than a web browser to access the machine and the private keys never leave the bastion machine. Security is handled by 2FA using a OTP generator (e.g. Google Authenticator). One of the major use cases for Keybox is to access AWS machines without public IP addresses.

The reason for all this being that I’ve had some occasions recently where I’ve needed terminal access to my home machines while away but the networks I’ve been connected to did not allow outbound SSH connections. It should also be useful for when I only have access to machines with web access (e.g. locked down Chromebooks) or borrowing machines.

Installing/Configuring

Download the Keybox tgz file from the releases section of the Keybox github page.

Keybox is uses Jetty to host the web app so needs a Java virtual machine to be installed. With this in mind as I am running this on a Raspberry Pi I also reduced the Java heap size in the launch script from 1024mb to 512mb, this shouldn’t be a problem as this instance is not likely to see large amounts of load. I started the service up and tested connecting direct to the Raspberry Pi on port 8443.

The next step was to expose the service to the outside world. To do this I wanted to mount it on a URL on my main machine to make it use my existing SSL certificate. This machine runs Apache so it needs to be configured to proxy for Keybox instance. I found some useful notes to get started here.

I added the following to inside the <VirtualHost> tags in the ssl.conf file:

SSLProxyEngine On
SSLProxyCheckPeerName off
SSLProxyCheckPeerCN off
SSLProxyCheckPeerExpire off
SSLProxyVerify none

ProxyRequests off
ProxyPreserveHost On
ProxyPass /box https://192.168.1.1:8443/box
ProxyPassReverse /box https://192.168.1.1:8443/box

RequestHeader set X-Forwarded-Proto "https" env=HTTPS

<LocationMatch "/box/admin/(terms.*)">
  ProxyPass wss://192.168.1.1:8443/box/admin/$1
  ProxyPassReverse wss://192.168.1.1:8443/box/admin/$1
</LocationMatch>

I also needed to make sure mod_proxy_wstunnel was enabled to ensure the websocket connections were forwarded. The entries at the start (SSLProxyCheckPeerName, SSLProxyCheckPeerExire and SSLProxyVerify) tell Apache not to validate the SSL certificate on the Keybox machine as it is a self signed certificate.

By default Keybox runs in the root of the Jetty server so it needs a quick update to move it to running in /box to match the proxy settings. Edit the keybox.xml file in jetty/webapps to change the contextPath:

<Configure class="org.eclipse.jetty.webapp.WebAppContext">
  <Set name="contextPath">/box</Set>
  <Set name="war"><Property name="jetty.home" default="." />/keybox</Set>
  <Set name="extractWAR">false</Set>
</Configure>

Now I can access Keybox at https://mymachine/box