Reverse SSH Tunnel Explained

SSH tunnels are a great tool to solve various problems with routing where a firewall does not allow to access a certain local network. They allow forwarding connections to a specified port on the server machine to a local or remote computer through an encrypted channel.

Although there are nice diagrams explaining how SSH tunnels work, they lack detailed description of parameters (e.g. specification of the network interfaces). In this post, I will try to explain the use of a reverse SSH tunnel with full functionality and its parameters. Only TCP ports are considered in this post, but this should work with UNIX sockets as well.

Full functionality: a use case with 4 computers

Reading the SSH manual for the first time may not be that easy to understand:

Nevertheless, plain English can be translated to a 2-dimensional language of drawings. Here is the diagram depicting the process of establishing a reverse SSH tunnel between "server" computer and "host" routed through "client", where command "ssh" is executed:

Diagram 1: A reverse SSH tunnel with full functionality. Download PDF, SVG.

A use case can be demonstrated with 4 computers, 3 of which reside in network 10.0.0.* and 2 computers reside in network 192.168.0.*:

Diagram 2: A use case with full functionality. Download PDF, SVG.

An HTTP request sent by command "curl" on the orange computer on the right will return an HTML page with a list of files in the directory where "python3 -m http.server 80" is being executed on the blue computer on the left.

This works only when "GatewayPorts" parameter of the SSHD daemon is set to "clientspecified", which allows "client" to choose a network interface to bind to:

Binding to all network interfaces on the server

When "GatewayPorts" parameter of the SSHD daemon is set to "clientspecified", leaving empty "bind_addr" (there is a colon ":" before "port" argument!) will bind to all network interfaces (Diagram 1).

However, this freedom of choice of network interfaces is not always the case and the binding behavior of the tunnel depends on the "GatewayPorts" setting.

When "GatewayPorts" parameter of the SSHD daemon is set to "yes", parameter "bind_addr" has no effect at all, the server will bind to all network interfaces regardless of the value of "bind_addr", so it is logical to create a tunnel by omitting "bind_addr" parameter:

Diagram 3: Binding to all interfaces when "GatewayPorts" is set to "yes". Download PDF, SVG.

Similarly, when "GatewayPorts" parameter of the SSHD daemon is set to "no", argument "bind_addr" has no effect at all, the server will bind to loopback interface with IPv4 address 127.0.0.1 or IPv6 address [::1] regardless of the value of "bind_addr". So it is better to omit "bind_addr" parameter completely. If you still need to bind to all network interfaces, then a trick with socket forwarding might help:

Diagram 4: Binding to all interfaces via socket forwarding. Download PDF, SVG.

By default "socat" binds "glbport" to all network interfaces, but this option can be specified explicitly as well:

Special case: endpoint of a tunnel is client itself

Usually, one end of a tunnel is terminated at the client itself. This special case can be described using the following diagram ("GatewayPorts" is set to "yes"):

Diagram 5: Endpoint is client. Binding to all interfaces on the server side. Download PDF, SVG.

Depending on the value of "GatewayPorts" setting, the binding behavior may change:

Diagram 6: Endpoint is client. Binding to loopback interface on the server side. Download PDF, SVG.

When unset, the default value of "GatewayPorts" parameter is "no", so Diagram 6 shows the most frequent scenario which may occur in daily life. The server will allow incoming connections to its "port" only from the localhost (itself).

Tips and Tricks

Use "tmux" to enable multiple terminals in a single screen.

Choose ports in the range 1025-65535, ports below 1025 require root permissions to be used.

Use identity file (private key) to log into the server without entering password: "-i ~/.ssh/id_rsa".

Option "-f" requests "ssh" to go to background just before command execution. Useful for automating tunnel creation in scripts.

Use flag "-N" for disabling execution of a remote command. This is useful for just forwarding ports, e.g.

but if there is a command to execute, then this flag should be omitted:

Debugging on the client side can be done by increasing verbosity ("-v" flag):

Debugging on the server side can be done by specifying the debug mode "-d":

If ssh client hangs without throwing any errors,  try to unset SSH_AUTH_SOCK variable and retry the attempt:

Listing the status of a specified port (e.g. port 80) can be done by grepping for it in the "netstat" output:

References:

"ssh - OpenSSH remote login client", manual page

"sshd_config - OpenSSH daemon configuration file", manual page

"SSH Tunneling Explained" by  Buddhika Chamith

"SSH port forwarding visualized" by Dirk Loss

"How does reverse SSH tunneling work?", StackOverflow