How It Works

This page explains the architecture and internals of SciTeX SSH.

Warning

Before setting up reverse tunnels, please check your organization’s acceptable use policy and network terms of service. Reverse tunnels may bypass institutional firewalls or network policies. The authors accept no responsibility for any consequences arising from the use of this software.

The Problem: NAT (Network Address Translation) Traversal

Machines behind NAT or institutional firewalls cannot receive incoming connections. A standard SSH connection from a remote client to a lab workstation fails because:

  1. The workstation has a private IP (e.g., 192.168.x.x) that is not routable from the internet.

  2. The router/firewall blocks unsolicited incoming connections.

  3. Port forwarding on institutional networks is typically not available to researchers.

The Solution: Reverse SSH Tunnels

A reverse tunnel inverts the connection direction. Instead of the client connecting to the workstation, the workstation connects outward to a publicly reachable bastion server and opens a listening port there. When a client connects to that port on the bastion, the traffic is forwarded back through the existing tunnel to the workstation.

┌─────────────────────────────────────────┐
│  Lab Workstation (behind NAT/firewall)  │
│                                         │
│  ┌──────────────────────────────────┐   │
│  │ systemd service                  │   │
│  │ autossh-tunnel-{port}.service    │   │
│  │   ┌──────────────────────────┐   │   │
│  │   │ autossh                  │   │   │
│  │   │ (auto-reconnect daemon)  │   │   │
│  │   └────────────┬─────────────┘   │   │
│  └────────────────┼─────────────────┘   │
│                   │                     │
│  localhost:22 ◄───┘                     │
│  (SSH server)                           │
└───────────────────┼─────────────────────┘
                    │
                    │  reverse tunnel
                    │  -R {port}:localhost:22
                    ▼
┌──────────────────────────────────────────┐
│  Bastion Server (public IP)              │
│                                          │
│  sshd listening on port {port}           │
│  forwards to tunnel → workstation:22     │
└──────────────────────┬───────────────────┘
                       │
                       │  ssh -p {port} bastion-server
                       │
┌──────────────────────┴───────────────────┐
│  Remote Client (researcher laptop)       │
│                                          │
│  ssh -p {port} user@bastion-server       │
│  → routed through tunnel to workstation  │
└──────────────────────────────────────────┘

Persistence via systemd and autossh

A plain ssh -R tunnel breaks when the network drops or the machine reboots. SciTeX SSH solves this with two layers of persistence:

Layer 1: autossh (connection-level persistence)

autossh is a program that monitors an SSH connection and automatically re-establishes it when it detects failure. It uses -M 0 (rely on SSH’s built-in ServerAliveInterval/ServerAliveCountMax) and runs as a daemon.

Layer 2: systemd (process-level persistence)

Each tunnel is registered as a systemd service unit. This provides:

  • Boot persistence: WantedBy=multi-user.target ensures the tunnel starts on boot.

  • Crash recovery: Restart=always with RestartSec=3 restarts autossh if it crashes.

  • Standard management: systemctl start/stop/status for the tunnel service.

The Service File

When you run scitex-ssh setup -p 2222 -b user@bastion -s ~/.ssh/id_rsa, the following systemd unit file is created:

File: /etc/systemd/system/autossh-tunnel-2222.service

[Unit]
Description=AutoSSH tunnel service
After=network-online.target
Wants=network-online.target

[Service]
User=<your-username>
Environment="AUTOSSH_GATETIME=0"
ExecStart=/usr/bin/autossh -M 0 -N \
    -o "PubkeyAuthentication=yes" \
    -o "PasswordAuthentication=no" \
    -i /home/<user>/.ssh/id_rsa \
    -R 2222:localhost:22 user@bastion
RestartSec=3
Restart=always

[Install]
WantedBy=multi-user.target

After writing this file, the setup script runs:

systemctl daemon-reload
systemctl enable autossh-tunnel-2222.service
systemctl restart autossh-tunnel-2222.service

The Three Operations

Operation

What it does

Under the hood

setup

Creates and starts a persistent reverse tunnel

Writes systemd unit file → daemon-reload → enable → start

status

Reports whether the tunnel is active

Runs systemctl status autossh-tunnel-{port}.service

remove

Stops and removes the tunnel permanently

stop → disable → delete unit file → daemon-reload

Each operation is available through all three interfaces:

  • Python: scitex_ssh.setup(), scitex_ssh.status(), scitex_ssh.remove()

  • CLI: scitex-ssh setup, scitex-ssh status, scitex-ssh remove

  • MCP: tunnel_setup, tunnel_status, tunnel_remove tools

Environment Variables

To avoid repeating the bastion server and key path on every command, set these environment variables:

Variable

Description

Example

SCITEX_SSH_BASTION_SERVER

Default bastion server address

user@bastion.example.com

SCITEX_SSH_SECRET_KEY_PATH

Default SSH private key path

~/.ssh/id_rsa

SCITEX_SSH_DEBUG_MODE

Enable verbose logging (1)

0

CLI flags always take precedence over environment variables.

Prerequisites

  • autossh: Must be installed on the host (sudo apt install autossh).

  • SSH key pair: Public key must be authorized on the bastion server (~/.ssh/authorized_keys).

  • Bastion server: Any machine with a public IP and an SSH server (sshd). The bastion must have GatewayPorts yes or GatewayPorts clientspecified in /etc/ssh/sshd_config if you want the forwarded port to be accessible from machines other than the bastion itself.

  • sudo access: Required on the lab workstation to write systemd unit files in /etc/systemd/system/ and run systemctl.

Alternative: No-sudo Setup via ~/.bashrc

If you do not have sudo access (e.g., shared HPC nodes, university servers), you can run autossh directly from your shell profile without systemd. Add to ~/.bashrc:

# Persistent reverse tunnel without sudo — starts on every login
# Checks if tunnel is already running before starting
if ! pgrep -f "autossh.*-R 2222:localhost:22" > /dev/null 2>&1; then
    autossh -M 0 -f -N \
        -o "PubkeyAuthentication=yes" \
        -o "PasswordAuthentication=no" \
        -o "ServerAliveInterval=30" \
        -o "ServerAliveCountMax=3" \
        -i ~/.ssh/id_rsa \
        -R 2222:localhost:22 user@bastion.example.com
fi

Trade-offs vs. systemd approach:

Aspect

systemd (with sudo)

~/.bashrc (no sudo)

Root access

Required

Not required

Starts on

Boot (no login needed)

User login only

Crash recovery

systemd restarts automatically

autossh reconnects SSH; no restart if autossh itself crashes

Management

systemctl start/stop/status

pgrep / pkill

Alternative: Persistent Session (screen, tmux, nohup)

For long-running sessions where you want the tunnel to survive logout without sudo or systemd:

# Option 1: screen (detaches from terminal)
screen -dmS tunnel autossh -M 0 -N \
    -o "ServerAliveInterval=30" -o "ServerAliveCountMax=3" \
    -i ~/.ssh/id_rsa -R 2222:localhost:22 user@bastion.example.com

# Reattach:  screen -r tunnel
# Kill:      screen -S tunnel -X quit

# Option 2: tmux
tmux new-session -d -s tunnel "autossh -M 0 -N \
    -o ServerAliveInterval=30 -o ServerAliveCountMax=3 \
    -i ~/.ssh/id_rsa -R 2222:localhost:22 user@bastion.example.com"

# Reattach:  tmux attach -t tunnel
# Kill:      tmux kill-session -t tunnel

# Option 3: nohup (simplest)
nohup autossh -M 0 -N \
    -o "ServerAliveInterval=30" -o "ServerAliveCountMax=3" \
    -i ~/.ssh/id_rsa -R 2222:localhost:22 user@bastion.example.com \
    > /dev/null 2>&1 &

# Kill:      pkill -f "autossh.*-R 2222:localhost:22"

Tip

To survive reboots without sudo, add the command to a cron @reboot job:

crontab -e
# Add this line:
@reboot autossh -M 0 -f -N -o "ServerAliveInterval=30" -o "ServerAliveCountMax=3" -i ~/.ssh/id_rsa -R 2222:localhost:22 user@bastion.example.com
Comparison of all approaches

Approach

Sudo

Survives logout

Survives reboot

Management

systemd (default)

Yes

Yes

Yes

systemctl

~/.bashrc

No

No (restarts on login)

No

pgrep / pkill

screen / tmux / nohup

No

Yes

No (unless cron @reboot)

screen -r / tmux attach

cron @reboot + nohup

No

Yes

Yes

crontab -e / pkill

Alternative: Direct Shell Scripts

If you have sudo access but prefer not to install Python, use the shell scripts directly:

# Download the scripts (one-time)
curl -o ~/.local/bin/setup-autossh-service.sh \
  https://raw.githubusercontent.com/ywatanabe1989/scitex-ssh/main/src/scitex_ssh/scripts/setup-autossh-service.sh
curl -o ~/.local/bin/remove-autossh-service.sh \
  https://raw.githubusercontent.com/ywatanabe1989/scitex-ssh/main/src/scitex_ssh/scripts/remove-autossh-service.sh
chmod +x ~/.local/bin/setup-autossh-service.sh ~/.local/bin/remove-autossh-service.sh

# Usage (requires sudo for systemd operations)
setup-autossh-service.sh -p 2222 -b user@bastion.example.com -s ~/.ssh/id_rsa
remove-autossh-service.sh -p 2222