Chapter 3: Persistent Sessions

The SSH Trap

You SSH into your server. You start a training run. You close your laptop to go home.

Training dies.

You didn't kill it. You didn't press Ctrl+C. You didn't run out of GPU memory. The SSH connection dropped — and every process you started inside that SSH session received a SIGHUP signal and terminated. Your 14-hour training run made it 47 minutes before your laptop lid closed, your WiFi disconnected, or your VPN hiccuped.

This is the SSH trap, and it catches every researcher at least once. It's the reason people stay late in the lab, sitting in front of their terminal, afraid to touch their laptop while training is running. It's the reason people keep a dedicated SSH window open on a second monitor and refuse to close it. It's the reason Friday night training runs get launched right before leaving and then checked obsessively from the phone until the SSH session on the phone also drops.

The problem is fundamental: SSH ties your processes to your connection. When the connection ends, your processes end. This is by design — it's how Unix sessions work. Your training script is a child process of the SSH shell. Kill the parent, kill the children.

You need a way to decouple your processes from your SSH connection. You need your training to run on the server, independent of whether you're connected, whether your laptop is open, whether your phone has signal, whether you're awake.

That tool is tmux.

---

How Tmux Works

Tmux (terminal multiplexer) solves the SSH trap with a simple architectural trick: it separates the server from the client.

When you start tmux on a machine, it creates a server process that runs in the background on that machine. Inside that server, you create sessions — each session is like a virtual terminal. Your actual terminal (the one connected through SSH) is just a client that displays what's happening inside the session.

Here's what that looks like:

Your SSH Connection              The Server Machine
┌──────────────┐                ┌──────────────────────────┐
│              │   SSH tunnel   │                          │
│  Terminal    │◄──────────────►│  tmux server (always on) │
│  (client)    │                │    │                     │
│              │                │    ├─ session: train     │
│              │                │    │   └─ python train.py│
│              │                │    │                     │
│              │                │    └─ session: eval      │
│              │                │        └─ python eval.py │
└──────────────┘                └──────────────────────────┘

When SSH disconnects:

┌──────────────┐                ┌──────────────────────────┐
│              │                │                          │
│  Terminal    │   ✗ gone ✗     │  tmux server (still on!) │
│  (dead)      │                │    │                     │
│              │                │    ├─ session: train     │
└──────────────┘                │    │   └─ python train.py│
                                │    │       (still runs!) │
                                │    └─ session: eval      │
                                │        └─ python eval.py │
                                │            (still runs!) │
                                └──────────────────────────┘

When your SSH connection drops, the client disappears — but the tmux server keeps running. Every process inside every tmux session continues as if nothing happened. Your training script doesn't know and doesn't care that you disconnected. It's talking to the tmux server, not to your SSH client.

When you SSH back in, you just attach a new client to the same tmux server. You pick up exactly where you left off. The terminal output is all still there. The processes are still running. Nothing was lost.

This is why tmux changes everything. Your processes don't live inside your SSH connection anymore. They live inside tmux. SSH is just the window you use to look at them.

---

Essential Commands

You need exactly six commands. No more.

Create a named session

tmux new -s train

This creates a new tmux session called train and attaches you to it. You'll notice your terminal now has a green bar at the bottom — that's the tmux status bar, confirming you're inside a session.

Always name your sessions. tmux new without -s gives you a session called 0, then 1, then 2. When you have three training runs, you want to know which is which. Name them: train, eval, download, experiment01 — whatever makes sense.

Detach from the session

Ctrl+b  then  d

This is a two-step key sequence. Press Ctrl+b (hold Ctrl, press b, release both). Then press d. The session disappears from your screen — but it's still running on the server. You're back at your normal SSH shell.

Detaching is not closing. The tmux session and everything in it continues to run. This is the entire point.

List sessions

tmux ls

Shows all running tmux sessions on this machine:

train: 1 windows (created Mon Mar 23 14:30:12 2026)
eval: 1 windows (created Mon Mar 23 15:45:03 2026)

Two sessions, both running. You can see when each was created.

Reattach to a session

tmux attach -t train

You're back inside the train session. Everything is as you left it. The terminal output that happened while you were detached is all there — just scroll up.

Kill a session

tmux kill-session -t train

This actually stops the session and kills all processes inside it. Use this when you're done with a session — a finished experiment, a completed download. Don't leave zombie sessions cluttering your server.

That's it

Six commands:

Command	What it does
tmux new -s NAME	Create a named session
Ctrl+b d	Detach (session keeps running)
tmux ls	List all sessions
tmux attach -t NAME	Reattach to a session
tmux kill-session -t NAME	Kill a session and its processes

Everything else tmux can do — splits, panes, windows, scripting — is useful but not essential. You can learn it later. These six commands are enough for everything in this guide.

---

The Test

Let's prove it works. This is not optional — you need to see it with your own eyes.

Step 1: SSH into your server

ssh lab-server

Step 2: Create a tmux session

tmux new -s test

You're now inside a tmux session called test. You should see the green status bar at the bottom.

Step 3: Start a long-running process

python3 -c "import time; [print(i) or time.sleep(1) for i in range(999)]"

You'll see numbers counting up, one per second: 0, 1, 2, 3...

Watch it count to at least 5. This is your training run. If you were to close the SSH connection right now without tmux, this process would die instantly.

Step 4: Detach

Press Ctrl+b, then d.

The counting disappears. You're back at the regular SSH prompt. The session is still running — you just can't see it.

Step 5: Disconnect

Close the SSH connection:

exit

Now close your terminal entirely. If you're on your phone, close Termius. If you're on your laptop, close the terminal app. The SSH connection is gone. Dead.

Step 6: Wait

Wait 30 seconds. Go get water. Check your phone. Do something else. The point is: you are no longer connected to the server in any way. Your training (the counter) is alone.

Step 7: Reconnect

Open your terminal again. SSH back in:

ssh lab-server

Step 8: Reattach

tmux attach -t test

Step 9: See the magic

The counter is still going. It didn't stop. If you detached at count 5 and waited 30 seconds, it's now at 35. It kept counting the entire time you were disconnected.

This is the core promise of tmux: your processes survive disconnection. The counter didn't know you left. It didn't know you came back. It just kept running inside the tmux server, indifferent to the state of your SSH connection.

Kill the counter with Ctrl+C, then kill the test session:

tmux kill-session -t test

---

Phone-Friendly Tmux

You're going to use tmux from your phone through Termius. Small screen, no physical keyboard, touch input. A few adjustments make this bearable.

Enable mouse mode

By default, tmux intercepts mouse events in a way that can confuse mobile terminals. Adding mouse mode makes scrolling and selecting work naturally. On your server, create or edit ~/.tmux.conf:

echo 'set -g mouse on' >> ~/.tmux.conf

If you already have a tmux session running, reload the config:

tmux source-file ~/.tmux.conf

With mouse mode on, you can scroll through terminal output by swiping on your phone — much easier than the keyboard-based alternative.

Scrolling without a mouse

If mouse scrolling doesn't work in your terminal (some mobile clients have quirks), use tmux's copy mode:

1. Press Ctrl+b, then [ — this enters copy mode.
2. Use arrow keys (or swipe, depending on your terminal) to scroll up through the output.
3. Press q to exit copy mode and return to normal.

Copy mode is how you'll read error messages and training logs on your phone. Practice it now so you're not fumbling with it at 2am when something crashes.

Keep session names short

When you list sessions on a phone screen, long names get cut off. Keep them short and meaningful:

Good	Bad
train	my-training-experiment-v2
exp01	bert-finetune-lr3e5-bs32
dl	downloading-imagenet-dataset

You'll see the full context when you attach to the session. The name just needs to be enough to pick the right one from tmux ls.

---

Tmux + Training: The Daily Pattern

This is the pattern you'll use every single day. It's simple, and it works.

1. SSH to your server

ssh lab-server

2. Create a named session for your experiment

tmux new -s experiment01

3. Start your training

cd /path/to/your/project
conda activate your-env
python train.py --config config.yaml

Training is running. You can see the loss going down, the epochs ticking by. Everything looks good.

4. Detach

Press Ctrl+b, then d.

The training keeps running. You're back at the SSH prompt.

5. Go live your life

Close the SSH connection. Close your laptop. Go home. Go to dinner. Go to sleep. Go on a weekend trip. It doesn't matter. Your training is running inside tmux on the server. It doesn't need you.

6. Check in from your phone

At any point — from the bus, from bed, from a restaurant — open Termius, SSH in, and check:

tmux attach -t experiment01

You see the latest training output. Loss is still going down. Epoch 47 of 100. Everything is fine. Detach again:

Press Ctrl+b, then d.

Total phone time: 15 seconds.

7. Come back when it's done

The next morning, or whenever you're back at your desk:

ssh lab-server
tmux attach -t experiment01

Training is done. Results are saved. You review them, clean up, and kill the session:

tmux kill-session -t experiment01

That's the whole workflow. Create session, start training, detach, live your life, check occasionally, come back when done. No babysitting. No anxiety. No staying late in the lab.

---

What This Unlocks

Before tmux, the question was: "Can I leave the lab while training is running?"

After tmux, that question disappears. You launch training and you leave. You check from your phone when it's convenient. If it's still running, great. If it finished, great. If it crashed — well, right now you'd have to SSH in from your phone and fix it with your thumbs, which is painful. But that's what the next chapters are for. Claude Code will handle the crashes. Tmux just guarantees that your processes survive your absence.

Every chapter from here on assumes tmux is part of your workflow. Claude Code will run inside tmux. Training will run inside tmux. Downloads, evaluations, monitoring — all tmux. It's the invisible foundation that makes everything else possible.

---

Checkpoint

Start a long-running process inside tmux:

ssh lab-server
tmux new -s test
python3 -c "import time; [print(i) or time.sleep(1) for i in range(999)]"

Detach with Ctrl+b d. Close Termius entirely. Wait a full minute. Reopen Termius, SSH back in, reattach with tmux attach -t test.

If the counter kept counting while you were gone — you've unlocked persistent sessions. Your training will never die from a disconnection again.