I cant ssh to computers connected via Tailscale, but all other services work ok

Question

I have 5 linux devices (Ubuntu 20's and Raspberry Pi's) in 3 different geographic locations. With Tailscale on each.

I am unable to ssh between these machines, via the Tailscale IP's.

• my "regular ssh" over the internet works fine, i.e key-not-password, DDNS, open non-standard high port on the router, that port-forwards incoming connections to port 22 on the ssh server.
• I can ping the machines, via tailscale
• A port scan shows that all correct ports are open (incl 22) via tailscale network
• All other services work fine over tailscale - just not ssh, e.g I can RDP/VNC in, and I ran a netcat server also works just fine.
• I can directly connect to the SSH server using netcat over tailscale, ssh server is listening and responding over tailscale network
• Ubuntu firewalls are off. /etc/hosts.allow is empty
• So the problem is definetly some combunation of SSH and tailscale.
• Possible I need the right magic in /etc/ssh/sshd_config?
• I tried adding ListenAddress 0.0.0.0 to sshd_config, no change.

When I try and establish an ssh connection, it just hangs for a while, then timesout. Example log below:

me@laptop:~$ ssh -vv [email protected]
OpenSSH_8.2p1 Ubuntu-4ubuntu0.4, OpenSSL 1.1.1f  31 Mar 2020
debug1: Reading configuration data /home/me/.ssh/config
debug1: Reading configuration data /etc/me/ssh_config
debug1: /etc/ssh/ssh_config line 19: include /etc/ssh/ssh_config.d/*.conf matched no files
debug1: /etc/ssh/ssh_config line 21: Applying options for *
debug2: resolve_canonicalize: hostname 100.100.1.1 is address
debug2: ssh_connect_direct
debug1: Connecting to 100.100.1.1 [100.100.1.1] port 22.
debug1: connect to address 100.100.1.1 port 22: Connection timed out
ssh: connect to host 100.100.1.1 port 22: Connection timed out

Any suggestions what to do? I'm out of ideas.

Answer 1

Ok, looks like a bug in ssh w/tailscale - using openssh v8.2

I was using:

OpenSSH_8.2p1 Ubuntu-4ubuntu0.4, OpenSSL 1.1.1f 31 Mar 2020

Thanks to a suggestion elsewhere, I compilied an earlier version of openssh:

OpenSSH_7.9p1, OpenSSL 1.1.1f 31 Mar 2020

... and now it works.