Slow SSH access
SSH into any user to certain worker machines takes longer than they should. If I enable verbose logs, I see:
debug3: sign_and_send_pubkey: RSA SHA256:Ukl1G/fqtOdjHn7yDm6D24dtm6UvfSCG4MsB3Qi6BfU
debug1: Authentication succeeded (publickey).
Authenticated to worker9 ([x.x.62.218]:22).
debug1: channel 0: new [client-session]
debug3: ssh_session2_open: channel_new: 0
debug2: channel 0: send open
debug1: Requesting no-more-sessions@openssh.com
debug1: Entering interactive session.It pauses there before finally showing a terminal.
We've already disabled reverse DNS. This bug may be relevant: https://bugs.launchpad.net/ubuntu/+source/openssh/+bug/300151
I used strace -t -f -p <sshd PID>, and I do see a pattern of what looks like some retries:
# grep ChallengeResp 1252.txt
[pid 1252] 02:03:18 write(10, "\0\0\1\\\n\n\nChallengeResponseAuthenti"..., 356 <unfinished ...>
[pid 1252] 02:03:19 write(9, "\0\0\1\\\n\n\nChallengeResponseAuthenti"..., 356 <unfinished ...>
[pid 1252] 02:03:19 write(10, "\0\0\1\\\n\n\nChallengeResponseAuthenti"..., 356 <unfinished ...>
[pid 1252] 02:03:19 write(11, "\0\0\1\\\n\n\nChallengeResponseAuthenti"..., 356 <unfinished ...>
[pid 1252] 02:03:20 write(11, "\0\0\1\\\n\n\nChallengeResponseAuthenti"..., 356 <unfinished ...>
[pid 1252] 02:03:21 write(9, "\0\0\1\\\n\n\nChallengeResponseAuthenti"..., 356 <unfinished ...>
[pid 1252] 02:03:22 write(9, "\0\0\1\\\n\n\nChallengeResponseAuthenti"..., 356) = 356
[pid 1252] 02:03:23 write(10, "\0\0\1\\\n\n\nChallengeResponseAuthenti"..., 356 <unfinished ...>
[pid 1252] 02:03:23 write(11, "\0\0\1\\\n\n\nChallengeResponseAuthenti"..., 356 <unfinished ...>
[pid 1252] 02:03:24 write(11, "\0\0\1\\\n\n\nChallengeResponseAuthenti"..., 356 <unfinished ...>
[pid 1252] 02:03:25 write(12, "\0\0\1\\\n\n\nChallengeResponseAuthenti"..., 356 <unfinished ...>
[pid 1252] 02:03:25 write(13, "\0\0\1\\\n\n\nChallengeResponseAuthenti"..., 356 <unfinished ...>
[pid 1252] 02:03:26 write(11, "\0\0\1\\\n\n\nChallengeResponseAuthenti"..., 356) = 356
[pid 1252] 02:03:26 write(11, "\0\0\1\\\n\n\nChallengeResponseAuthenti"..., 356 <unfinished ...>
[pid 1252] 02:03:27 write(11, "\0\0\1\\\n\n\nChallengeResponseAuthenti"..., 356 <unfinished ...>