After the first user connects and disconnects, try to connect again, get following error (Basically look like ocserv crashs on every second try)

ocserv works fine on Ubuntu 18.04.5 LTS, Installed by using sudo apt install ocserv, not compiled from source
starts on system boot
Upgraded to Ubuntu 20.04.1 LTS, the crash in the title starts to appear
Tried fresh Ubuntu 20.04.1 LTS, the crash in the title starts to appear
The ocserv configuration is bare minimum, not complex at all, just get it up and running. Most of the values are default.
Even tried using sudo systemctl start ocserv, crashs the same way

Oct 05 05:49:39 ubuntu ocserv[184906]: sec-mod: using 'pam' authentication to authenticate user (session: 9tgcgO)
Oct 05 05:49:39 ubuntu ocserv[184906]: PAM-auth conv: echo-off, msg: 'Password: '
Oct 05 05:49:42 ubuntu ocserv[184899]: main[test]:10.0.0.11:60782 new user session
Oct 05 05:49:42 ubuntu ocserv[196809]: worker[test]: 10.0.0.11 suggesting DPD of 90 secs
Oct 05 05:49:42 ubuntu ocserv[196809]: worker[test]: 10.0.0.11 configured link MTU is 1500
Oct 05 05:49:42 ubuntu ocserv[196809]: worker[test]: 10.0.0.11 peer's link MTU is 1406
Oct 05 05:49:42 ubuntu ocserv[196809]: worker[test]: 10.0.0.11 sending IPv4 192.168.100.62
Oct 05 05:49:42 ubuntu ocserv[196809]: worker[test]: 10.0.0.11 IPv6 routes/DNS disabled because the agent is not known.
Oct 05 05:49:42 ubuntu ocserv[196809]: worker[test]: 10.0.0.11 adding DNS 8.8.8.8
Oct 05 05:49:42 ubuntu ocserv[196809]: worker[test]: 10.0.0.11 DTLS ciphersuite: PSK-NEGOTIATE
Oct 05 05:49:42 ubuntu ocserv[196809]: worker[test]: 10.0.0.11 Link MTU is 1406 bytes
Oct 05 05:49:42 ubuntu ocserv[196809]: worker[test]: 10.0.0.11 selected DTLS compression method lzs
Oct 05 05:49:42 ubuntu ocserv[196809]: worker[test]: 10.0.0.11 selected CSTP compression method lzs
Oct 05 05:49:42 ubuntu ocserv[184899]: main[test]:10.0.0.11:60782 user logged in
Oct 05 05:49:42 ubuntu ocserv[196809]: worker[test]: 10.0.0.11 setting up DTLS-PSK connection
Oct 05 05:49:42 ubuntu ocserv[184906]: sec-mod: initiating session for user 'test' (session: 9tgcgO)
Oct 05 05:49:46 ubuntu ocserv[196809]: worker[test]: 10.0.0.11 received BYE packet; exiting
Oct 05 05:49:46 ubuntu ocserv[196809]: worker[test]: 10.0.0.11 sent periodic stats (in: 0, out: 400) to sec-mod
Oct 05 05:49:46 ubuntu ocserv[184906]: sec-mod: invalidating session of user 'test' (session: 9tgcgO)
Oct 05 05:49:46 ubuntu ocserv[184899]: main[test]:10.0.0.11:60782 user disconnected (reason: user disconnected, rx: 0, tx: 400)
Oct 05 05:49:49 ubuntu ocserv[184906]: sec-mod: using 'pam' authentication to authenticate user (session: 1SySvG)
Oct 05 05:49:49 ubuntu ocserv[184899]: main: main-sec-mod-cmd.c:106: command socket for sec-mod closed
Oct 05 05:49:49 ubuntu ocserv[184899]: main: main.c:1179: error in command from sec-mod
Oct 05 05:49:49 ubuntu ocserv[184899]: main: termination request received; waiting for children to die
Oct 05 05:49:49 ubuntu ocserv[184899]: main:10.0.0.11:50723 user disconnected (reason: unspecified, rx: 0, tx: 0)
Oct 05 05:49:49 ubuntu ocserv[196933]: worker[test]: 10.0.0.11 worker-auth.c:743: error receiving auth reply message
Oct 05 05:49:49 ubuntu ocserv[196933]: **worker[test]: 10.0.0.11 worker-auth.c:1598: failed authentication for 'test'**
Oct 05 05:49:50 ubuntu systemd[1]: ocserv.service: Succeeded.

Bold lines are in red in terminal, not sure if that helps.