2 factor-authentication not working even after upgrade to omnibus 8.9.6

I have been having issue with 2 factor-authentication since like version 8.2.x or so and i have performed like 3 upgrades already and now on current latest 8.9.6 yesterday and yet 2 factor-authentication does not work for any user on the gitlab server. The Google Authenticator app gets the code but when i enter the pin code, it returns invalid code every single time, the same error message i have been getting since version 8.2.x when i started trying out the 2 factor

Where do i look to fix this or is this some setting in the gitlab.rb file or something i need to do? This is running latest code and everything is working fine so why is 2 factor not working?

Please help

Thanks

P.S. I migrated to omnibus from a source installation from version 6 to version 8, so not sure if that was an issue. Also i just need the 2 factor to work so is there a way to just make the 2 factor start from fresh and start working?