Automate fetching SSH fingerprints of keys to ban - #365056

What does this MR do and why?

This task is an improvement of !87541 (merged). It does two things.

1. When a user is uploading their key, it is now scanning the key list at config/security/banned_ssh_keys.yml instead of being hard coded.
2. A new rake task will read/clone a git repository and update the yaml config config/security/banned_ssh_keys.yml. One thing to mention is that this rake task will try to add all the public SSH keys in the git repository without further checking if it is a bad ssh key or not (but if this is needed then how?). Further, it will only upload 200MB to the yml file incase there are too many keys to degrade the performance.

Screenshots or screen recordings

Screenshots are required for UI changes, and strongly recommended for all other merge requests.

The UI screenshot will be the exactly the same as pervious MR.

The command logs of updating banned ssh keys is shown as below.

gitpod /workspace/gitlab-development-kit/gitlab (issue-365056) $ bin/rake "gitlab:security:update_banned_ssh_keys[https://github.com/rapid7/ssh-badkeys]"
I, [2022-11-23T13:10:39.271472 #132397]  INFO -- : start to clone the git repository at /tmp/d20221123-132397-3p965b/ssh-badkeys
I, [2022-11-23T13:10:39.758336 #132397]  INFO -- : Git clone finished, and now add bad keys to config/security/banned_ssh_keys.yml.
I, [2022-11-23T13:10:39.759766 #132397]  INFO -- : update bad ssh keys in /tmp/d20221123-132397-3p965b/ssh-badkeys/authorized/array-networks-vapv-vxag.pub
I, [2022-11-23T13:10:39.760368 #132397]  INFO -- : update bad ssh keys in /tmp/d20221123-132397-3p965b/ssh-badkeys/authorized/barracuda_load_balancer_vm.pub
I, [2022-11-23T13:10:39.760678 #132397]  INFO -- : update bad ssh keys in /tmp/d20221123-132397-3p965b/ssh-badkeys/authorized/ceragon-fibeair-cve-2015-0936.pub
I, [2022-11-23T13:10:39.760963 #132397]  INFO -- : update bad ssh keys in /tmp/d20221123-132397-3p965b/ssh-badkeys/authorized/exagrid-cve-2016-1561.pub
I, [2022-11-23T13:10:39.761180 #132397]  INFO -- : update bad ssh keys in /tmp/d20221123-132397-3p965b/ssh-badkeys/authorized/f5-bigip-cve-2012-1493.pub
I, [2022-11-23T13:10:39.761508 #132397]  INFO -- : update bad ssh keys in /tmp/d20221123-132397-3p965b/ssh-badkeys/authorized/loadbalancer.org-enterprise-va.pub
I, [2022-11-23T13:10:39.761768 #132397]  INFO -- : update bad ssh keys in /tmp/d20221123-132397-3p965b/ssh-badkeys/authorized/monroe-dasdec-cve-2013-0137.pub
I, [2022-11-23T13:10:39.762060 #132397]  INFO -- : update bad ssh keys in /tmp/d20221123-132397-3p965b/ssh-badkeys/authorized/quantum-dxi-v1000.pub
I, [2022-11-23T13:10:39.762302 #132397]  INFO -- : update bad ssh keys in /tmp/d20221123-132397-3p965b/ssh-badkeys/authorized/vagrant-default.pub
I, [2022-11-23T13:10:39.762834 #132397]  INFO -- : update bad ssh keys in /tmp/d20221123-132397-3p965b/ssh-badkeys/host/Actiontec_q2000_rsa.pub
I, [2022-11-23T13:10:39.763091 #132397]  INFO -- : update bad ssh keys in /tmp/d20221123-132397-3p965b/ssh-badkeys/host/Alice_1121_rsa.pub
I, [2022-11-23T13:10:39.763320 #132397]  INFO -- : update bad ssh keys in /tmp/d20221123-132397-3p965b/ssh-badkeys/host/Cisco_RV315W_dsa.pub
I, [2022-11-23T13:10:39.763610 #132397]  INFO -- : update bad ssh keys in /tmp/d20221123-132397-3p965b/ssh-badkeys/host/Cisco_RV315W_rsa.pub
I, [2022-11-23T13:10:39.763849 #132397]  INFO -- : update bad ssh keys in /tmp/d20221123-132397-3p965b/ssh-badkeys/host/Cisco_rtp300_dsa.pub
I, [2022-11-23T13:10:39.764085 #132397]  INFO -- : update bad ssh keys in /tmp/d20221123-132397-3p965b/ssh-badkeys/host/Cisco_rtp300_rsa.pub
I, [2022-11-23T13:10:39.764351 #132397]  INFO -- : update bad ssh keys in /tmp/d20221123-132397-3p965b/ssh-badkeys/host/Cisco_rv120w_dsa.pub
I, [2022-11-23T13:10:39.764639 #132397]  INFO -- : update bad ssh keys in /tmp/d20221123-132397-3p965b/ssh-badkeys/host/Cisco_rv120w_rsa.pub
I, [2022-11-23T13:10:39.764963 #132397]  INFO -- : update bad ssh keys in /tmp/d20221123-132397-3p965b/ssh-badkeys/host/Comtrend_AR5387UN_rsa.pub
I, [2022-11-23T13:10:39.765208 #132397]  INFO -- : update bad ssh keys in /tmp/d20221123-132397-3p965b/ssh-badkeys/host/EVW3226_rsa.pub
I, [2022-11-23T13:10:39.765489 #132397]  INFO -- : update bad ssh keys in /tmp/d20221123-132397-3p965b/ssh-badkeys/host/Edimax_AR-7167_dsa.pub
I, [2022-11-23T13:10:39.765754 #132397]  INFO -- : update bad ssh keys in /tmp/d20221123-132397-3p965b/ssh-badkeys/host/Edimax_AR-7167_rsa.pub
I, [2022-11-23T13:10:39.765977 #132397]  INFO -- : update bad ssh keys in /tmp/d20221123-132397-3p965b/ssh-badkeys/host/Huawei_bm626_dsa.pub
I, [2022-11-23T13:10:39.766243 #132397]  INFO -- : update bad ssh keys in /tmp/d20221123-132397-3p965b/ssh-badkeys/host/Huawei_bm626_rsa.pub
I, [2022-11-23T13:10:39.766490 #132397]  INFO -- : update bad ssh keys in /tmp/d20221123-132397-3p965b/ssh-badkeys/host/Innacomm_w3400v_rsa.pub
I, [2022-11-23T13:10:39.766701 #132397]  INFO -- : update bad ssh keys in /tmp/d20221123-132397-3p965b/ssh-badkeys/host/Linksys_X1000_rsa.pub
I, [2022-11-23T13:10:39.766897 #132397]  INFO -- : update bad ssh keys in /tmp/d20221123-132397-3p965b/ssh-badkeys/host/Moxa_6150_rsa.pub
I, [2022-11-23T13:10:39.767125 #132397]  INFO -- : update bad ssh keys in /tmp/d20221123-132397-3p965b/ssh-badkeys/host/Moxa_ia240_dsa.pub
I, [2022-11-23T13:10:39.767384 #132397]  INFO -- : update bad ssh keys in /tmp/d20221123-132397-3p965b/ssh-badkeys/host/Moxa_ia240_rsa.pub
I, [2022-11-23T13:10:39.767604 #132397]  INFO -- : update bad ssh keys in /tmp/d20221123-132397-3p965b/ssh-badkeys/host/Ont_g4020w_rsa.pub
I, [2022-11-23T13:10:39.767819 #132397]  INFO -- : update bad ssh keys in /tmp/d20221123-132397-3p965b/ssh-badkeys/host/Pace_V5542_dsa.pub
I, [2022-11-23T13:10:39.768059 #132397]  INFO -- : update bad ssh keys in /tmp/d20221123-132397-3p965b/ssh-badkeys/host/Quanta_LTE.pub
I, [2022-11-23T13:10:39.768309 #132397]  INFO -- : update bad ssh keys in /tmp/d20221123-132397-3p965b/ssh-badkeys/host/Sagemcom_2740_rsa.pub
I, [2022-11-23T13:10:39.768573 #132397]  INFO -- : update bad ssh keys in /tmp/d20221123-132397-3p965b/ssh-badkeys/host/Sagemcom_sx682_dsa.pub
I, [2022-11-23T13:10:39.768809 #132397]  INFO -- : update bad ssh keys in /tmp/d20221123-132397-3p965b/ssh-badkeys/host/Seagate_GoFlex_dsa.pub
I, [2022-11-23T13:10:39.769038 #132397]  INFO -- : update bad ssh keys in /tmp/d20221123-132397-3p965b/ssh-badkeys/host/Seagate_GoFlex_rsa.pub
I, [2022-11-23T13:10:39.769342 #132397]  INFO -- : update bad ssh keys in /tmp/d20221123-132397-3p965b/ssh-badkeys/host/Telefonica-de-Espana_rsa.pub
I, [2022-11-23T13:10:39.769637 #132397]  INFO -- : update bad ssh keys in /tmp/d20221123-132397-3p965b/ssh-badkeys/host/Tplink_tdw8960n-V1_rsa.pub
I, [2022-11-23T13:10:39.769861 #132397]  INFO -- : update bad ssh keys in /tmp/d20221123-132397-3p965b/ssh-badkeys/host/Tplink_w8950n_rsa.pub
I, [2022-11-23T13:10:39.770085 #132397]  INFO -- : update bad ssh keys in /tmp/d20221123-132397-3p965b/ssh-badkeys/host/Tplink_w8950nd_rsa.pub
I, [2022-11-23T13:10:39.770319 #132397]  INFO -- : update bad ssh keys in /tmp/d20221123-132397-3p965b/ssh-badkeys/host/Trendnet_tdmc500_rsa.pub
I, [2022-11-23T13:10:39.770564 #132397]  INFO -- : update bad ssh keys in /tmp/d20221123-132397-3p965b/ssh-badkeys/host/Trendnet_tew715apo_dsa.pub
I, [2022-11-23T13:10:39.770840 #132397]  INFO -- : update bad ssh keys in /tmp/d20221123-132397-3p965b/ssh-badkeys/host/Trendnet_tew715apo_rsa.pub
I, [2022-11-23T13:10:39.771056 #132397]  INFO -- : update bad ssh keys in /tmp/d20221123-132397-3p965b/ssh-badkeys/host/Trendnet_tew816drm_dsa.pub
I, [2022-11-23T13:10:39.771300 #132397]  INFO -- : update bad ssh keys in /tmp/d20221123-132397-3p965b/ssh-badkeys/host/Trendnet_tew816drm_rsa.pub
I, [2022-11-23T13:10:39.771526 #132397]  INFO -- : update bad ssh keys in /tmp/d20221123-132397-3p965b/ssh-badkeys/host/Trendnet_tvip310pi_dsa.pub
I, [2022-11-23T13:10:39.771781 #132397]  INFO -- : update bad ssh keys in /tmp/d20221123-132397-3p965b/ssh-badkeys/host/Trendnet_tvip310pi_rsa.pub
I, [2022-11-23T13:10:39.771995 #132397]  INFO -- : update bad ssh keys in /tmp/d20221123-132397-3p965b/ssh-badkeys/host/Westermo_MRD310_dsa.pub
I, [2022-11-23T13:10:39.772236 #132397]  INFO -- : update bad ssh keys in /tmp/d20221123-132397-3p965b/ssh-badkeys/host/Westermo_MRD310_rsa.pub
I, [2022-11-23T13:10:39.772447 #132397]  INFO -- : update bad ssh keys in /tmp/d20221123-132397-3p965b/ssh-badkeys/host/Zhone_6512a1_rsa.pub
I, [2022-11-23T13:10:39.772668 #132397]  INFO -- : update bad ssh keys in /tmp/d20221123-132397-3p965b/ssh-badkeys/host/Zyxel_fsg2200_rsa.pub
I, [2022-11-23T13:10:39.772912 #132397]  INFO -- : update bad ssh keys in /tmp/d20221123-132397-3p965b/ssh-badkeys/host/Zyxel_p870h_rsa.pub
I, [2022-11-23T13:10:39.773126 #132397]  INFO -- : update bad ssh keys in /tmp/d20221123-132397-3p965b/ssh-badkeys/host/Zyxel_pmg1006_dsa.pub
I, [2022-11-23T13:10:39.773421 #132397]  INFO -- : update bad ssh keys in /tmp/d20221123-132397-3p965b/ssh-badkeys/host/Zyxel_pmg1006_rsa.pub
I, [2022-11-23T13:10:39.773655 #132397]  INFO -- : update bad ssh keys in /tmp/d20221123-132397-3p965b/ssh-badkeys/host/Zyxel_sbg3300_rsa.pub
I, [2022-11-23T13:10:39.773906 #132397]  INFO -- : update bad ssh keys in /tmp/d20221123-132397-3p965b/ssh-badkeys/host/advantech_eki_rsa.pub
I, [2022-11-23T13:10:39.774165 #132397]  INFO -- : update bad ssh keys in /tmp/d20221123-132397-3p965b/ssh-badkeys/host/kali-rpi2.pub
I, [2022-11-23T13:10:39.774428 #132397]  INFO -- : update bad ssh keys in /tmp/d20221123-132397-3p965b/ssh-badkeys/host/moovbox_host_dsa.pub
I, [2022-11-23T13:10:39.774684 #132397]  INFO -- : update bad ssh keys in /tmp/d20221123-132397-3p965b/ssh-badkeys/host/moovbox_host_rsa.pub
I, [2022-11-23T13:10:39.774911 #132397]  INFO -- : update bad ssh keys in /tmp/d20221123-132397-3p965b/ssh-badkeys/host/tandberg-vcs.pub
I, [2022-11-23T13:10:39.775217 #132397]  INFO -- : update bad ssh keys in /tmp/d20221123-132397-3p965b/ssh-badkeys/host/zyxel-q100_rsa.pub
I, [2022-11-23T13:10:39.775476 #132397]  INFO -- : update bad ssh keys in /tmp/d20221123-132397-3p965b/ssh-badkeys/host/zyxel-vmg1312_rsa.pub
...

How to set up and validate locally

1. run the rake task to add more ssh keys. $ bin/rake "gitlab:security:update_banned_ssh_keys[https://github.com/BenBE/kompromat.git]". There are a lot of keys in this repository, and it will only add keys until the yaml file is close to 200MB.
2. check if config/security/banned_ssh_keys.yml is updated. git diff config/security/banned_ssh_keys.yml

• some additional thinking.
  • did not check the git repo carefully as this is mainly for admin to operate. It needs an admin to decide if the repository is good enough or not.

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

• I have evaluated the MR acceptance checklist for this MR.

Related issues and MRs

• Related to Prevent users from using known insecure public key (#24614 - closed)