Tune secret detection rules
Why are we doing this work
Some rules are noisier than others. The rules in the Secret Detection analyzer are not immune to this reality and could use some attention. As secret detection continues to be a more widely used feature category, we should take a moment to tune some of the rules which are more prone to flagging false positives.
Relevant links
-
keyhacks project
- This project enumerates methods by which keys reported from specific services can be validated. Might be an interesting means of validating detected secrets for true positives?
- GitHub Universe secret detection notes
Non-functional requirements
- Documentation:
- Feature flag:
- Performance:
- Testing:
Implementation plan
Designs
- Show closed items
Relates to
- #342539Backlog
Activity
-
Newest first Oldest first
-
Show all activity Show comments only Show history only
- Thomas Woodham changed milestone to %Backlog
changed milestone to %Backlog
- Thomas Woodham added to epic &6435
added to epic &6435
- 🤖 GitLab Bot 🤖 added typefeature label
added typefeature label
- Author Developer
@zrice - in our last Static Analysis weekly, you mentioned a few rules which could use some tuning. Might you be able to populate some of those in this issue? We'd like to pull this work into the coming quarter while we're focused on reliability and availability work.
Collapse replies @twoodham Taking a look at the Secret Analyzer's gitleaks config it looks like the rule that I was concerned about being noisey, twitter, is not an issue with the Secret Analyzer. The twitter token rule is surrounded by quotations in the analyzer but not in the default gitleaks config (this has been fixed in v7.6.0)
That said, there is some work to be done with the rules:
- Determine which secrets from the semgrep ruleset would be worthwhile to include. Porting the semgrep rules to gitleaks rules is simple since they are mostly just regex.
- Add more restrictive allow-list regex/files. See https://github.com/zricethezav/gitleaks/issues/575#issuecomment-898476714 for additional context.
- Add a tree-sitter false positive reduction feature to gitleaks.
Add more restrictive allow-list regex/files. See https://github.com/zricethezav/gitleaks/issues/575#issuecomment-898476714 for additional context.
This would be considered ~"feature::maintenance" so I will work on refining this issue around this point.
1
- 🤖 GitLab Bot 🤖 added [deprecated] Accepting merge requests label
added [deprecated] Accepting merge requests label
- Thomas Woodham changed milestone to %14.4
changed milestone to %14.4
- Thomas Woodham added workflowplanning breakdown label
added workflowplanning breakdown label
- Taylor McCaslin marked this issue as related to #342539
marked this issue as related to #342539
- Developer
This might be a simple fix related to this issue: #342539
- Author Developer
- Thomas Woodham changed milestone to %14.5
changed milestone to %14.5
- 🤖 GitLab Bot 🤖 removed [deprecated] Accepting merge requests label
removed [deprecated] Accepting merge requests label
- Thomas Woodham changed milestone to %Backlog
changed milestone to %Backlog
- Thomas Woodham removed workflowplanning breakdown label
removed workflowplanning breakdown label
- 🤖 GitLab Bot 🤖 added [deprecated] Accepting merge requests label
added [deprecated] Accepting merge requests label
- Author Developer
@zrice - might you have some rules in mind that should be targeted for tuning?
- Developer
In Early 2022, GitHub will start doing commit rejection for "well-identifiable" secrets, here's an example of 5, if we can find a list of these, I'd like to get rules added:
GitHub also claimed they worked with all these companies to get them to swithc to well identifiable secrets, so we can probably look at their docs and find patterns:
Mentioned in the public GitHub session here: https://youtu.be/ABjrJMjbdRw?t=866
Collapse replies - Developer
GitHub has kindly also provided a list of these API key slugs here: https://docs.github.com/en/code-security/secret-scanning/about-secret-scanning#about-secret-scanning-for-private-repositories
Provider Supported secret API slug Adafruit IO Adafruit IO Key adafruit_io_key Adobe Adobe Device Token adobe_device_token Adobe Adobe Service Token adobe_service_token Adobe Adobe Short-Lived Access Token adobe_short_lived_access_token Adobe Adobe JSON Web Token adobe_jwt Alibaba Cloud Alibaba Cloud Access Key ID alibaba_cloud_access_key_id Alibaba Cloud Alibaba Cloud Access Key Secret alibaba_cloud_access_key_secret Amazon Web Services (AWS) Amazon AWS Access Key ID aws_access_key_id Amazon Web Services (AWS) Amazon AWS Secret Access Key aws_secret_access_key Amazon Web Services (AWS) Amazon AWS Session Token aws_session_token Amazon Web Services (AWS) Amazon AWS Temporary Access Key ID aws_temporary_access_key_id Asana Asana Personal Access Token asana_personal_access_token Atlassian Atlassian API Token atlassian_api_token Atlassian Atlassian JSON Web Token atlassian_jwt Atlassian Bitbucket Server Personal Access Token bitbucket_server_personal_access_token Azure Azure DevOps Personal Access Token azure_devops_personal_access_token Azure Azure SAS Token azure_sas_token Azure Azure Service Management Certificate azure_management_certificate Azure Azure Storage Account Key azure_storage_account_key Beamer Beamer API Key beamer_api_key Checkout.com Checkout.com Production Secret Key checkout_production_secret_key Checkout.com Checkout.com Test Secret Key checkout_test_secret_key Clojars Clojars Deploy Token clojars_deploy_token CloudBees CodeShip CloudBees CodeShip Credential codeship_credential Contentful Contentful Personal Access Token contentful_personal_access_token Databricks Databricks Access Token databricks_access_token Discord Discord Bot Token discord_bot_token Doppler Doppler Personal Token doppler_personal_token Doppler Doppler Service Token doppler_service_token Doppler Doppler CLI Token doppler_cli_token Doppler Doppler SCIM Token doppler_scim_token Doppler Doppler Audit Token doppler_audit_token Dropbox Dropbox Access Token dropbox_access_token Dropbox Dropbox Short Lived Access Token dropbox_short_lived_access_token Duffel Duffel Live Access Token duffel_live_access_token Duffel Duffel Test Access Token duffel_test_access_token Dynatrace Dynatrace Access Token dynatrace_access_token Dynatrace Dynatrace Internal Token dynatrace_internal_token EasyPost EasyPost Production API Key easypost_production_api_key EasyPost EasyPost Test API Key easypost_test_api_key Facebook Facebook Access Token facebook_access_token Fastly Fastly API Token fastly_api_token Finicity Finicity App Key finicity_app_key Flutterwave Flutterwave Live API Secret Key flutterwave_live_api_secret_key Flutterwave Flutterwave Test API Secret Key flutterwave_test_api_secret_key Frame.io Frame.io JSON Web Token frameio_jwt Frame.io Frame.io Developer Token frameio_developer_token FullStory FullStory API Key fullstory_api_key GitHub GitHub Personal Access Token github_personal_access_token GitHub GitHub OAuth Access Token github_oauth_access_token GitHub GitHub Refresh Token github_refresh_token GitHub GitHub App Installation Access Token github_app_installation_access_token GitHub GitHub SSH Private Key github_ssh_private_key GoCardless GoCardless Live Access Token gocardless_live_access_token GoCardless GoCardless Sandbox Access Token gocardless_sandbox_access_token Google Firebase Cloud Messaging Server Key firebase_cloud_messaging_server_key Google Google API Key google_api_key Google Google Cloud Private Key ID google_cloud_private_key_id Google Google Cloud Storage Access Key Secret google_cloud_storage_access_key_secret Google Google Cloud Storage Service Account Access Key ID google_cloud_storage_service_account_access_key_id Google Google Cloud Storage User Access Key ID google_cloud_storage_user_access_key_id Google Google OAuth Client ID google_oauth_client_id Google Google OAuth Client Secret google_oauth_client_secret Grafana Grafana API Key grafana_api_key Hashicorp Terraform Terraform Cloud / Enterprise API Token terraform_api_token Hubspot Hubspot API Key hubspot_api_key Intercom Intercom Access Token intercom_access_token Ionic Ionic Personal Access Token ionic_personal_access_token Ionic Ionic Refresh Token ionic_refresh_token JFrog JFrog Platform Access Token jfrog_platform_access_token JFrog JFrog Platform API Key jfrog_platform_api_key Linear Linear API Key linear_api_key Linear Linear OAuth Access Token linear_oauth_access_token Lob Lob Live API Key lob_live_api_key Lob Lob Test API Key lob_test_api_key Mailchimp Mailchimp API Key mailchimp_api_key Mailgun Mailgun API Key mailgun_api_key MessageBird MessageBird API Key messagebird_api_key New Relic New Relic Personal API Key new_relic_personal_api_key New Relic New Relic REST API Key new_relic_rest_api_key New Relic New Relic Insights Query Key new_relic_insights_query_key New Relic New Relic License Key new_relic_license_key npm npm Access Token npm_access_token NuGet NuGet API Key nuget_api_key Onfido Onfido Live API Token onfido_live_api_token Onfido Onfido Sandbox API Token onfido_sandbox_api_token OpenAI OpenAI API Key openai_api_key Palantir Palantir JSON Web Token palantir_jwt PlanetScale PlanetScale Database Password planetscale_database_password PlanetScale PlanetScale OAuth Token planetscale_oauth_token PlanetScale PlanetScale Service Token planetscale_service_token Plivo Plivo Auth ID plivo_auth_id Plivo Plivo Auth Token plivo_auth_token Postman Postman API Key postman_api_key Proctorio Proctorio Consumer Key proctorio_consumer_key Proctorio Proctorio Linkage Key proctorio_linkage_key Proctorio Proctorio Registration Key proctorio_registration_key Proctorio Proctorio Secret Key proctorio_secret_key Pulumi Pulumi Access Token pulumi_access_token PyPI PyPI API Token pypi_api_token RubyGems RubyGems API Key rubygems_api_key Samsara Samsara API Token samsara_api_token Samsara Samsara OAuth Access Token samsara_oauth_access_token SendGrid SendGrid API Key sendgrid_api_key Sendinblue Sendinblue API Key sendinblue_api_key Sendinblue Sendinblue SMTP Key sendinblue_smtp_key Shippo Shippo Live API Token shippo_live_api_token Shippo Shippo Test API Token shippo_test_api_token Shopify Shopify App Shared Secret shopify_app_shared_secret Shopify Shopify Access Token shopify_access_token Shopify Shopify Custom App Access Token shopify_custom_app_access_token Shopify Shopify Private App Password shopify_private_app_password Slack Slack API Token slack_api_token Slack Slack Incoming Webhook URL slack_incoming_webhook_url Slack Slack Workflow Webhook URL slack_workflow_webhook_url SSLMate SSLMate API Key sslmate_api_key SSLMate SSLMate Cluster Secret sslmate_cluster_secret Stripe Stripe API Key stripe_api_key Stripe Stripe Live API Secret Key stripe_live_secret_key Stripe Stripe Test API Secret Key stripe_test_secret_key Stripe Stripe Live API Restricted Key stripe_live_restricted_key Stripe Stripe Test API Restricted Key stripe_test_restricted_key Stripe Stripe Webhook Signing Secret stripe_webhook_signing_secret Tableau Tableau Personal Access Token tableau_personal_access_token Telegram Telegram Bot Token telegram_bot_token Tencent Cloud Tencent Cloud Secret ID tencent_cloud_secret_id Twilio Twilio Access Token twilio_access_token Twilio Twilio Account String Identifier twilio_account_sid Twilio Twilio API Key twilio_api_key GitHub has kindly also provided a list of these API key slugs here: https://docs.github.com/en/code-security/secret-scanning/about-secret-scanning#about-secret-scanning-for-private-repositories
Slightly confused here, is this api key slugs for their revocation api or slugs from the api key provider?
- Developer
Yeah you're right, these are just internal identifiers rather than slug prefixes
Stripe for example is:
- pk_test
- sk_test
- pk_live
- sk_live
- Developer
We just need to go through vendor API docs to see if they have API slugs that are easy to detect.
I'm happy to help do this work next week. Maybe we just have a fun 30 minute meeting in a google doc for anyone to join and go collect key patterns.
@tmccaslin I'm starting to work my way through this list and compile some regexes. This involves creating a bunch of accounts
😬 for these providers.
- 🤖 GitLab Bot 🤖 removed typefeature label
removed typefeature label
- Thomas Woodham changed milestone to %14.5
changed milestone to %14.5
- Thomas Woodham added workflowplanning breakdown label
added workflowplanning breakdown label
- Zach Rice mentioned in merge request gitlab-org/security-products/analyzers/secrets!124 (closed)
mentioned in merge request gitlab-org/security-products/analyzers/secrets!124 (closed)
- 🤖 GitLab Bot 🤖 removed [deprecated] Accepting merge requests label
removed [deprecated] Accepting merge requests label
- Thomas Woodham changed the description
Compare with previous version changed the description
- Thomas Woodham added Stretch label
added Stretch label
- Zach Rice added workflowin dev label and removed workflowplanning breakdown label
added workflowin dev label and removed workflowplanning breakdown label
- Taylor McCaslin mentioned in issue #345171 (closed)
mentioned in issue #345171 (closed)
- Taylor McCaslin mentioned in merge request gitlab-org/security-products/analyzers/kics!9 (closed)
mentioned in merge request gitlab-org/security-products/analyzers/kics!9 (closed)
Topic: Secrets Starter
@gitlab-org/secure/static-analysis-be here is the initial result from my spread sheet work I've been doing the past couple days. My process was to create accounts for all these providers and generate three api keys for each entry. I scrambled one of those api keys and used that to populate this table. The
Notes
column is a regex-ish guide to start writing rules.h = hexadecimal A/a = alphanumerical s = alphanumerical + special characters like "_" and "-" d = digits
UNIQUE Provider Description Notes Value (scambled) n Adobe Client ID (Oauth Web) [32h] 4ab4b080d9ce4072a6be2629c399d653 y Adobe Client Secret [p8e-][32a] p8e-mBc6QIzXhnmHGhlfbSM8nNmdeF6w1fhj y Alibaba AccessKey ID [LTAI][24a] LTAI5t1oc8FSbpswMPd3JTKN n Alibaba AccessKey Secret [30a] 76DyIO8tu5s6ZfNPMbyBMH1PZGsRzN n Asana Client ID [16d] 1204326363763489 n Asana Client Secret [32h] d0e94828b0549eee7368e53f6cb41d17 n Atlassian API Token [24a] yZIjJoDDNbJl1WlrASpM463E n BitBucket Client ID [32a] Zf3D0LXCM3EIMbgJpUNnkRtOfOueHznB n BitBucket Client Secret [64s] HKH6h-ik4DgO5pG7ZbxtyItNygJMZ1D6z_vkBdhfCmWFyhjLvbiZEF3O-8WV9A76 n BitBucket (more) Client Secret [64s] bKz6XewBpPOooBQDFFmNqBRBrYkrN9S67jO5axILYXx9o6xFj_iMkMP5wyissyQJ y Beamer API key [b_][44s] b_Wk3IIH700rIMUnNdJOAdikk45X7eDelZTTCzJ0Skohs= y Checkout.com API Public Sandbox Key need to verify all of checkout but first need sales account pk_test_xxxxxxxxxxxxxxxxxxxxxxxxxx y Checkout.com API Public Producton Key pk_xxxxxxxxxxxxxxxxxxxxxxxxxx y Checkout.com API Secret Sandbox Key sk_test_xxxxxxxxxxxxxxxxxxxxxxxxxx y Checkout.com API Secret Production sk_xxxxxxxxxxxxxxxxxxxxxxxxxx y Clojars API Token [CLOJARS_][60h] CLOJARS_34bf0e88955ff5a1c328d6a7491acc4f48e865a7b8dd4d70a70749037443 n CloudBees UserToken We want cloudbees codeship but unsure how to get to their api dashboard 7772c5cc-9b44-4863-9c59-93f31bbbfe8b n Contentful Delivery API [43s] sjo4HwBHz5ZQwCO-5h8yMF5qfjRDeOBdnC-OefFGcn0 n Contentful Preview API (Draft status shit) [43s] 0TNLpl9EbaoiefqjsMLM1hQ-bwZuZbJgmnD4teg2o59 n Contentful Space ID (part of slug) [12a] 75aahd5p1kj7 y Databricks API Token [dapi][32h] dapi1a3c3c45d67790b1f234667a8bc8012c n Discord Public Key [64h] e7322523fb86ed64c836a979cf8465fbd436378c653c1db38f9ae87bc62a6fd5 n Discord Client ID [18d] 908561293872267899 n Discord Client Secret [32s] 8dyfuiRyqFvVc3RRr_edRk-fK__JItpZ y Doppler Personal API Token [dp.pt.][43A] dp.pt.gj5QCuhZaap37aCbpzP1Duxm7susBo1KFV8mNt4iJXN n Dropbox App Key [15a] ds7qoiqb4je0rqq n Dropbox App Secret [15a] qfxr7lh83tbkmaz y Dropbox Short Lived Access Token [sl.][135s] sl.A7pTsyLKWoDvMKXhdKEz9R6ZO7Tf6HJqr0jdogqnCrYrYROA75rSFomkqvEbAdTb3gQVbP18yVkfPhChqVtNMt6398ipek5_StUTrHolTfBizojfwPNu43MeVF4-57J_XRwOESs y Dropbox Long Lived Access Token [11A][AAAAAAAAAA][43s] ungi76nUb5QAAAAAAAAAAfyt1jX1bdsZhuIysBl6CpoZNjWg_OM_KEunvbMNW-Gz y Duffel App test token [duffel_test_][43A] duffel_test_gXjm31mMe36vCiHYTtHHfCmNGwxj-Ib1o4liEZdTKCT m Duffel App live token not sure need a proper account for this, probably get a community member to help us out y Dynatrace API v2 [dt0c01.][24A][.][64A] dt0c01.RBOVG3QKMDQASN2TP4ZOOBYS.M3JWBBGAMSEFBYR3AZI62OOROQRBJ2JDJO66UHDP4Z77B27IENHOTCBVP7TQ2QLI y EasyPost Production API [EZAK][54A] EZAK20d88ba26944531abd4a7eb4b5f2bff9w9bTQT4LacmgoGcVqfA2zB y EasyPost Test API [EZTK][54A] EZTK20d88ba26944531abd3a7eb4b5f2bff9gLvg0zLoOq19JvCv6MgTBB Facebook ??? n Fastly Personal API Token [32s] uhZtofOcNnzoH6F5-m0bzsLvCqIjzNFG n Finicity Partner Secret [20A] F6uaXf3yEdBwnAg35q1Z n Finicity App Key [32h] 1863cd09a446ede7669062c4ee2bf7ca y Flutterweave Public Key [FLWPUBK_TEST-][32h][-X] FLWPUBK_TEST-3ffc862cbaff344d1ad33d81109e0db6-X y Flutterweave Secret Key [FLWSECK_TEST-][32h][-X] FLWSECK_TEST-ca9f9a7af7953aed44391b1338b6f9ef-X y Flutterweave Encrypted Key [FLWPUBK_TEST][12h] FLWSECK_TEST1a4043e5ed0c y Frame.io PAT [fio-u-][64s] fio-u-utrwGPpiRfGmkSu-S73kwmImy3YQH7pIMvBNWF2u3Ee2xZy6rz1Kx1YeZX07HyIA y fullstory API Key [MTZDRVM5L3phY2hyaWNlOUBnbWFpbC5jb206][52s] MTZDRVM5L3phY2hyaWNlOUBnbWFpbC5jb206Bo8qLA8xfo7Zgf39OqxVkz7a+P5IoSfrwG8L3xMZ4U0u/STP5B43 y GoCardless API Token [live_][40s] live_AOAWYR5Vbq070_oT6gBWelZUSYukL-wvtLr524d2 y Grafana API Key [eyJrIjoi][72-81s] eyJrIjoiWWpNc1ZPS5RjdjdHTWlyVmtTcjI0Vk9xN1RkOGI1V3YiLCJuIjoieZl5b19vb30vb23iLCJpZCI6MX0= y Hashicorp Terraform Org Token [14A][.atlastv1.][68A] Nz0Vmo1pPzmszg.atlasv1.mZc9GYppy9YzIObgFHb6jFMfsA0AcLczXkDFyt5OskkFrcsmTCd4g4BQyyzpUb8qf9o y Hashicorp Terraform User Token [14A][.atlastv1.][68A] Soy7om6xDDcatg.atlasv1.XHZdIkZvIKJ3iFYvaQ9yqbyc8z4Hw0dbByFuNTRZ3BfemAy1j3jCZF1C9YTYhycpY8Z y Hubspot API Key [8h][-][4h][-][4h][-][4h][-][12h]
dcd37820-bbf6-418e-b187-f471c64e36d4 y Intercom App API Key dG9rOjVjMjRjZWQ3XkZkNMJfNKkzMj9iNDI4XzhmMDlkNCI4NTdiNDoxOjB= n Intercom Client ID [8h][-][4h][-][4h][-][4h][-][12h]
9173gc12-91f2-4825-8ad9-805232b1d98b n Intercom Client secret [8h][-][4h][-][4h][-][4h][-][12h]
a0dc7725-c719-41cb-b9dd-0d50b262dfc6 y Ionic PAT [ion_][42A] ion_ukKfe2u7TrYBLzhtam6G46iqAcj1BU2llPb40iHm5K JFrog couldn't get it working y Linear PAT [lin_api_][40A] lin_api_tPmBQyIDelPeLZZB4rUiMZJQqPMIggqOEK0k3Bbo n Linear Client ID [32h] b4a7f4d7d9ee1fbccb62eb3ddc2c2a42 n Linear Client Secret [32h] 467d84b435f37a33ecb603e9883bfed1 y Lob Secret API Live [live_][35h] live_5c03155f2266f3e00c20179c5b65502ab92 y Lob Secret API Test [test_][35h] test_c83b8ca059938e5281ea376f834daa68a49 y Lob Publishable API Live [live_pub_][31h] live_pub_0e1fe852b24b86f5cbd57192df2cabb y Lob Publishable API Test [test_pub_][31h] test_pub_be51547e7c05d83700ca3ef5bc517c3 y Mailchimp API Key [32h][-us20] 37f8bed34221abdc9c658654592b1128-us20 y Mailgun Private API Key [key-][32h] key-dcc68f3f32ecee4dbe156cbd869db54f y Mailgun Public Validation Key [pubkey-][32h] pubkey-1e14c9dbe3c1b868554e745247d5d036 n Mailgun Http webhook signing key [32h][-][8h][-][8h]
275ca4bf433194fb9e41cb1d30b8d926-22ccdde5-933fceee y Mapbox Access Token [pk.][83A] pk.eyJ2IjoiZmV4YWEiLBJhIjoiJ2t2bGhvaFJzZGdhOBJ3bWFnbjh0dBd4diJ9.fxmBJ3SL31MF6t0JDXk7Yq n MessageBird API Key Live [25A] QZ2o4gkpz9lldgum1V1yvmEKs n MessageBird API Client [8h][-][4h][-][4h][-][4h][-][12h]
f194d5db-5826-24c7-b421-2f5897308c49 y New Relic User API Key [NRAK-][27A] NRAK-FFDW5VZ3B292B69NU9BL7F07OI9 n New Relic User API ID [64A] 8ECC3D9A863BDE368D12EBC661C697212DEF4549B1EE56B5BDAAFA1CD7F2D979 y New Relic Ingest Browser [NRJS-][19h] NRJS-96bbe36499d469a6bc4 y npm access token [npm_][36A] npm_JEbsYXyMwnGPcqZwk6fXplQNjmDHuJ0zoriU nuget need a microsoft account onfido need sales account openai need account palantir need account y planetscale password [pscale_pw__][43s] pscale_pw__qerkvUp-boFZrRdC0In6y9uWjPSYdWAK1MCV2iORmZ planetscale host lip3w7munvns.us-east-3.psdb.cloud y planetscale service token [pscale_tkn_][43s] pscale_tkn_r7EGkxciX8Jdr_PjZ6O1NFP-47051zzcAqORihDW3Zf plivo needs work accunt y postman api key [PMAK-][24h][-][34h]
PMAK-628534d1b6d710004b0cc1c1-0b7434831ddd15645fe09352dcc4d048b6 proctorio need account y pulumi api token [pul1-][40h] pul-1fbfa873a23b100a42f877f56fb7b3f61c1e2077 y rubygem api token [rubygems_][48h] rubygems_ac4f80d5d6c06ecdd417f11bcab2b614735fcf082d851aeb samsara need account :( y sengrid api key [SG.][66s] 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 y sendinblue api key [xkeysib-][81h] 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 y shippo live api token [shippo_live_][40h] shippo_live_bbfb50ece8b8e0cddb45698e55401cfebc48e99b y shippo test api token [shippo_test_][40h] shippo_test_6a1b78ccf8a21263b27cfc6c2f2081a614a6e8d5 n sslmate api token Needs verification 41459_bEZFFRrKjxsOB4bZuBjt tableau waiting on account telegram todo tenecent need credit card n linkedin client secret [16A] JBxJIyEInUlAVLwN n linkedin client id [14A] 88wj9qlu7vcbtj n twitch secret [26A] zwqjvffo863374zioz1gd26thgsmu5 y typeform api token [tfp_][59s] tfp_HeDK4ZeCwwCHdvSnVnzKtjiyB1rLFyzspa7Y3Fo9yAp8_3pcodefNH4DhnE 2 Collapse replies - Developer
Phenomenal work, this far exceeds my hopes for this update to our rules. I think another tactic we can use is to provide instructions in the docs for how vendors can submit their API keys so we just have a regular influx of key patterns. Maybe we setup a confidential issue template that we link to.
2 Funny you mention that because that is something that @mhenriksen has proposed more formally here https://gitlab.com/gitlab-com/gl-security/appsec/appsec-team/-/issues/213
- Maintainer
👏 awesome work @zrice!! - Contributor
Great job @zrice!
- Taylor McCaslin mentioned in issue #345453 (closed)
mentioned in issue #345453 (closed)
- Taylor McCaslin marked this issue as related to #345453 (closed)
marked this issue as related to #345453 (closed)
- Taylor McCaslin mentioned in merge request gitlab-com/www-gitlab-com!93727 (merged)
mentioned in merge request gitlab-com/www-gitlab-com!93727 (merged)
Just found https://docs.gitguardian.com/secrets-detection/detectors/generics/generic_high_entropy_secret
We should probably update our generic secret rule
Collapse replies Going to make a minor adjustment to non-unique rules in the spread sheet to match some of the information described in the link above. Below is a little explainer.
Example of old rule
Beginning string quotation │ End string quotation │ │ ▼ ▼ (?i)(discord)(.{0,20})['\"][a-z0-9]{16}['\"] ▲ ▲ ▲ ┌───────────┘ │ │ │ │ │ Secret identifier │ identifier suffix AND assignment symbol ex: '_api_token ='
This could lead to false positives like
discord_users(os.getEnv("DISCORDAPITOKENT")
. The example rule also does not give us goodidentifier
values. In this case the identifier group matches on justdiscord
where the identifier suffix and assignment symbol group gets matched on_users(os.getEnv(
. It would be nice if we could get our groups matching to be more accurate. Regex101 link to see this in actionTweaked Rule
Beginning string quotation │ End string quotation │ │ ▼ ▼ (?i)(discord[a-z0-9_ .\-,]{0,25})(=|>|:=|\|\|:|<=|=>|:).{0,5}['\"]([a-z0-9]{16})['\"] ▲ ▲ ▲ │ │ │ │ │ │ assignment symbol identifier name Secret
This rule does not match the false positive
discord_users(os.getEnv("DISCORDAPITOKENT")
. Given a true-positive likediscord_api_token := "ZsQq3iqb4je0rqq1"
we get a match and useful groupings for entropy checking if we choose. Regex101 link to see this in actioncc @gitlab-org/secure/static-analysis-be a little more context for writing secret detection rules
Edited by Zach Rice1 - Maintainer
@zrice - Mentioning it here in case it's useful. I suggest noting anything you learn from this work that might be useful to vendors in contributing their secret detection rules.
- Zach Rice mentioned in merge request gitlab-org/security-products/analyzers/secrets!126 (merged)
mentioned in merge request gitlab-org/security-products/analyzers/secrets!126 (merged)
- Maintainer
(Comment copied from gitlab-org/security-products/analyzers/secrets!126 (comment 736284902))
@zrice - For documentation changes, I suggest:
- In https://docs.gitlab.com/ee/user/application_security/secret_detection/#supported-secrets, remove mention of individual vendors. Instead, list only categories of secrets, for example: encryption keys, cloud services.
- Link to the
gitleaks.toml
file (just as you suggested). - Link to #345453 (closed) to help anyone who wants to contribute a rule (just as you suggested).
- 🤖 GitLab Bot 🤖 changed milestone to %14.6
changed milestone to %14.6
- 🤖 GitLab Bot 🤖 added missed:14.5 label
added missed:14.5 label
- Zach Rice mentioned in merge request !74909 (merged)
mentioned in merge request !74909 (merged)
- Zach Rice removed missed:14.5 label
removed missed:14.5 label
- Zach Rice closed
closed