DAST site profiles containing invalid URLs breaks the site profiles list page
Summary
Reproduced on GitLab.com and 14.6.2-ee
If a DAST site profile contains an invalid URL, the Manage DAST scans page no longer shows a list of Site profiles under the project's Security & Compliance > Configuration > Site Profiles page and instead displays a Could not fetch site profiles. Please refresh the page, or try again later.
message.
Steps to reproduce
- Create a DAST site profile with a valid URL (eg.
https://site1.com/
). Then save it.- It should be noted that if you try to create an invalid URL when creating the site profile for the first time, the validation works as expected and will prevent the save.
- Edit the site profile in step 1, and then change the URL to
https://site1.com/, https://site2.com/
. Then save it. - If you try to view the project's Security & Compliance > Configuration > Site Profiles page, you will now see a
Could not fetch site profiles. Please refresh the page, or try again later.
message.
Example Project
https://gitlab.com/anton/263110-dast-site-profiles/-/security/configuration/dast_scans#site-profiles
What is the current bug behavior?
- When a site profile is being edited, the URL isn't validated/checked.
- When the site profiles list is being shown and a site profile has an invalid URL, this breaks the page load.
What is the expected correct behavior?
- Site profile URLs should be validated when you are editing an existing one.
Relevant logs and/or screenshots
If you inspect the Network tab in your browser, you will see a failed POST 500
request to /api/graphql
and the server log /var/log/gitlab/gitlab-rails/production_json.log
will show a URI::InvalidURIError
bad URI(is not URI?): \"https://site1.com/, https://site2.com/\"
message or similar.
Expand for full log
{ "method":"POST", "path":"/api/graphql", "format":"*/*", "controller":"GraphqlController", "action":"execute", "status":500, "time":"2022-01-21T02:57:53.588Z", "params":[ { "key":"_json", "value":[ { "operationName":"DastProfiles", "variables":"[FILTERED]", "query":"query DastProfiles($fullPath: ID!, $after: String, $before: String, $first: Int, $last: Int) {\n project(fullPath: $fullPath) {\n id\n dastProfiles(after: $after, before: $before, first: $first, last: $last) {\n pageInfo {\n ...PageInfo\n __typename\n }\n edges {\n node {\n id\n name\n dastSiteProfile {\n id\n targetUrl\n __typename\n }\n dastScannerProfile {\n id\n scanType\n __typename\n }\n dastProfileSchedule {\n id\n active\n startsAt\n timezone\n cadence {\n unit\n duration\n __typename\n }\n __typename\n }\n branch {\n name\n exists\n __typename\n }\n editPath\n __typename\n }\n __typename\n }\n __typename\n }\n __typename\n }\n}\n\nfragment PageInfo on PageInfo {\n hasNextPage\n hasPreviousPage\n startCursor\n endCursor\n __typename\n}\n" }, { "operationName":"DastSiteProfiles", "variables":"[FILTERED]", "query":"query DastSiteProfiles($fullPath: ID!, $after: String, $before: String, $first: Int, $last: Int) {\n project(fullPath: $fullPath) {\n id\n siteProfiles: dastSiteProfiles(\n after: $after\n before: $before\n first: $first\n last: $last\n ) {\n pageInfo {\n ...PageInfo\n __typename\n }\n edges {\n cursor\n node {\n id\n profileName\n normalizedTargetUrl\n targetUrl\n targetType\n editPath\n validationStatus\n referencedInSecurityPolicies\n auth {\n enabled\n url\n usernameField\n passwordField\n username\n __typename\n }\n excludedUrls\n requestHeaders\n __typename\n }\n __typename\n }\n __typename\n }\n __typename\n }\n}\n\nfragment PageInfo on PageInfo {\n hasNextPage\n hasPreviousPage\n startCursor\n endCursor\n __typename\n}\n" }, { "operationName":"DastScannerProfiles", "variables":"[FILTERED]", "query":"query DastScannerProfiles($fullPath: ID!, $after: String, $before: String, $first: Int, $last: Int) {\n project(fullPath: $fullPath) {\n id\n scannerProfiles: dastScannerProfiles(\n after: $after\n before: $before\n first: $first\n last: $last\n ) {\n pageInfo {\n ...PageInfo\n __typename\n }\n edges {\n cursor\n node {\n id\n profileName\n spiderTimeout\n targetTimeout\n scanType\n useAjaxSpider\n showDebugMessages\n editPath\n referencedInSecurityPolicies\n __typename\n }\n __typename\n }\n __typename\n }\n __typename\n }\n}\n\nfragment PageInfo on PageInfo {\n hasNextPage\n hasPreviousPage\n startCursor\n endCursor\n __typename\n}\n" }, { "operationName":"DastFailedSiteValidations", "variables":"[FILTERED]", "query":"query DastFailedSiteValidations($fullPath: ID!) {\n project(fullPath: $fullPath) {\n id\n validations: dastSiteValidations(status: FAILED_VALIDATION) {\n nodes {\n id\n normalizedTargetUrl\n __typename\n }\n __typename\n }\n __typename\n }\n}\n" } ] }, { "key":"graphql", "value":{ "_json":[ { "operationName":"DastProfiles", "variables":"[FILTERED]", "query":"query DastProfiles($fullPath: ID!, $after: String, $before: String, $first: Int, $last: Int) {\n project(fullPath: $fullPath) {\n id\n dastProfiles(after: $after, before: $before, first: $first, last: $last) {\n pageInfo {\n ...PageInfo\n __typename\n }\n edges {\n node {\n id\n name\n dastSiteProfile {\n id\n targetUrl\n __typename\n }\n dastScannerProfile {\n id\n scanType\n __typename\n }\n dastProfileSchedule {\n id\n active\n startsAt\n timezone\n cadence {\n unit\n duration\n __typename\n }\n __typename\n }\n branch {\n name\n exists\n __typename\n }\n editPath\n __typename\n }\n __typename\n }\n __typename\n }\n __typename\n }\n}\n\nfragment PageInfo on PageInfo {\n hasNextPage\n hasPreviousPage\n startCursor\n endCursor\n __typename\n}\n" }, { "operationName":"DastSiteProfiles", "variables":"[FILTERED]", "query":"query DastSiteProfiles($fullPath: ID!, $after: String, $before: String, $first: Int, $last: Int) {\n project(fullPath: $fullPath) {\n id\n siteProfiles: dastSiteProfiles(\n after: $after\n before: $before\n first: $first\n last: $last\n ) {\n pageInfo {\n ...PageInfo\n __typename\n }\n edges {\n cursor\n node {\n id\n profileName\n normalizedTargetUrl\n targetUrl\n targetType\n editPath\n validationStatus\n referencedInSecurityPolicies\n auth {\n enabled\n url\n usernameField\n passwordField\n username\n __typename\n }\n excludedUrls\n requestHeaders\n __typename\n }\n __typename\n }\n __typename\n }\n __typename\n }\n}\n\nfragment PageInfo on PageInfo {\n hasNextPage\n hasPreviousPage\n startCursor\n endCursor\n __typename\n}\n" }, { "operationName":"DastScannerProfiles", "variables":"[FILTERED]", "query":"query DastScannerProfiles($fullPath: ID!, $after: String, $before: String, $first: Int, $last: Int) {\n project(fullPath: $fullPath) {\n id\n scannerProfiles: dastScannerProfiles(\n after: $after\n before: $before\n first: $first\n last: $last\n ) {\n pageInfo {\n ...PageInfo\n __typename\n }\n edges {\n cursor\n node {\n id\n profileName\n spiderTimeout\n targetTimeout\n scanType\n useAjaxSpider\n showDebugMessages\n editPath\n referencedInSecurityPolicies\n __typename\n }\n __typename\n }\n __typename\n }\n __typename\n }\n}\n\nfragment PageInfo on PageInfo {\n hasNextPage\n hasPreviousPage\n startCursor\n endCursor\n __typename\n}\n" }, { "operationName":"DastFailedSiteValidations", "variables":"[FILTERED]", "query":"query DastFailedSiteValidations($fullPath: ID!) {\n project(fullPath: $fullPath) {\n id\n validations: dastSiteValidations(status: FAILED_VALIDATION) {\n nodes {\n id\n normalizedTargetUrl\n __typename\n }\n __typename\n }\n __typename\n }\n}\n" } ] } } ], "meta.user":"anton", "meta.caller_id":"GraphqlController#execute", "meta.remote_ip":"x.x.x.x", "meta.feature_category":"dynamic_application_security_testing", "meta.client_id":"user/2", "correlation_id":"01FSX93AAN5M05MD7WMM03K8F1", "graphql":[ { "depth":7, "complexity":47, "used_fields":[ "Project.id", "PageInfo.__typename", "PageInfo.hasNextPage", "PageInfo.hasPreviousPage", "PageInfo.startCursor", "PageInfo.endCursor", "DastProfileConnection.pageInfo", "DastProfile.id", "DastProfile.name", "DastSiteProfile.id", "DastSiteProfile.targetUrl", "DastSiteProfile.__typename", "DastProfile.dastSiteProfile", "DastScannerProfile.id", "DastScannerProfile.scanType", "DastScannerProfile.__typename", "DastProfile.dastScannerProfile", "DastProfileSchedule.id", "DastProfileSchedule.active", "DastProfileSchedule.startsAt", "DastProfileSchedule.timezone", "DastProfileCadence.unit", "DastProfileCadence.duration", "DastProfileCadence.__typename", "DastProfileSchedule.cadence", "DastProfileSchedule.__typename", "DastProfile.dastProfileSchedule", "DastProfileBranch.name", "DastProfileBranch.exists", "DastProfileBranch.__typename", "DastProfile.branch", "DastProfile.editPath", "DastProfile.__typename", "DastProfileEdge.node", "DastProfileEdge.__typename", "DastProfileConnection.edges", "DastProfileConnection.__typename", "Project.dastProfiles", "Project.__typename", "Query.project" ], "used_deprecated_fields":[ ], "variables":"{\"fullPath\"=>\"anton/263110-dast-site-profiles\", \"first\"=>10}", "operation_name":"DastProfiles" }, { "depth":6, "complexity":37, "used_fields":[ "Project.id", "PageInfo.__typename", "PageInfo.hasNextPage", "PageInfo.hasPreviousPage", "PageInfo.startCursor", "PageInfo.endCursor", "DastSiteProfileConnection.pageInfo", "DastSiteProfileEdge.cursor", "DastSiteProfile.id", "DastSiteProfile.profileName", "DastSiteProfile.normalizedTargetUrl", "DastSiteProfile.targetUrl", "DastSiteProfile.targetType", "DastSiteProfile.editPath", "DastSiteProfile.validationStatus", "DastSiteProfile.referencedInSecurityPolicies", "DastSiteProfileAuth.enabled", "DastSiteProfileAuth.url", "DastSiteProfileAuth.usernameField", "DastSiteProfileAuth.passwordField", "DastSiteProfileAuth.username", "DastSiteProfileAuth.__typename", "DastSiteProfile.auth", "DastSiteProfile.excludedUrls", "DastSiteProfile.requestHeaders", "DastSiteProfile.__typename", "DastSiteProfileEdge.node", "DastSiteProfileEdge.__typename", "DastSiteProfileConnection.edges", "DastSiteProfileConnection.__typename", "Project.dastSiteProfiles", "Project.__typename", "Query.project" ], "used_deprecated_fields":[ ], "variables":"{\"fullPath\"=>\"anton/263110-dast-site-profiles\", \"first\"=>10}", "operation_name":"DastSiteProfiles" }, { "depth":5, "complexity":26, "used_fields":[ "Project.id", "PageInfo.__typename", "PageInfo.hasNextPage", "PageInfo.hasPreviousPage", "PageInfo.startCursor", "PageInfo.endCursor", "DastScannerProfileConnection.pageInfo", "DastScannerProfileEdge.cursor", "DastScannerProfile.id", "DastScannerProfile.profileName", "DastScannerProfile.spiderTimeout", "DastScannerProfile.targetTimeout", "DastScannerProfile.scanType", "DastScannerProfile.useAjaxSpider", "DastScannerProfile.showDebugMessages", "DastScannerProfile.editPath", "DastScannerProfile.referencedInSecurityPolicies", "DastScannerProfile.__typename", "DastScannerProfileEdge.node", "DastScannerProfileEdge.__typename", "DastScannerProfileConnection.edges", "DastScannerProfileConnection.__typename", "Project.dastScannerProfiles", "Project.__typename", "Query.project" ], "used_deprecated_fields":[ ], "variables":"{\"fullPath\"=>\"anton/263110-dast-site-profiles\", \"first\"=>10}", "operation_name":"DastScannerProfiles" }, { "depth":4, "complexity":15, "used_fields":[ "Project.id", "DastSiteValidation.id", "DastSiteValidation.normalizedTargetUrl", "DastSiteValidation.__typename", "DastSiteValidationConnection.nodes", "DastSiteValidationConnection.__typename", "Project.dastSiteValidations", "Project.__typename", "Query.project" ], "used_deprecated_fields":[ ], "variables":"{\"fullPath\"=>\"anton/263110-dast-site-profiles\"}", "operation_name":"DastFailedSiteValidations" } ], "remote_ip":"x.x.x.x", "user_id":2, "username":"anton", "ua":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/97.0.4692.71 Safari/537.36", "queue_duration_s":0.012021, "request_urgency":"default", "target_duration_s":1, "redis_calls":2, "redis_duration_s":0.000619, "redis_read_bytes":183, "redis_write_bytes":1220, "redis_shared_state_calls":1, "redis_shared_state_duration_s":0.000239, "redis_shared_state_write_bytes":53, "redis_sessions_calls":1, "redis_sessions_duration_s":0.00038, "redis_sessions_read_bytes":183, "redis_sessions_write_bytes":1167, "db_count":10, "db_write_count":0, "db_cached_count":0, "db_replica_count":0, "db_primary_count":10, "db_replica_cached_count":0, "db_primary_cached_count":0, "db_replica_wal_count":0, "db_primary_wal_count":0, "db_replica_wal_cached_count":0, "db_primary_wal_cached_count":0, "db_replica_duration_s":0.0, "db_primary_duration_s":0.008, "cpu_s":0.087416, "mem_objects":34380, "mem_bytes":2093336, "mem_mallocs":8664, "mem_total_bytes":3468536, "pid":28242, "exception.class":"URI::InvalidURIError", "exception.message":"bad URI(is not URI?): \"https://site1.com/, https://site2.com/\"", "exception.backtrace":[ "ee/app/models/dast_site_validation.rb:72:in `get_normalized_url_base'", "ee/app/graphql/types/dast_site_profile_type.rb:66:in `normalized_target_url'", "lib/gitlab/graphql/present/field_extension.rb:18:in `resolve'", "lib/gitlab/graphql/tracers/timer_tracer.rb:20:in `trace'", "lib/gitlab/graphql/tracers/logger_tracer.rb:14:in `trace'", "lib/gitlab/graphql/tracers/metrics_tracer.rb:13:in `trace'", "lib/gitlab/graphql/tracers/application_context_tracer.rb:23:in `trace'", "lib/gitlab/graphql/tracers/timer_tracer.rb:20:in `trace'", "lib/gitlab/graphql/generic_tracing.rb:40:in `with_labkit_tracing'", "lib/gitlab/graphql/generic_tracing.rb:30:in `platform_trace'", "lib/gitlab/graphql/tracers/logger_tracer.rb:14:in `trace'", "lib/gitlab/graphql/tracers/metrics_tracer.rb:13:in `trace'", "lib/gitlab/graphql/tracers/application_context_tracer.rb:23:in `trace'", "lib/gitlab/graphql/tracers/timer_tracer.rb:20:in `trace'", "lib/gitlab/graphql/generic_tracing.rb:40:in `with_labkit_tracing'", "lib/gitlab/graphql/generic_tracing.rb:30:in `platform_trace'", "lib/gitlab/graphql/tracers/logger_tracer.rb:14:in `trace'", "lib/gitlab/graphql/tracers/metrics_tracer.rb:13:in `trace'", "lib/gitlab/graphql/tracers/application_context_tracer.rb:23:in `trace'", "app/graphql/gitlab_schema.rb:49:in `multiplex'", "app/controllers/graphql_controller.rb:134:in `execute_multiplex'", "app/controllers/graphql_controller.rb:48:in `execute'", "ee/lib/gitlab/ip_address_state.rb:10:in `with'", "ee/app/controllers/ee/application_controller.rb:44:in `set_current_ip_address'", "app/controllers/application_controller.rb:490:in `set_current_admin'", "lib/gitlab/session.rb:11:in `with_session'", "app/controllers/application_controller.rb:481:in `set_session_storage'", "lib/gitlab/i18n.rb:105:in `with_locale'", "lib/gitlab/i18n.rb:111:in `with_user_locale'", "app/controllers/application_controller.rb:475:in `set_locale'", "app/controllers/application_controller.rb:469:in `set_current_context'", "ee/lib/omni_auth/strategies/group_saml.rb:41:in `other_phase'", "lib/gitlab/metrics/elasticsearch_rack_middleware.rb:16:in `call'", "lib/gitlab/middleware/rails_queue_duration.rb:33:in `call'", "lib/gitlab/middleware/speedscope.rb:13:in `call'", "lib/gitlab/request_profiler/middleware.rb:17:in `call'", "lib/gitlab/database/load_balancing/rack_middleware.rb:23:in `call'", "lib/gitlab/metrics/rack_middleware.rb:16:in `block in call'", "lib/gitlab/metrics/web_transaction.rb:46:in `run'", "lib/gitlab/metrics/rack_middleware.rb:16:in `call'", "lib/gitlab/jira/middleware.rb:19:in `call'", "lib/gitlab/middleware/go.rb:20:in `call'", "lib/gitlab/etag_caching/middleware.rb:21:in `call'", "lib/gitlab/middleware/multipart.rb:173:in `call'", "lib/gitlab/middleware/read_only/controller.rb:50:in `call'", "lib/gitlab/middleware/read_only.rb:18:in `call'", "lib/gitlab/middleware/same_site_cookies.rb:27:in `call'", "lib/gitlab/middleware/handle_malformed_strings.rb:21:in `call'", "lib/gitlab/middleware/basic_health_check.rb:25:in `call'", "lib/gitlab/middleware/handle_ip_spoof_attack_error.rb:25:in `call'", "lib/gitlab/middleware/request_context.rb:21:in `call'", "config/initializers/fix_local_cache_middleware.rb:11:in `call'", "lib/gitlab/middleware/compressed_json.rb:26:in `call'", "lib/gitlab/middleware/rack_multipart_tempfile_factory.rb:19:in `call'", "lib/gitlab/middleware/sidekiq_web_static.rb:20:in `call'", "lib/gitlab/metrics/requests_rack_middleware.rb:75:in `call'", "lib/gitlab/middleware/release_env.rb:13:in `call'" ], "db_duration_s":0.0, "view_duration_s":0.00031, "duration_s":0.07783 }
Output of checks
Results of GitLab environment info
Expand for output related to GitLab environment info
(For installations with omnibus-gitlab package run and paste the output of: `sudo gitlab-rake gitlab:env:info`) (For installations from source run and paste the output of: `sudo -u git -H bundle exec rake gitlab:env:info RAILS_ENV=production`)
Results of GitLab application Check
Expand for output related to the GitLab application check
(For installations with omnibus-gitlab package run and paste the output of:
sudo gitlab-rake gitlab:check SANITIZE=true
)(For installations from source run and paste the output of:
sudo -u git -H bundle exec rake gitlab:check RAILS_ENV=production SANITIZE=true
)(we will only investigate if the tests are passing)
Possible fixes
Workarounds
-
If the
DastSiteProfile
ID is known, you can update the Target URL in the UI:https://gitlab.example.com/<group>/<project>/-/security/configuration/dast_scans/dast_site_profiles/<profile_id>/edit
-
Run the following code via
sudo gitlab-rails c
:# Get the project project = Project.find(x) # Get a list of site profiles and identify the ID of the bad profile site_profiles = project.dast_site_profiles # Load the profile profile = site_profiles.find(x) # Or just load the last profile created as that is the most likely one to be broken profile = site_profiles.last # Change the URL to a valid one profile.dast_site.url = '<VALID URL>' # Validate the URL before we save profile.dast_site.validate! # Save the profile profile.dast_site.save