Allow changing the port(s) used by API Security
Problem
When running API Security, it is a common configuration to add a service
definition to the job with the target app, database, etc.
Sometimes the port numbers conflict with those we use by default, such as port 5000.
This issue has impacted two customers at the time of writing. https://gitlab.zendesk.com/agent/tickets/304578
Proposal
- Default to special port 0, but allow override with new variable
- Special Port: "0" (means find an available port and use it)
- Engine writes out URL to file on disk
- worker entry Read the
_API
variable from a file.
- worker entry Read the
Example Place to Put Write File
public class Startup
{
public void Configure(IApplicationBuilder app, ILogger<Startup> log)
{
// IApplicationBuilder exposes an IFeatureCollection property, ServerFeatures
var addressFeature = app.ServerFeatures.Get<IServerAddressesFeature>();
foreach(var address in addressFeature.Addresses)
{
_log.LogInformation("Listing on address: " + address);
}
}
// ... other configuration
}
Allow the user to change the port using a new variable.
1. [ ] Pick an unused port for default (5001)(https://docs.gitlab.com/ee/administration/package_information/defaults.html#ports)
2. [ ] Add a new variable to allow the port to be configured by the user at runtime.
3. [ ] ...
Implementation Proposal
-
Set up API port file to let worker-entry know current port in USE -
Add new variable _API_PORT_FILE
(e.g.FUZZAPI_API_PORT_FILE
,DAST_API_API_PORT_FILE
) as non configurable variable. Default togl-api-security-port.log
-
web\SDK\entrypoints\analyzer-dast-api
-
web\SDK\entrypoints\analyzer-fuzz-api
-
-
Update scanner
to write the file based on_API_PORT_FILE
file path. See how to automatically choose a free port in asp net core. If possible, limit file content only to the port number. -
Update worker-entry
to consume file before callingwait_for_api_fuzzer
(which internally calls toglapifuzzing.set_api(self.api)
)-
Load new configuration variable _API_PORT_FILE
(new worker-entry property, print out information in logs) -
After printing configuration handle the file pointed by _API_PORT_FILE
-
Wait for the file to be created. Exponential waits up to N retries. Log process status, until wait finished (found file o give up) -
On found file: -
read port from a file, then update api
property of theworker-entry
-
-
On give up: -
report failure, suggest actionable items
-
-
-
-
Update worker-entry
tests-
Set _PORT_FILE
to/output/
-
build/jobs/tests_int_openapi
-
build/jobs/tests_int_postman
-
build/jobs/tests_int_worker-entry_py36
-
build/jobs/tests_int_worker-entry_py39
-
- ~Expose
_PORT_FILE
to test the environment ~ NOTE: Better to add when needed[ ]web/Test/docker-compose.gitlab.dast.yml
[ ]web/Test/docker-compose.gitlab.har.yml
[ ]web/Test/docker-compose.gitlab.openapi.yml
[ ]web/Test/docker-compose.gitlab.postman.yml
[ ]web/Test/docker-compose.python.py36.yml
[ ]web/Test/docker-compose.python.py39.yml
[ ]web/Test/docker-compose.worker-entry_py36.yml
[ ]web/Test/docker-compose.worker-entry_py39.yml
[ ]web/SDK/worker-entry/tox.ini
-
Update web/SDK/entrypoints/tests/01-fuzz-general-test.sh
/web/SDK/entrypoints/tests/02-dast-general-test.sh
to test new variable_API_PORT
-
Update WorkerEntryTest
class to allow specifying_API_PORT_FILE
parameter. -
Add test for new backoff functionality
-
-
-
Add a new variable to specify _API_PORT
as a non-configurable variable. Default to0
- Set
ASPNETCORE_URLS
in the entry point-
Move declaration from web/PeachWeb/Dockerfile
,web/PeachWeb/Dockerfile-fips
to entry points scripts
-
-
Update entry points ( web\SDK\entrypoints\analyzer-dast-api
,web\SDK\entrypoints\analyzer-fuzz-api
)-
Add new variable: _API_PORT
(e.g.FUZZAPI_API_PORT
,DAST_API_API_PORT
) default to0
-
Set _API
based on_API_PORT
-
Set ASPNETCORE_URLS
based on_API_PORT
-
- Worker-Entry
-
Load new configuration variable _API_PORT
(new worker-entry property, print out information in logs) -
Fail to start if _API_PORT
lower or equals 1024
-
- Update
worker-entry
tests-
ExposeNOTE: Better to add when needed_PORT_FILE
to test the environment[ ]web/Test/docker-compose.gitlab.dast.yml
[ ]web/Test/docker-compose.gitlab.har.yml
[ ]web/Test/docker-compose.gitlab.openapi.yml
[ ]web/Test/docker-compose.gitlab.postman.yml
[ ]web/Test/docker-compose.python.py36.yml
[ ]web/Test/docker-compose.python.py39.yml
[ ]web/Test/docker-compose.worker-entry_py36.yml
[ ]web/Test/docker-compose.worker-entry_py39.yml
[ ]web/SDK/worker-entry/tox.ini
-
-
Update web/SDK/entrypoints/tests/01-fuzz-general-test.sh
/web/SDK/entrypoints/tests/02-dast-general-test.sh
to test new variable_API_PORT
-
UpdateInstead allowing specifyingWorkerEntryTest
class to allow specifying_API_PORT
parameter_API_PORT_FILE
content to point o a different port. -
Add test to check _API_PORT
fails when set lower or equals to 1024
- Set
-
Document new variable -
Add _API_PORT
to the list of configuration variables -
Add a troubleshoot entry section about when to use this new variable.
-