Allow changing the port(s) used by API Security
Problem
When running API Security, it is a common configuration to add a service definition to the job with the target app, database, etc.
Sometimes the port numbers conflict with those we use by default, such as port 5000.
This issue has impacted two customers at the time of writing. https://gitlab.zendesk.com/agent/tickets/304578
Proposal
- Default to special port 0, but allow override with new variable
- Special Port: "0" (means find an available port and use it)
- Engine writes out URL to file on disk
- worker entry Read the
_APIvariable from a file.
- worker entry Read the
Example Place to Put Write File
public class Startup
{
public void Configure(IApplicationBuilder app, ILogger<Startup> log)
{
// IApplicationBuilder exposes an IFeatureCollection property, ServerFeatures
var addressFeature = app.ServerFeatures.Get<IServerAddressesFeature>();
foreach(var address in addressFeature.Addresses)
{
_log.LogInformation("Listing on address: " + address);
}
}
// ... other configuration
}
Allow the user to change the port using a new variable.
1. [ ] Pick an unused port for default (5001)(https://docs.gitlab.com/ee/administration/package_information/defaults.html#ports)
2. [ ] Add a new variable to allow the port to be configured by the user at runtime.
3. [ ] ...
Implementation Proposal
-
Set up API port file to let worker-entry know current port in USE -
Add new variable _API_PORT_FILE(e.g.FUZZAPI_API_PORT_FILE,DAST_API_API_PORT_FILE) as non configurable variable. Default togl-api-security-port.log-
web\SDK\entrypoints\analyzer-dast-api -
web\SDK\entrypoints\analyzer-fuzz-api
-
-
Update scannerto write the file based on_API_PORT_FILEfile path. See how to automatically choose a free port in asp net core. If possible, limit file content only to the port number. -
Update worker-entryto consume file before callingwait_for_api_fuzzer(which internally calls toglapifuzzing.set_api(self.api))-
Load new configuration variable _API_PORT_FILE(new worker-entry property, print out information in logs) -
After printing configuration handle the file pointed by _API_PORT_FILE-
Wait for the file to be created. Exponential waits up to N retries. Log process status, until wait finished (found file o give up) -
On found file: -
read port from a file, then update apiproperty of theworker-entry
-
-
On give up: -
report failure, suggest actionable items
-
-
-
-
Update worker-entrytests-
Set _PORT_FILEto/output/-
build/jobs/tests_int_openapi -
build/jobs/tests_int_postman -
build/jobs/tests_int_worker-entry_py36 -
build/jobs/tests_int_worker-entry_py39
-
- ~Expose
_PORT_FILEto test the environment ~ NOTE: Better to add when needed[ ]web/Test/docker-compose.gitlab.dast.yml[ ]web/Test/docker-compose.gitlab.har.yml[ ]web/Test/docker-compose.gitlab.openapi.yml[ ]web/Test/docker-compose.gitlab.postman.yml[ ]web/Test/docker-compose.python.py36.yml[ ]web/Test/docker-compose.python.py39.yml[ ]web/Test/docker-compose.worker-entry_py36.yml[ ]web/Test/docker-compose.worker-entry_py39.yml[ ]web/SDK/worker-entry/tox.ini
-
Update web/SDK/entrypoints/tests/01-fuzz-general-test.sh/web/SDK/entrypoints/tests/02-dast-general-test.shto test new variable_API_PORT -
Update WorkerEntryTestclass to allow specifying_API_PORT_FILEparameter. -
Add test for new backoff functionality
-
-
-
Add a new variable to specify _API_PORTas a non-configurable variable. Default to0- Set
ASPNETCORE_URLSin the entry point-
Move declaration from web/PeachWeb/Dockerfile,web/PeachWeb/Dockerfile-fipsto entry points scripts
-
-
Update entry points ( web\SDK\entrypoints\analyzer-dast-api,web\SDK\entrypoints\analyzer-fuzz-api)-
Add new variable: _API_PORT(e.g.FUZZAPI_API_PORT,DAST_API_API_PORT) default to0 -
Set _APIbased on_API_PORT -
Set ASPNETCORE_URLSbased on_API_PORT
-
- Worker-Entry
-
Load new configuration variable _API_PORT(new worker-entry property, print out information in logs) -
Fail to start if _API_PORTlower or equals 1024
-
- Update
worker-entrytests-
ExposeNOTE: Better to add when needed_PORT_FILEto test the environment[ ]web/Test/docker-compose.gitlab.dast.yml[ ]web/Test/docker-compose.gitlab.har.yml[ ]web/Test/docker-compose.gitlab.openapi.yml[ ]web/Test/docker-compose.gitlab.postman.yml[ ]web/Test/docker-compose.python.py36.yml[ ]web/Test/docker-compose.python.py39.yml[ ]web/Test/docker-compose.worker-entry_py36.yml[ ]web/Test/docker-compose.worker-entry_py39.yml[ ]web/SDK/worker-entry/tox.ini
-
-
Update web/SDK/entrypoints/tests/01-fuzz-general-test.sh/web/SDK/entrypoints/tests/02-dast-general-test.shto test new variable_API_PORT -
UpdateInstead allowing specifyingWorkerEntryTestclass to allow specifying_API_PORTparameter_API_PORT_FILEcontent to point o a different port. -
Add test to check _API_PORTfails when set lower or equals to 1024
- Set
-
Document new variable -
Add _API_PORTto the list of configuration variables -
Add a troubleshoot entry section about when to use this new variable.
-