Allow changing the port(s) used by API Security

Problem

When running API Security, it is a common configuration to add a service definition to the job with the target app, database, etc. Sometimes the port numbers conflict with those we use by default, such as port 5000.

This issue has impacted two customers at the time of writing. https://gitlab.zendesk.com/agent/tickets/304578

Proposal

Example Place to Put Write File

public class Startup
{
    public void Configure(IApplicationBuilder app, ILogger<Startup> log)
    {
        // IApplicationBuilder exposes an IFeatureCollection property, ServerFeatures
        var addressFeature = app.ServerFeatures.Get<IServerAddressesFeature>();
        foreach(var address in addressFeature.Addresses)
        {
            _log.LogInformation("Listing on address: " + address);
        }
    }

    // ... other configuration
}

Allow the user to change the port using a new variable. 1. [ ] Pick an unused port for default (5001)(https://docs.gitlab.com/ee/administration/package_information/defaults.html#ports) 2. [ ] Add a new variable to allow the port to be configured by the user at runtime. 3. [ ] ...

Implementation Proposal

  1. Set up API port file to let worker-entry know current port in USE

    1. Add new variable _API_PORT_FILE (e.g. FUZZAPI_API_PORT_FILE , DAST_API_API_PORT_FILE) as non configurable variable. Default to gl-api-security-port.log
      1. web\SDK\entrypoints\analyzer-dast-api
      2. web\SDK\entrypoints\analyzer-fuzz-api
    2. Update scanner to write the file based on _API_PORT_FILE file path. See how to automatically choose a free port in asp net core. If possible, limit file content only to the port number.
    3. Update worker-entry to consume file before calling wait_for_api_fuzzer (which internally calls to glapifuzzing.set_api(self.api))
      1. Load new configuration variable _API_PORT_FILE (new worker-entry property, print out information in logs)
      2. After printing configuration handle the file pointed by _API_PORT_FILE
        1. Wait for the file to be created. Exponential waits up to N retries. Log process status, until wait finished (found file o give up)
        2. On found file:
          1. read port from a file, then update api property of the worker-entry
        3. On give up:
          1. report failure, suggest actionable items
    4. Update worker-entry tests
      1. Set _PORT_FILE to /output/
        1. build/jobs/tests_int_openapi
        2. build/jobs/tests_int_postman
        3. build/jobs/tests_int_worker-entry_py36
        4. build/jobs/tests_int_worker-entry_py39
      2. ~Expose _PORT_FILE to test the environment ~ NOTE: Better to add when needed
        1. [ ] web/Test/docker-compose.gitlab.dast.yml
        2. [ ] web/Test/docker-compose.gitlab.har.yml
        3. [ ] web/Test/docker-compose.gitlab.openapi.yml
        4. [ ] web/Test/docker-compose.gitlab.postman.yml
        5. [ ] web/Test/docker-compose.python.py36.yml
        6. [ ] web/Test/docker-compose.python.py39.yml
        7. [ ] web/Test/docker-compose.worker-entry_py36.yml
        8. [ ] web/Test/docker-compose.worker-entry_py39.yml
        9. [ ] web/SDK/worker-entry/tox.ini
      3. Update web/SDK/entrypoints/tests/01-fuzz-general-test.sh / web/SDK/entrypoints/tests/02-dast-general-test.sh to test new variable _API_PORT
      4. Update WorkerEntryTest class to allow specifying _API_PORT_FILE parameter.
      5. Add test for new backoff functionality
  2. Add a new variable to specify _API_PORT as a non-configurable variable. Default to 0

    1. Set ASPNETCORE_URLS in the entry point
      1. Move declaration from web/PeachWeb/Dockerfile, web/PeachWeb/Dockerfile-fips to entry points scripts
    2. Update entry points (web\SDK\entrypoints\analyzer-dast-api, web\SDK\entrypoints\analyzer-fuzz-api)
      1. Add new variable: _API_PORT (e.g. FUZZAPI_API_PORT , DAST_API_API_PORT) default to 0
      2. Set _API based on _API_PORT
      3. Set ASPNETCORE_URLS based on _API_PORT
    3. Worker-Entry
      1. Load new configuration variable _API_PORT (new worker-entry property, print out information in logs)
      2. Fail to start if _API_PORT lower or equals 1024
    4. Update worker-entry tests
      1. Expose _PORT_FILE to test the environment NOTE: Better to add when needed
        1. [ ] web/Test/docker-compose.gitlab.dast.yml
        2. [ ]web/Test/docker-compose.gitlab.har.yml
        3. [ ] web/Test/docker-compose.gitlab.openapi.yml
        4. [ ] web/Test/docker-compose.gitlab.postman.yml
        5. [ ] web/Test/docker-compose.python.py36.yml
        6. [ ] web/Test/docker-compose.python.py39.yml
        7. [ ] web/Test/docker-compose.worker-entry_py36.yml
        8. [ ] web/Test/docker-compose.worker-entry_py39.yml
        9. [ ] web/SDK/worker-entry/tox.ini
    5. Update web/SDK/entrypoints/tests/01-fuzz-general-test.sh / web/SDK/entrypoints/tests/02-dast-general-test.sh to test new variable _API_PORT
    6. Update WorkerEntryTest class to allow specifying _API_PORT parameter Instead allowing specifying _API_PORT_FILE content to point o a different port.
    7. Add test to check _API_PORT fails when set lower or equals to 1024
  3. Document new variable

    1. Add _API_PORT to the list of configuration variables
    2. Add a troubleshoot entry section about when to use this new variable.
Edited by Herber Madrigal