Error waiting for API Security 'http://127.0.0.1:5000' to become available

Summary

While running the apifuzzer_fuzz job it throws the following error: Error waiting for API Security 'http://127.0.0.1:5000' to become available.

Steps to reproduce

  • Create a fuzzing CI pipeline:
apifuzzer_fuzz:
  needs: ["fuzzing:generate-swagger-documentation", "build:web"]

  variables:
    FUZZAPI_TARGET_URL: http://backend:8080
    FUZZAPI_OPENAPI: storage/api-docs/api-docs.json
    FUZZAPI_PROFILE: Medium-20
    MYSQL_ROOT_PASSWORD: mysql-password
    MYSQL_DATABASE: mysql-database
    INFLUXDB_DB: influx-database

  services:
    - name: $WEB_IMAGE_NAME:$CI_COMMIT_SHA
      entrypoint: ["/bin/sh", "-c"]
      command:
        [
          "cp .env.ci .env && php artisan key:generate && php artisan jwt:secret && apache2-foreground",
        ]
      alias: backend
    - name: mysql:5.7
      alias: mysql
    - name: redislabs/rebloom
      alias: redis
    - name: influxdb:1.8
      alias: influxdb
  • Run it
  • See error

Example Project

What is the current bug behavior?

What seems to be happening is that a background process in the fuzz job is attempting to set up a web server on port 80. However, since there's a service that is Laravel image, which also listens on port 80 (the base Apache Docker image does so). That is only a problem because this is using the Kubernetes executor, which means that the services (which run as containers on the same pod the job spins up) are available at localhost, which means the two web servers conflict and the fuzz job cannot run successfully. The service cannot be removed because that is what the fuzz job is testing.

[0m32;1m$ /peach/analyzer-fuzz-api[0m0;m
analyzer-fuzz-api
20:47:24 [INF] API Security: Gitlab API Security Worker Entry
20:47:24 [INF] API Security: --------------------------------
20:47:24 [INF] API Security:
20:47:24 [INF] API Security: version: 1.6.216
20:47:24 [INF] API Security: api: http://127.0.0.1:5000
20:47:24 [INF] API Security: config: /peach/configs/gitlab-api-fuzzing-config.yml
20:47:24 [INF] API Security: openapi: storage/api-docs/api-docs.json
20:47:24 [INF] API Security: profile: Medium-20
20:47:24 [INF] API Security: project: PROJECT
20:47:24 [INF] API Security: security report: gl-api-fuzzing-report.json
20:47:24 [INF] API Security: security report asset path: gl-assets
20:47:24 [INF] API Security: ci_project_url: URL
20:47:24 [INF] API Security: ci_job_id: 2194500159
20:47:24 [INF] API Security: service_start_timeout: 300
20:47:24 [INF] API Security: target_url: http://backend:8080
20:47:24 [INF] API Security: timeout: 30
20:47:24 [INF] API Security: verbose: False
20:47:24 [INF] API Security:
20:47:24 [INF] API Security: Waiting for API Security (http://127.0.0.1:5000) to become available...
20:47:24 [INF] API Security: Backing off 0.5 seconds afters 1 tries
20:47:24 [INF] API Security: Backing off 0.8 seconds afters 2 tries
20:47:25 [INF] API Security: Backing off 3.3 seconds afters 3 tries
20:47:28 [INF] API Security: Backing off 6.0 seconds afters 4 tries
20:47:34 [INF] API Security: Backing off 2.0 seconds afters 5 tries
20:47:36 [INF] API Security: Backing off 18.7 seconds afters 6 tries
20:47:55 [INF] API Security: Backing off 58.1 seconds afters 7 tries
20:48:53 [INF] API Security: Backing off 16.2 seconds afters 8 tries
20:49:10 [INF] API Security: Backing off 87.9 seconds afters 9 tries
20:50:38 [INF] API Security: Backing off 106.3 seconds afters 10 tries
20:52:24 [WAR] API Security: Waiting for url 'http://127.0.0.1:5000', failed: catching classes that do not inherit from BaseException is not allowed
20:52:24 [ERR] API Security: Error waiting for API Security 'http://127.0.0.1:5000' to become available.
/peach/analyzer-fuzz-api: line 92:    15 Aborted                 (core dumped) dotnet /peach/Peach.Web.dll &> $FUZZAPI_LOG_SCANNER
Stopping scanner...
/peach/analyzer-fuzz-api: line 49: kill: (15) - No such process
/peach/analyzer-fuzz-api: line 51: kill: (15) - No such process
Waiting for scanner to terminate

What is the expected correct behavior?

fuzzing job passes.

Relevant logs and/or screenshots

20:47:25.486 [DBG] <Peach.Web.Data.Services.DatabaseMigrator> Applied migration 31
Unhandled exception. System.IO.IOException: Failed to bind to address http://[::]:80: address already in use.
 ---> Microsoft.AspNetCore.Connections.AddressInUseException: Address in use
 ---> System.Net.Sockets.SocketException (98): Address in use
   at System.Net.Sockets.Socket.UpdateStatusAfterSocketErrorAndThrowException(SocketError error, String callerName)
   at System.Net.Sockets.Socket.DoBind(EndPoint endPointSnapshot, SocketAddress socketAddress)
   at System.Net.Sockets.Socket.Bind(EndPoint localEP)
   at Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets.SocketConnectionListener.Bind()
   --- End of inner exception stack trace ---
   at Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets.SocketConnectionListener.Bind()
   at Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets.SocketTransportFactory.BindAsync(EndPoint endpoint, CancellationToken cancellationToken)
   at Microsoft.AspNetCore.Server.Kestrel.Core.KestrelServer.<>c__DisplayClass21_0`1.<<StartAsync>g__OnBind|0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.AddressBinder.BindEndpointAsync(ListenOptions endpoint, AddressBindContext context)
   --- End of inner exception stack trace ---
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.AddressBinder.BindEndpointAsync(ListenOptions endpoint, AddressBindContext context)
   at Microsoft.AspNetCore.Server.Kestrel.Core.ListenOptions.BindAsync(AddressBindContext context)
   at Microsoft.AspNetCore.Server.Kestrel.Core.AnyIPListenOptions.BindAsync(AddressBindContext context)
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.AddressBinder.AddressesStrategy.BindAsync(AddressBindContext context)
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.AddressBinder.BindAsync(IServerAddressesFeature addresses, KestrelServerOptions serverOptions, ILogger logger, Func`2 createBinding)
   at Microsoft.AspNetCore.Server.Kestrel.Core.KestrelServer.StartAsync[TContext](IHttpApplication`1 application, CancellationToken cancellationToken)
   at Microsoft.AspNetCore.Hosting.GenericWebHostService.StartAsync(CancellationToken cancellationToken)
   at Microsoft.Extensions.Hosting.Internal.Host.StartAsync(CancellationToken cancellationToken)
   at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.RunAsync(IHost host, CancellationToken token)
   at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.RunAsync(IHost host, CancellationToken token)
   at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.Run(IHost host)
   at Peach.Web.Program.Main(String[] args) in /builds/gitlab-org/security-products/analyzers/api-fuzzing-src/web/PeachWeb/Program.cs:line 49

Output of checks

This bug happens on GitLab.com

Results of GitLab environment info

Expand for output related to GitLab environment info 

(For installations with omnibus-gitlab package run and paste the output of:
`sudo gitlab-rake gitlab:env:info`)

(For installations from source run and paste the output of:
`sudo -u git -H bundle exec rake gitlab:env:info RAILS_ENV=production`)

Results of GitLab application Check

Expand for output related to the GitLab application check 
(For installations with omnibus-gitlab package run and paste the output of:
sudo gitlab-rake gitlab:check SANITIZE=true)


(For installations from source run and paste the output of:
sudo -u git -H bundle exec rake gitlab:check RAILS_ENV=production SANITIZE=true)


(we will only investigate if the tests are passing)

Possible fixes

The ideal would be for the fuzz job to fallback to another port if port 80 is in use. Another option would be to have an environment variable that can set the port for the fuzz background process.

Edited by Filip Aleksic