Draft: Add framework for detecting abusive requests

What does this MR do and why?

Now every request will detect_abuse. The results are stored in an abuse object. You can check if a request is abusive by calling abuse.detected?.

If an AbuseDetector has confidence greater than 0.5 the confidence and messages will be added to logging payload under json.abuse. This will help us prevent abuse by detecting it at its origin which is the request level.

By logging abusive requests, we can respond quicker. We can also start building asynchronous actions to respond to abuse requests that show up in the logs. For example, we can move any abuse request to its own "abuse index" in Elasticsearch for longer retention periods (instead of just a couple of days). Also, we can use Cloudflare's API to challenge an IP address with a captcha or temporary ban if that IP address shows up multiple times in our abuse index.

Potential use cases:

• Log abusive searches
• Log users putting XSS attempts in their profile / README
• Log crypto-mining libraries being added to pipelines.

If no AbuseDetector is defined for a controller action, abuse detection will default to the ApplicationAbuseDetector.

To add your own custom abuse detection logic for a request, add an AbuseDetector in the app/abuse/abuse_detectors directory. If your controller is called ExampleController, you should create ExampleAbuseDetector. That would look something like this:

In app/abuse/abuse_detectors/example_abuse_detector.rb

module AbuseDetectors
  class ExampleAbuseDetector < ApplicationAbuseDetector
    # Detection method should match the action in the controller
    def show
      @confidence = 1 # a number between 0..1
      @messages = ['reasons why we think request is abusive']
    end
  end
end

You can also define custom detections and for multiple routes:

module AbuseDetectors
  class ExampleAbuseDetector < ApplicationAbuseDetector
    detect_abuse :create, :update, with: :detect_abusive_payloads

    def detect_abusive_payloads
      # logic here
      @confidence = 1
      @messages = []
    end

    # Detection method should match the action in the controller
    def show
      @confidence = 1 # a number between 0..1
      @messages = ['reasons why we think request is abusive']
    end
  end
end

If you have the same detection method for all of your routes, the detection method defaults to detect. You can do something like this:

module AbuseDetectors
  class ExampleAbuseDetector < ApplicationAbuseDetector
    detect_abuse :create, :update, :index, :show

    # This is the default detection method if not specified in
    # the `detect_abuse` class method.
    def detect
      # logic here
      @confidence = 1
      @messages = []
    end
  end
end

Note: running multiple detection methods is supported, but because @confidence and @messages are simple instance variables, you will need to be careful to not overwrite them.

TODO

• add logging to UI requests
• add logging to API requests

Screenshots or screen recordings

How to set up and validate locally

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

• I have evaluated the MR acceptance checklist for this MR.