Add export mechanism for Gitlab data for use in external systems (like SIEMs)

Everyone can contribute. Help move this issue forward while earning points, leveling up and collecting rewards.

Proposal

Implement a mechanism to pull data on a reoccurring basis into a flat file that can be consumed by external systems & tools.

Background:

Many IT staff may not have familiarity with Gitlab and coding/scripting/API skills required to reliably export data from Gitlab. Having the information exported in a manner that allows consumption by an external system (such as a SIEM system or data analytics tool [Splunk/DataDog/ELK/etc.]) will allow data to be analyzed and integrated into external workflows, alerting mechanisms, and analytics processes. By building a mechanism that exports all fields from all datatypes, consistent and supportable external processes can be achieved by users that may not have the technical background/staff/time required to write scripts that leverage Gitlab's APIs.

What is the competitive advantage or differentiation for this feature?

By making a Gitlab supported export mechanism available, Gitlab customers can quickly integrate with other tools in their enterprise environment. This value-add capability makes Gitlab an attractive proposition for quick return on investment and provides a method of gaining insights into Gitlab data in tools the customer is familiar with and has already implemented. It enhances rapid adoption of the platform and provides enterprise customers with a vendor supported method of interfacing their version control and code-storage system to external tools.

Intended Users:

  • Non-Gitlab IT staff, Operations staff, Cybersecurity, Data Analysts

Data Types of particular interest:

  • Projects
  • Groups
  • Languages
  • Vulnerabilities
  • Commits
  • Integrations
  • Pipeline Schedules

Suggested Mechanisms:

  • Docker container

    • Can be passed Gitlab token information and desired data type[project|group|vulnerability], output format [json|csv|tsv], starting location [/org|/org/group|/org/group/project], flat file output destination [/tmp|/home/<user>/<dest_dir/].
    • All available fields for the requested data type should be exported to the flat file, one-line-per-result, CR LF line terminator.
    • Provides an export mechanism that can be consumed by any external tool.
    • Secondary objective: Supporting cloud storage solutions (AWS/GCP/OCI/Azure Storage Bucket).

  • Splunk Application

    • Similar to the docker mechanism, Splunk apps are a collection of shell scripts that execute on a cron job and corresponding dashboards with prefabricated queries that return statistics about the data.
    • Splunk apps offer the ability to configure the settings for the 'app' inside the Splunk GUI using a web page.
    • Tokens and other secret data are stored by Splunk in an encrypted format.
    • 'Community supported' versions exist and may be used as a template: https://apps.splunk.com/app/6848/

Permissions/Security

  • Access would be dictated by the token passed to the mechanism.
Edited by 🤖 GitLab Bot 🤖