Skip to content

Add scraping scripts

Andras Herczeg requested to merge add-scraping-scripts into main

What does this merge request do and why?

This MR adds two Golang scripts that were used to:

  1. Enrich the documentation_v1 dataset with GitLab docs
  2. Scrape questions from Stack Overflow to to build stackoverflow_v1

How to set up and validate locally

Numbered steps to set up and validate the change are strongly suggested.

Merge request checklist

  • I've ran the affected pipeline(s) to validate that nothing is broken.
  • Tests added for new functionality. If not, please raise an issue to follow up.
  • Documentation added/updated, if needed.

Merge request reports