Add Huggingface API
What does this merge request do and why?
This MR adds the necessary abstraction to call Huggingface models via the API
How to set up and validate locally
Running the pipeline with the desired HF model specified.
Merge request checklist
-
I've ran the eval_codebase.py
pipeline to validate that nothing is broken. https://console.cloud.google.com/bigquery?project=unreview-poc-390200e5&ws=!1m5!1m4!4m3!1sunreview-poc-390200e5!2sgl_gitlab_experiments!3stest_huggingface_model -
Tests added for new functionality. If not, please raise an issue to follow up. -
Documentation added/updated, if needed.
Edited by Hongtao Yang