removed old tutorial

# Create a Hog project starting from a local project

This tutorial will show how to create a hog project starting from an example Vivado project (IBERT). For this tutorial we will use Hog version `Hog2021.2-9`

## Step 1: create an example Vivado project

Let's start our tutorial by switching to a new empty directory:

```bash

mkdir Example_project

cd Example_project

```

Now we open Vivado console and we create an empty RTL project, we leave everything as default and we select `Kintex-7 KC705 Evaluation Platform` in the `Default Part` panel.

Now we go to the IP catalogue and we type `IBERT` in the search bar and we click on `IBERT 7 Series GTX`.

![](./figures/IBERT.png)

We leave the IP configuration as default, and we click on `IP location`. We change the IP directory to be `IP` (we create the new directory on the GUI). 

![](./figures/ip_location.png)

Now we click on `Done` and we skip the "Generate Output Product".

Now we go on `IP Sources` tab, we right click on the IP and we click on `Open IP Example Design`.

![](./figures/open_ip.png)

We can now close the first project (`project_1`).

If we browse the new project, we notice that there are:

- 1 verilog file *ibert_7series_gtx_0_ex/imports/example_ibert_7series_gtx_0.v*

- 1 IP (xci) *IP/ibert_7series_gtx_0/ibert_7series_gtx_0.xci*

- 1 constraint file (xdc) *ibert_7series_gtx_0_ex/imports/example_ibert_7series_gtx_0.xdc*

- 2 txt files *ibert_7series_gtx_0_ex/imports/example_top_verilog.txt* and *ibert_7series_gtx_0_ex/imports/xdc_7ser_gtx.txt*

## Step 2: create and initialise a git repository

It's time to create a new git repository and add our project files on it.

We start by creating a new blank project on CERN gitlab by clicking [here](https://gitlab.cern.ch/projects/new#blank_project). 

![](./figures/gitlab.png)

We select `Example_project` as project name and we click on `Create Project`.

Now we copy the project URL, e.g. *https://gitlab.cern.ch/$USER/example_project.git*, that will be added as our repo URL.

Now, on a shell, we type:

```bash

git init

git remote add origin https://gitlab.cern.ch/$USER/example_project.git

```

We can now add the **source files** (not the project files) on the git repository.

```bash

git add ibert_7series_gtx_0_ex/imports/example_ibert_7series_gtx_0.v

git add IP/ibert_7series_gtx_0/ibert_7series_gtx_0.xci

git add ibert_7series_gtx_0_ex/imports/example_ibert_7series_gtx_0.xdc

git add ibert_7series_gtx_0_ex/imports/example_top_verilog.txt ibert_7series_gtx_0_ex/imports/xdc_7ser_gtx.txt

git commit -m "First commit, imported project files"

```

## Step 3: import Hog submodule and create Hog project

Now that we have a project in a git repository, we can convert it to a Hog project, to benefit from all the Hog functionalities. 

The first step will be to add Hog submodule in the root path of your repository, preferably using its **relative path**.

The relative path can be found from out repository URL:

- our repository is in [https://gitlab.cern.ch/\$USER/Example\_project](https://gitlab.cern.ch/$USER/Example_project)

- hog repository is in [https://gitlab.cern.ch/hog/Hog](https://gitlab.cern.ch/hog/Hog)

so in our case we will have to add the Hog submodule as:

```bash

  git submodule add ../../hog/Hog.git

  cd Hog

  git checkout Hog2021.2-9

  cd ..

```

:::{note}

If you are working on `gitlab.com`, we advice to add the Hog submodule using its `gitlab.com` mirror. 

- [https://gitlab.com/hog-cern/hog](https://gitlab.com/hog-cern/hog)

To add the repository using the relative path, you should then do

```bash

  git submodule add ../../hog-cern/Hog.git

  cd Hog

  git checkout Hog2021.2-9

  cd ..

```

:::

Next step will be to create the text files containing all project properties and list of project files, as required by hog.

We have to create the following files:

- `Top/<project name>/hog.conf` containing all the project properties, such as *PART*, *Synthesis Strategy*...

- `Top/<project name>/list/<library>.src`containing all the source files (VHDL, verilog, IPs, etc.) that will be put in the VHDL library `<library>`

- `Top/<project name>/list/<simulation_set>.sim` containing all the simulation files that will be put in the simulation set `<simulation_set>`

- `Top/<project name>/list/<constraints_name>.con` containing all the constraints files

We can let Hog handle the creation of the list files automatically, by launching `source Hog/Init.sh` (only in Vivado). However, in our case, we will create all the list files manually.

So we start by creating hog.conf:

```bash

mkdir -p Top/Example_project/list

vim Top/Example_project/hog.conf

```

and we fill it with all the properties required by our project. In our case, we left almost everything as default, so we'll only have to set the *PART* property, which is a *main* property, according to [hog manual](../02-User-Manual/01-Hog-local/01-conf.md#main-section-project-variables):

```ini

#vivado

[main]

PART=xc7k325tffg900-2

```

Now we add all the project files to hog list files. We start by adding all the source files in their HDL library. Since we have only IPs and verilog files, that can't be associated to an HDL library, the library name won't be important. So we can add all the files in a single src file called `work.src`

```bash

vim Top/Example_project/list/work.src

```

Here we add all our files apart from the constraints and simulation ones, and we have to specify which one is the top file:

```

ibert_7series_gtx_0_ex/imports/example_ibert_7series_gtx_0.v top=example_ibert_7series_gtx_0

IP/ibert_7series_gtx_0/ibert_7series_gtx_0.xci

ibert_7series_gtx_0_ex/imports/example_top_verilog.txt 

ibert_7series_gtx_0_ex/imports/xdc_7ser_gtx.txt

```

Finally, we add the constraints in a file called `constraints.con`:

```bash

vim Top/Example_project/list/constraints.con

```

```

ibert_7series_gtx_0_ex/imports/example_ibert_7series_gtx_0.xdc

```

We now add all the files we just created on git and we created a new tag called v0.0.1 as required by Hog:

```bash

git add Top/Example_project/hog.conf

git add Top/Example_project/list/work.src

git add Top/Example_project/list/constraints.con

git commit -m "Converted project to hog project"

git tag v0.0.1 -m "First hog tag"

```

We can now create the Hog project:

```bash

./Hog/CreateProject.sh Example_project

```

The new project will be created in `Projects/Example_project/Example_project.xpr`.

We can open it and confront with the old one; we notice that they are identical.

We can finally build the project by running

```bash

./Hog/LaunchWorkflow.sh Example_project

```

## Step 4: git push

Now we push everything on git:

```bash

git checkout -b Test_branch

git push origin Test_branch

git push --tags

```

# CI Tutorial

:::{warning}

This tutorial refers to an old version of Hog. Please, refer to the official documentation for the updated instructions.

:::

Requirements for this tutorial are a CERN account, a CERN service account ([https://account.cern.ch/account/Management/NewAccount.aspx](https://account.cern.ch/account/Management/NewAccount.aspx)).

## Create Openstack machine

The first step is to create an Openstack Virtual Machine, running CENTOS7. CERN user can easily request a virtual machine for production, test and development purposes. For more information, visit [https://clouddocs.web.cern.ch/](https://clouddocs.web.cern.ch/).

To create the virtual machine, go to [https://openstack.cern.ch/](https://openstack.cern.ch/), click in the left sidebar on Compute->Instances. Then click on `Launch Instance`.

![](figures/launch-instances.png)

Choose an Instance Name (e.g. `tutorial-hog`) and click on `Next`.

![](figures/inst-0.png)

Select the `CC7 - x86_64` image and click on `Next`.

![](figures/inst-1.png)

Select the largest flavour available (typically `m2.large`) and click on `Next`.

![](figures/inst-2.png)

This will take few minutes. Meanwhile create an external volume to be attached to the VM. From the left sidebar, click on `Volumes->Volumes`, and then on `CREATE VOLUME`.

![](figures/volume-0.png)

Choose a Volume Name (e.g. `HOG-DISK`), and allocate at least 100 GB to it. Then click on `CREATE VOLUME`.

![](figures/volume-1.png)

Once the volume has been created and the Virtual Machine has been setup, click on `Actions->MANAGE ATTACHMENTS`.

![](figures/volume-2.png)

Select in the list the VM you just created, e.g. `tutorial-hog`, and click on `ATTACH VOLUME`. 

![](figures/volume-3.png)

We are now ready to configure the machine to run the Hog CI.

### Configure the Openstack VM 

Login to `lxplus.cern.ch`

```bash

ssh -XY user@lxplus.cern.ch

```

Clone the Hog VM setup repository, in any folder inside your AFS area. 

```bash

git clone https://gitlab.cern.ch/hog/vm-setup.git

```

```{note}

EOS is not accessible from the Virtual Machine, so don't clone the repository there.

```

Now login into your VM.

```bash

ssh -XY tutorial-hog

```

Go to the just cloned `vm-setup` folder, and switch to `root` permission.

```bash

sudo su

```

Now install the required dependencies,

```bash

./install_dependecies.sh

```

Then format and mount the volume,

```bash

./volume-setup.sh

```

Finally, install the required software (Vivado, QuestaSim, Quartus) in the attached volume (`/mnt/vd`).

Follow the instructions to install them from the correspondent vendor website.

```{warning}

You are the one responsible for correctly licensing the software.

```

Once this is done, we can install our gitlab-runner. First, we need to find the gitlab-runner registering token. 

Open your gitlab repository on a web browser, and go to `Settings->CI/CD->Runners`. Copy the registration token. 

![](figures/gitlab-token.png)

Go back to your virtual machine, and launch the `install_runner.sh` script, without options to see the Usage description.

```bash

 Hog - Install gitlab runner

 ---------------------------

 Install the Gitlab Runner on a linux machine

 Usage: ./install_runner.sh -u <user_name> -g <user_group> -t <gitlab_token> [OPTIONS]

 Options: --tags: <list of tags for this VM>, default: hog,vivado

          --url: <Gitlab url>. Default: https://gitlab.cern.ch

          --output_limit: <maximum size of output log in the CI jobs. Default: 0 (no limit)>

```

As you can see, it is possible to specify the tags for the runner and the gitlab_url, to create a runner for other Gitlab's locations (e.g. gitlab.com).

```{note}

By default, Hog requires the `vivado` and `hog` tags to run its CI. However, it is possible to change the required tags, individually for each CI job, as we'll be shown later in the tutorial.

```

Launch now the script with the right options. This will create the home folder for your service account on the VM, and register the gitlab runner. The gitlab runner configuration will be saved on `/etc/gitlab-runner/config.toml`, and will look like this,

```toml

concurrent = 1

check_interval = 0

[session_server]

  session_timeout = 1800

[[runners]]

  name = "Hog runner on tutorial-hog"

  url = "https://gitlab.cern.ch"

  token = "<my_token>"

  executor = "shell"

  [runners.custom_build_dir]

  [runners.cache]

    [runners.cache.s3]

    [runners.cache.gcs]

```

Here you can change the concurrent value, to set the maximum numbers of gitlab CI jobs this machine will run in parallel.

If you go now back to your gitlab repository webpage, under `Settings->CI/CD->Runners`, you should see your new runner under the `Other available runners` list.

We are now ready to configure the repository to run the Hog-CI.

## Setup the Hog-CI on your repository

### Configure Gitlab CI/CD variables

To run the Hog-CI, we need to configure some variables in the CI/CD page of our Gitlab repository. Open a web browser and go to [https://gitlab.cern.ch/<user>/PiLUP_devel/-/settings/ci_cd#js-runners-settings](https://gitlab.cern.ch/<user>/PiLUP_devel/-/settings/ci_cd#js-runners-settings).

Expand the `Variables` section and add the following variables.

- __HOG_USER__: Your service account;

- __HOG_EMAIL__: The email of your service account;

- __HOG_PASSWORD__: The password of your service account (Mask it!!);

- __HOG_PATH__: The `PATH` environmental variable for your VM, should include the Vivado bin directory;

- __HOG_PUSH_TOKEN__: The push token you have generated for your service account. You can obtain one [here](https://gitlab.cern.ch/-/profile/personal_access_tokens), logging in with your service account a select `api` as scope;

- __HOG_XIL_LICENSE__: The Xilinx License Server.

More information on the CI/CD variables that Hog uses can be found [here](../02-User-Manual/02-Hog-CI/03-GitLab-CI/01-setup-CI.md#environment-variables).

```{note}

These variables can be also written inside the `.gitlab-ci.yml` file. Gitlab uses the following priority scheme for the environmental variables.

`Gitlab CI/CD Variables -> Jobs Specific Variables -> Global gitlab-ci.yml variables`.

More info [here](https://docs.gitlab.com/ee/ci/variables/)

```

Other interesting variables are:

- __EOS_MGM_URL__: EOS root url, e.g. `root://eosuser.cern.ch`

- __HOG_OFFICIAL_BIN_PATH__: The path for archiving the official binary files of your project. If it starts with `/eos`, it will copy the binary files into the CERN EOS storage, otherwise on the local machine.

- __HOG_CHECK_PROJVER__: If set to 1, Hog CI runs only the projects that have been modified with respect to the target branch;

- __HOG_CHECK_SYNTAX__: If set to 1, Hog will check the syntax of your source files, before running the synthesis;

- __HOG_CHECK_YAMLREF__: If set to 1, Hog will check that the reference Hog version in your `.gitlab-ci.yml` is the same as the cloned submodule;

- __HOG_IP_EOS_PATH__: EOS Path where Hog will store the generated IP files, to speed up the CI;

- __HOG_USE_DOXYGEN__: If set to 1, Hog will produce the doxygen documentation and copy to EOS;

- __HOG_CREATE_OFFICIAL_RELEASE__: If set to 1, Hog creates an official release note using the version and timing summaries taken from the artifact of the projects.

### Remove merge commit

Go to [https://gitlab.cern.ch/<user>/PiLUP_devel/edit](https://gitlab.cern.ch/<user>/PiLUP_devel/edit), expand the __Merge Request settings__ and select *Fast-forward merge*.

<img style="float: middle;" width="700" src="../02-User-Manual/02-Hog-CI/figures/fast-forward.png">

### Pipeline configuration

Go to [https://gitlab.cern.ch/<user>/PiLUP_devel/ci_cd](https://gitlab.cern.ch/<user>/PiLUP_devel/ci_cd), expand the __General pipelines settings__, and set the following configurations,

- Select *git clone*

- Set *Git shallow clone* to 0

- Set *Timeout* to a long threshold, for example 1d

<img style="float: middle;" width="700" src="../02-User-Manual/02-Hog-CI/figures/pipeline.png">

Remember to save the changes!

### Add your service account to the members 

Go to [https://gitlab.cern.ch/<user>/PiLUP_devel/-/project_members](https://gitlab.cern.ch/<user>/PiLUP_devel/-/project_members) and give *Maintainer* rights to your service account

### The gitlab-ci.yml configuration file

Now we need to add our CI configuration file. Open a terminal and go back to the path, where you cloned the `PiLUP_devel` repository. 

Copy the Hog `gitlab-ci.yml` template to the main folder of your repository.

```bash

cp ./Hog/Templates/gitlab-ci.yml .gitlab-ci.yml

```

Let's have a look at the content of this file. Open it with your favourite editor.

```yaml

include:

    - project: 'hog/Hog'

      file: '/hog.yml'

      ref: 'v0.2.1'

#################### example ###########################

### Change 'example' with your project name

GEN:example:

    extends: .generate_project

    variables:

      extends: .vars

      PROJECT_NAME: example

      HOG_ONLY_SYNTH: 0 # if 1 runs only the synthesis

SIM:example:

    extends: .simulate_project

    variables:

      extends: .vars

      PROJECT_NAME: example

```

We need to change the reference value to point to correct Hog version. In this tutorial we are using `Hog2021.2`. 

```yaml

include:

    - project: 'hog/Hog'

      file: '/hog.yml'

      ref: 'Hog2021.2-2'

```

The project that we want to run on the CI is called `simple`, so the yaml would look like,

```yaml

GEN:simple:

    extends: .generate_project

    variables:

      extends: .vars

      PROJECT_NAME: simple

      HOG_ONLY_SYNTH: 0 # if 1 runs only the synthesis

SIM:simple:

    extends: .simulate_project

    variables:

      extends: .vars

      PROJECT_NAME: simple

```

Save the file and push it to the remote.

```bash

git add .gitlab-ci.yml

git commit -m "FEATURE: Enabling Hog Continuous Integration"

git push

```

We are now ready to open the merge request, by either visiting the link git suggests in the output log, or going to [https://gitlab.cern.ch/<user>/PiLUP_devel/-/merge_requests](https://gitlab.cern.ch/<user>/PiLUP_devel/-/merge_requests), and click on `New merge request`. 

As source branch, select `Converting_to_hog` and as target the master branch in your repository. Click then on `Compare branches and continue`.

In the next page, write in the description,

```text

MAJOR_VERSION

```

In this way, Hog will increase the Major number, when creating the new tag once the merge request is accepted. By default, Hog increases the patch number. More info [here](../02-User-Manual/02-Hog-CI/03-GitLab-CI/02-gitlab-workflow).

Leave the remaining fields empty and click on `Create merge request`.

![](figures/create-mr.png)

This will trigger the merge request pipeline (info [here](../02-User-Manual/02-Hog-CI/01-CI-Introduction.md#merge-pull-request-pipeline)). While we wait for the pipeline to finish, one can explore the different executed jobs.

![](figures/pipeline.png)

Once this done, Hog will write a message in the Merge Request page, with a summary of the build projects with timing and versions. You can also have a look at the produced binary files, by downloading the artifacts of the `collect_artifacts` job.

Decompress the download archive, and have a look at the `bin` folder. It contains the `note.md` file that will be used later for the release, and a folder for each build project, in our case only `PiLUP_Kintex_template`. Inside that you'll find the reports and the produced bit files. 

:::{admonition} Local and CI bit-files

:class: tip

Try to do a diff between the bit-file you produced locally in the previous chapter, and the one just downloaded from the CI. They should be identical!

:::

Now we can finally merge our request. Go back to the merge request page, and click on `Merge`. 

![](figures/merge.png)

This triggers the `master` and later the `tag` pipelines, which create the tag and gitlab release. Finally, you can have a look at the generated release at [https://gitlab.cern.ch/<user>/PiLUP_devel/-/release](https://gitlab.cern.ch/<user>/PiLUP_devel/-/release).

![](figures/release.png)

You can see how the pipeline artifacts have been attached to the release. If you configured EOS, these have been also copied to your storage. 

#### The dynamic CI (Optional)

An experimental feature of Gitlab allows the developer to dynamically generate the yaml configuration. Hog supports this feature, even if it still recommended to use the standard CI.

To enable the dynamic CI, one should include the `hog-dynamic.yml` in the `.gitlab-ci.yml`, 

```yaml

include:

    - project: 'hog/Hog'

      file: '/hog-dynamic.yml'

      ref: 'Hog2021.2'

```

That's it! For the dynamic CI you don't need to configure individually each job. Hog will look for each project in the `Top` folder and create dynamically the CI jobs, which will be run in a `child` pipeline. By default, both implementation and simulation jobs are created for each project. It is possible to configure the single projects, by adding a `ci.conf` file in the Top project directory. More info [here](../02-User-Manual/02-Hog-CI/03-GitLab-CI/05-Dynamic-CI).

Create a new branch, commit the modified `.gitlab-ci.yml` and open a new merge request.

```bash

git checkout -b dynamic_ci

git add .gitlab-ci.yml

git commit -m "FEATURE: Dynamic Continuous Integration"

git push

```

Opening the merge request enables the dynamic CI pipeline. 

![](figures/dynamic-ci.png)

The jobs are created in the `dynamic_generate` stage, and run in the child pipeline, in the Upstream section.

The dynamic CI allows the developers to save resources and computing time, when enabling the __HOG_CHECK_PROJVER__ variable. 

If this is enabled, in the standard CI, Hog runs in any case all the generation and simulation jobs specified in the YAML file, but it stops the execution before starting the synthesis or the simulation, if the project has not been changed with respect to the target branch.

The dynamic CI, on the other hand, performs this check before creating the individual project jobs. If you now, enable the variable in the CI/CD settings of your repository, and relaunch the pipeline by doing a fake commit (just change the README for instance), you will see that the `child-pipeline` this time consists only of the `collect_artifact` job. 

![](figures/dynamic-skip.png)