Commit 1b7f98d0 authored by Marcus M. Scheunemann's avatar Marcus M. Scheunemann

article about testing and deploying a Pelican website with Gitlab's CI, using Pygments module

parent 32e8161e
Pipeline #41676387 passed with stage
in 1 minute and 37 seconds
Automatically deploy your Pelican website using Gitlab
######################################################
:date: 2018-12-22 21:28
:modified: 2018-12-22 21:28
:tags: git, CI, gitlab, runner, variables, automatic, deploy
:category: dev/null
:summary: This article describes step by step how to set up jobs with `Gitlab <https://gitlab.com/>`__'s continous itnegration (CI) tools for building and deploying a static website generated with `Pelican <https://github.com/getpelican/pelican>`__. The source of the website needs to be hosted (publically or privatley) on `Gitlab <https://gitlab.com/scheunemann/website>`__. As an example, this web site is the result of the current `master branch <https://gitlab.com/scheunemann/website/tree/master>`__. For each change on a remote branch on Gitlab, the website gets build as a test. If there is a change on the branch "master", an additional job deploys the webpage to an external web server using the file transfer protocol (FTP).
.. role:: raw-html(raw)
:format: html
This article describes step by step how to set up jobs with `Gitlab <https://gitlab.com/>`__'s continous integration (CI) tools for building and deploying a static website generated with `Pelican <https://github.com/getpelican/pelican>`__.
The source of the website needs to be hosted (publically or privatley) on `Gitlab <https://gitlab.com/scheunemann/website>`__.
As an example, this page is the result of the current `master branch <https://gitlab.com/scheunemann/website/tree/master/content/articles/2018-12-22-auto-deploy-static-pelican-website-with-gitlab/main.rst>`__.
For each change on a remote branch on Gitlab, the website gets build as a test. If there is a change on the branch "master", an additional job deploys the webpage to an external web server using the file transfer protocol (FTP).
Requirements
============
- all used python modules should be mentioned by a `requirements file <https://pip.readthedocs.io/en/1.1/requirements.html>`_ placed in the root directory of the repository (It is called `requirements.txt <https://gitlab.com/scheunemann/website/blob/master/requirements.txt>`__ for this project).
- basic Git knowledge
- a Gitlab account. CI tools for all open source projects are free, but there is also limited free access for private projects.
Set up continous integration (CI)
=================================
We create a file called `.gitlab-ci.yml <https://mms.ai>`_ in the root directory of the repository. It describes what the runner has to do on changes in a branch on gitlab.
Preparing the build environment
-------------------------------
The file .gitlab-ci.yml starts with defining a docker image. I've chosen the latest ubuntu image ``ubuntu:latest`` just because of Ubuntu's popularity.
.. code-block:: yaml
image: ubuntu:latest
The minimal ubuntu environment will need some packages in order to allow us to use Python. Before we introduce any job for running scripts, we specify what should be installed before any job:
.. code-block:: yaml
before_script:
- apt update -qq && apt install -y -qq git python-pip
- git submodule update --init
- pip install virtualenv -q
- virtualenv pelican
- source pelican/bin/activate
- pip install -q -r requirements.txt
Before any job, the following happens:
1) updating ubuntu and installing `git` and `pip` (we need `git` for updating all submodules, e.g., themes and plugins, you can skip that if you don't use submodules)
2) creating a virtual environment to allow for installing specific versions of Python packages independent of global system packages
3) installing all Python packages required by your website
Creating a job for buidling the website
---------------------------------------
We define a "job" called "build website" for building the pelican site. If the site gets build without errors, the job is successful. The line ``stage: build`` is optional.
.. code-block:: yaml
build website:
stage: build
script:
- pelican content -o output -s pelicanconf.py
You can already merge all three sections into one YAML file called .gitlab-ci.yml, add it to your root of your project and push it to the remote repository on Gitlab.
The output under "CI/CD :raw-html:`&rarr;` Pipelines" will look like the following, with the job "build website" still running.
.. image:: {attach}job-build-website-running.png
:alt: The job "Build website" is still running
When the job is done successfully, i.e., the website was built, the sign changes to a green tick:
.. image:: {attach}job-build-website-passed.png
:alt: The job "Build website" is still running
Refer to the full output for the job `#139894076 <https://gitlab.com/scheunemann/website/-/jobs/139894076>`__ of the pipline `#41645760 <https://gitlab.com/scheunemann/website/pipelines/41645760>`__ for details.
Deploying the built website with FTP
------------------------------------
If we change the master branch, the same job "build website" gets triggered. In addition, I want to deploy to webpage to an external webserver if the job "build website" was successful.
The additional job is called "deploy website".
.. code-block:: yaml
deploy website:
stage: deploy
only:
- master
script:
- pelican content -o output -s publishconf.py
- apt install -y -qq lftp
- lftp $FTP_SERVER -u $FTP_USER,$FTP_PASS -e "mirror -R output/ / ; quit"
What's happening in "deploy website":
1) the job gets assigned to the stage "deploy"
2) we define that this job gets triggered only in the branch "master"
3) we build the page with the "publishconf.py" settings
4) we install the client ``lftp``
5) lftp gets triggered so that the code is uploaded to an external webserver via ftp
The commit and the job execution is shown in commit `32e8161e <https://gitlab.com/scheunemann/website/commit/32e8161e45b95751fbfbf7acd2db89e10418c93d>`_. Although both jobs are part of the ".gitlab-ci.yml", only one has been triggered as we pushed to the branch "article-auto-deploy" (i.e., not to the branch "master").
However, when merged into master, the pipeline consists of two stages as can be seen here:
.. image:: {attach}jobs-for-building-and-deploying-website-running.png
:alt: The job "Build website" is still running, and the deploy job is in the pipe
In the following, all three executions after each commit are shown. Only when the branch "article-auto-deploy" was merged into "master", both jobs got triggered:
.. image:: {attach}all-three-commits-with-different-stages.png
:alt: All three commits and the triggered jobs.
Setting up environment variables
--------------------------------
You will notice environment variables (e.g. `$FTP_PASS`) in the upper example. As the file .gitlab-ci.yml can be publically acessible in our repo, we don't want everybody to see our credentials for the server. I firstly set up a dedicated FTP user on my server. I'll then used the Gitlab feature "Settings :raw-html:`&rarr;` CI \ CD :raw-html:`&rarr;` Variables" for setting up variables as follows:
.. image:: {attach}set-environment-varibles-for-CI.png
:alt: Setting up environment variables to use in .gitlab-ci.yml
I made those variables only accessible to my protected branch master. Otherwise, everybody with access to another branch might be able to adapt the YAML file and retrieve my FTP password with putting `echo $FTP_PASS` one of the jobs.
The full script
---------------
The full script as described here can be found in my website repository: https://gitlab.com/scheunemann/website/blob/6331d62b6b93ac573e8d3198c19d9bcfc57af7a0/.gitlab-ci.yml.
Troubleshooting
==================================
Runner
------
We use Gitlab's Shared Runners for a quick start. They are enabled by default on newer gitlab versions. Check if they are enabled under "Settings :raw-html:`&rarr;` CI/CD :raw-html:`&rarr;` Runners". In the image belove, you see that the runner #44028 comes with docker pre-installed:
.. image:: {attach}select-shared-runner-gitlab.png
:alt: Select Shared Runners on gitlab
More information
----------------
- Runners: https://docs.gitlab.com/ee/ci/runners/README.html.
- Gitlab's CI quick start: https://docs.gitlab.com/ee/ci/quick_start/
......@@ -100,6 +100,9 @@ JINJA_ENVIRONMENT = {
TYPOGRIFY = True
#TYPOGRIFY_IGNORE_TAGS = []
PYGMENTS_STYLE = 'murphy'
PYGMENTS_RST_OPTIONS = {'linenos': 'table', 'anchorlinenos' : 'true'}
##
## Plugin settings: pelican-youtube
##
......
......@@ -3,4 +3,5 @@ python-gettext==4.0
pybtex>=0.22.0
pelican_youtube>=0.2.0
pelican-bib==0.2.7
typogrify>=2.0.7
\ No newline at end of file
typogrify>=2.0.7
Pygments>=2.3.1
\ No newline at end of file
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment