Commit 696adbc8 authored by Marcus M. Scheunemann's avatar Marcus M. Scheunemann

Merge branch 'article-auto-deploy' and 'bootstrapify'

parents 6331d62b dff2d565
Pipeline #41692772 passed with stages
in 6 minutes and 10 seconds
[submodule "themes/pelican-twitchy"]
path = themes/pelican-twitchy
url = https://github.com/ingwinlu/pelican-twitchy.git
[submodule "plugins/pelican-bootstrapify"]
path = plugins/pelican-bootstrapify
url = https://github.com/ingwinlu/pelican-bootstrapify
Automatically deploy your Pelican website using GitLab
######################################################
:date: 2018-12-22 21:28
:modified: 2018-12-30 17:51
:tags: Pelican, CI, GitLab, runner, automatic deploy
:category: misc
:summary: This article describes step by step how to set up jobs with GitLab_'s continuous integration (CI) tools for building and deploying a static website generated with Pelican_. The source of the website needs to be hosted (publicly or privately) on `GitLab <https://gitlab.com/scheunemann/website>`__. As an example, this page is the result of the `current master`_. For each change on a remote branch on GitLab, the website gets build as a test. If there is a change on the branch "master", an additional job deploys the website to an external web server using the file transfer protocol (FTP).
.. _Python: https://www.python.org/
.. _GitLab: https://gitlab.com/
.. _Pelican: http://docs.getpelican.com
.. _current master: https://gitlab.com/scheunemann/website/tree/master/
.. _page: https://gitlab.com/scheunemann/website/tree/master/content/articles/2018-12-22-auto-deploy-static-pelican-website-with-gitlab/main.rst
.. _requirements.txt: https://gitlab.com/scheunemann/website/blob/master/requirements.txt
.. _.gitlab-ci.yml: https://gitlab.com/scheunemann/website/blob/6331d62b6b93ac573e8d3198c19d9bcfc57af7a0/.gitlab-ci.yml
This article describes step by step how to set up jobs with GitLab_'s continuous integration (CI) tools for building and deploying a static website generated with Pelican_.
The source of the website needs to be hosted (publicly or privately) on `GitLab <https://gitlab.com/scheunemann/website>`__.
As an example, this page is the result of the `current master`_ branch.
For each change on a remote branch on GitLab, the website gets build as a test. If there is a change on the branch "master", an additional job deploys the website to an external web server using the file transfer protocol (FTP).
The full script as described below can be found in my website repository: `.gitlab-ci.yml`_.
Requirements
============
- all used python modules should be mentioned by a `requirements file <https://pip.readthedocs.io/en/1.1/requirements.html>`_ placed in the root directory of the repository (It is called requirements.txt_ for this project).
- basic Git knowledge
- a GitLab account. CI tools for all open source projects are free, but there is also limited free access for private projects.
Set up continuous integration (CI)
==================================
We create a file called :code:`.gitlab-ci.yml` in the root directory of the repository. It describes the jobs executed by the runner on changes in a branch in the remote repository (hosted on GitLab).
Preparing the build environment
-------------------------------
The file :code:`.gitlab-ci.yml` starts with defining a docker image. I chose the latest Ubuntu image :code:`ubuntu:latest` just because of Ubuntu's popularity.
.. code-block:: yaml
image: ubuntu:latest
The minimal Ubuntu environment needs some packages in order to allow us to use Python_. We specify what should be installed before executing any job:
.. code-block:: yaml
:linenostart: 3
before_script:
- apt update -qq && apt install -y -qq git python-pip
- git submodule update --init
- pip install virtualenv -q
- virtualenv pelican
- source pelican/bin/activate
- pip install -q -r requirements.txt
Before any job, the following happens:
1) updating Ubuntu and installing :code:`git` and :code:`pip` (we need :code:`git` for updating all submodules, e.g., themes and plugins, you can skip that if you don't use submodules)
2) creating a virtual environment to allow for installing specific versions of Python packages independent of global system packages
3) installing all Python packages required by your website
Creating a job for building the website
---------------------------------------
We define a job :code:`build website` for building the pelican site. If the site gets build without errors, the job is successful. The line :code:`stage: build` is optional.
.. code-block:: yaml
:linenostart: 11
build website:
stage: build
script:
- pelican content -o output -s pelicanconf.py
You can already merge all three sections into one YAML file called :code:`.gitlab-ci.yml`, add it to your root of your project and push it to the remote repository on GitLab.
The output under **CI/CD ➔ Pipelines** will look like the following, with the job :code:`build website` still running.
.. image:: {attach}job-build-website-running.png
:alt: The job "Build website" is still running
When the job is done successfully, i.e., the website was built, the sign changes to a green tick:
.. image:: {attach}job-build-website-passed.png
:alt: The job "Build website" is still running
Refer to the full output for the job `#139894076 <https://gitlab.com/scheunemann/website/-/jobs/139894076>`__ of the pipeline `#41645760 <https://gitlab.com/scheunemann/website/pipelines/41645760>`__ for details.
Deploying the built website with FTP
------------------------------------
If we change the master branch, the same job :code:`build website` gets triggered. In addition, I want to deploy the web site to an external web server if the job :code:`build website` was successful.
The additional job is called :code:`deploy website`.
.. code-block:: yaml
:linenostart: 16
deploy website:
stage: deploy
only:
- master
script:
- pelican content -o output -s publishconf.py
- apt install -y -qq lftp
- lftp $FTP_SERVER -u $FTP_USER,$FTP_PASS -e "mirror -R output/ / ; quit"
What's happening in :code:`deploy website`:
1) the job gets assigned to the :code:`deploy` stage
2) we define that this job gets triggered only in the branch "master"
3) we build the page with the "publishconf.py" settings
4) we install the client :code:`lftp`
5) :code:`lftp` gets triggered so that the code is uploaded to an external web server via ftp
The commit and the job execution is shown in commit `32e8161e <https://gitlab.com/scheunemann/website/commit/32e8161e45b95751fbfbf7acd2db89e10418c93d>`_. Although both jobs are part of the `.gitlab-ci.yml`_, only one has been triggered as we pushed to the branch "article-auto-deploy" (i.e., not to the branch "master").
However, when merged into master, the pipeline consists of two stages as can be seen here:
.. image:: {attach}jobs-for-building-and-deploying-website-running.png
:alt: The job "Build website" is still running, and the deploy job is in the pipe
In the following, all three executions after each commit are shown. Only when the branch "article-auto-deploy" was merged into "master", both jobs got triggered:
.. image:: {attach}all-three-commits-with-different-stages.png
:alt: All three commits and the triggered jobs.
Setting up environment variables
--------------------------------
You will notice environment variables (e.g. :code:`$FTP_PASS`) in the upper example. As the file :code:`.gitlab-ci.yml` can be publicly accessible in our repo, we don't want everybody to see our credentials for the server. I firstly set up a dedicated FTP user on my server. I'll then used the GitLab feature **Settings ➔ CI/CD ➔ Variables** for setting up variables as follows:
.. image:: {attach}set-environment-varibles-for-CI.png
:alt: Setting up environment variables to use in ".gitlab-ci.yml" script.
I made those variables only accessible to my protected branch master. Otherwise, everybody with access to another branch might be able to adapt the YAML file and retrieve my FTP password with putting :code:`echo $FTP_PASS` one of the jobs.
Troubleshooting
==================================
Runner
------
We use GitLab's Shared Runners for a quick start. They are enabled by default on newer GitLab versions. Check if they are enabled under **Settings ➔ CI/CD ➔ Runners**. In the image below, you see that the runner #44028 comes with docker pre-installed:
.. image:: {attach}select-shared-runner-gitlab.png
:alt: Select Shared Runners on GitLab
More information
----------------
- Runners: https://docs.gitlab.com/ee/ci/runners/README.html.
- GitLab's CI quick start: https://docs.gitlab.com/ee/ci/quick_start/
......@@ -90,7 +90,7 @@ CATEGORIES_SAVE_AS = 'categories/index.html'
THEME_TEMPLATES_OVERRIDES = []
# Plugins
PLUGIN_PATHS = ['../pelican-plugins', ]
PLUGIN_PATHS = ['plugins', '../pelican-plugins', ]
PLUGINS = ['i18n_subsites', ]
JINJA_ENVIRONMENT = {
'extensions': ['jinja2.ext.i18n'],
......@@ -100,12 +100,28 @@ JINJA_ENVIRONMENT = {
TYPOGRIFY = True
#TYPOGRIFY_IGNORE_TAGS = []
PYGMENTS_STYLE = 'murphy'
PYGMENTS_RST_OPTIONS = {'linenos': 'table', 'anchorlinenos' : 'true'}
##
## Plugin settings: pelican-youtube
##
# installed via pip
PLUGINS.append('pelican_youtube')
##
## Plugin settings: pelican-bootstrapify
##
PLUGINS.append('pelican-bootstrapify')
BOOTSTRAPIFY = {
'table': ['table', 'table-striped', 'table-hover'],
'img': ['img-fluid', 'border', 'border-primary'],
'blockquote': ['blockquote'],
#'code': ['badge badge-light']
}
##
## Plugin settings: pelican-bibtex
##
......
Subproject commit b2e9e93630d2162fce035439b0142e67b749cb73
......@@ -3,4 +3,6 @@ python-gettext==4.0
pybtex>=0.22.0
pelican_youtube>=0.2.0
pelican-bib==0.2.7
typogrify>=2.0.7
\ No newline at end of file
typogrify>=2.0.7
Pygments>=2.3.1
beautifulsoup4>=4.6.3
\ No newline at end of file
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment