Commit 7b57ad84 authored by Daniele Procida's avatar Daniele Procida
Browse files

Added structured documentation

All sections in existing documentation (README.md and /docs) are marked as one of:

* having been transferred
* requiring clarification

The new documentation is in /docs, and is written in RST. Once any questions have
been resolved, it is ready to be published to ReadTheDocs.
parent d80f9a83
......@@ -26,3 +26,8 @@ celerybeat-schedule
# django files
media/
/static_collected
# documentation
/docs/_*
/docs/env
CoReport Introduction
CoReport Introduction - now in docs
-------------------------------------------------------------------------------
CoReport is an [open-source](https://gitlab.com/what-digital/covid19-report/-/blob/master/LICENSE) tool that enables an organization to collect structured status reports from other organizations at set intervals.
......@@ -9,7 +9,7 @@ The development of CoReport was initiated by [what.](https://what.digital) in co
Thanks to Victor Yunenko (victor.yunenko@what.digital), Jonathan Stoppani (jonathan.stoppani@divio.com), Anand Patel (anand@what.digital), Elliott White (elliott@what.digital), Mario Colombo (mario@what.digital), Marcus Kuhn (marcus@what.digital).
Functionality
Functionality - now in docs
-------------------------------------------------------------------------------
- An admin can create organizations and invite users to them.
......@@ -24,11 +24,11 @@ Functionality
CoReport is under active development.
Development Setup
Development Setup - now in docs
-------------------------------------------------------------------------------
Built on Python 3.6, Django 2.2, Webpack 4, TypeScript 3.
### Docker setup
### Docker setup - now in docs
This setup is highly inefficient and error-prone, needed mostly for production simulations and requirements complication. When you're developing the backend of a project on regular basis you should set it up using docker only once and then use the native setup below.
......@@ -41,7 +41,7 @@ This setup is highly inefficient and error-prone, needed mostly for production s
- `yarn start`
- `docker-compose up`
### Native setup
### Native setup - is this required?
- `pip install -r requirements.txt`
- in `.env-for-native` uncomment `IS_NATIVE_SETUP=true`
- `divio project pull db test`
......@@ -51,7 +51,7 @@ This setup is highly inefficient and error-prone, needed mostly for production s
- `yarn install --pure-lockfile`
- `yarn start`
### Update requirements.txt
### Update requirements.txt - now in docs
```bash
docker-compose run --rm web fish --command 'pip-reqs compile; pip-reqs resolve; pip install --no-index --no-deps --requirement requirements.urls'
......@@ -59,19 +59,19 @@ docker-compose run --rm web fish --command 'pip-reqs compile; pip-reqs resolve;
For installing the compiled requirements in docker you have to rebuild it with `docker-compose build`.
### Shell
### Shell - now in docs
- `docker-compose run --rm web fish` - a disposable container
### Translations
### Translations - need clarification
- `django-admin makemessages -l {lang_code} --no-wrap`
### How to drop the database
### How to drop the database - now in docss
You can flush the local & server db in the following way:
- remove the db container along with its data volume
- run migrations - now you have an empty db
- push it to the server `divio project push db test`
### How to edit an external package
### How to edit an external package - need clarification
- Firstly the files that pycharm/intellij shows you in the docker remote files are editable but have no effect - it's just a copy-past of the real files
- See the divio guide - https://docs.divio.com/en/latest/how-to/create-addon.html
......
# Minimal makefile for Sphinx documentation
#
# You can set these variables from the command line.
SPHINXOPTS =
SPHINXBUILD = sphinx-build
SPHINXPROJ = covid19report
SOURCEDIR = .
BUILDDIR = _build
VENV = env/bin/activate
PORT = 9090
# Put it first so that "make" without argument is like "make help".
help:
@$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
install:
@echo "... setting up virtualenv"
python3 -m venv env
. $(VENV); pip install --upgrade -r requirements.txt
@echo "\n" \
"--------------------------------------------------------------- \n" \
"* watch, build and serve the documentation: make run \n" \
"* check spelling: make spelling \n" \
"\n" \
"enchant must be installed in order for pyenchant (and therefore \n" \
"spelling checks) to work. \n" \
"--------------------------------------------------------------- \n"
clean:
-rm -rf _build/*
run:
. $(VENV); sphinx-autobuild $(ALLSPHINXOPTS) --ignore ".git/*" --ignore "*.scss" . -a _build/html --host 0.0.0.0 --port $(PORT)
test:
. $(VENV); $(SPHINXBUILD) -b html . _build/html
html:
. $(VENV); $(SPHINXBUILD) -b html . _build/html
spelling:
. $(VENV); $(SPHINXBUILD) -b spelling $(ALLSPHINXOPTS) . _build/spelling
@echo
@echo "Check finished. Wrong words can be found in " \
"_build/spelling/output.txt."
quickstart:
. $(VENV); sphinx-quickstart
.PHONY: help install clean run Makefile
# Catch-all target: route all unknown targets to Sphinx using the new
# "make mode" option. $(O) is meant as a shortcut for $(SPHINXOPTS).
%: Makefile
@$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
Explanation
============
Key functionality
-----------------
Administrators can:
* can create organisations and invite users to them
* manage groups that to which different organisations are assigned
* manage questions, that are assigned to groups; groups determine which questions apply to each organisation
* export all answers provided by users for purposes of data analysis
Users can:
* see the questions for their organisations
* enter and edit (until a deadline) their answers
The system will notify users when answers are overdue.
In development
--------------
The system will provide real-time reporting of the data, in a dashboard accessible only to authorised users.
# Configuration file for the Sphinx documentation builder.
#
# This file only contains a selection of the most common options. For a full
# list see the documentation:
# http://www.sphinx-doc.org/en/master/config
# -- Path setup --------------------------------------------------------------
# If extensions (or modules to document with autodoc) are in another directory,
# add these directories to sys.path here. If the directory is relative to the
# documentation root, use os.path.abspath to make it absolute, like shown here.
#
# import os
# import sys
# sys.path.insert(0, os.path.abspath('.'))
# -- Project information -----------------------------------------------------
project = 'COVID-19 Report'
copyright = '2020, What Digital and others'
author = 'What Digital and others'
# -- General configuration ---------------------------------------------------
# Add any Sphinx extension module names here, as strings. They can be
# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
# ones.
extensions = [
]
# Add any paths that contain templates here, relative to this directory.
templates_path = ['_templates']
# List of patterns, relative to source directory, that match files and
# directories to ignore when looking for source files.
# This pattern also affects html_static_path and html_extra_path.
exclude_patterns = ['_build', 'Thumbs.db', '.DS_Store']
# -- Options for HTML output -------------------------------------------------
# The theme to use for HTML and HTML Help pages. See the documentation for
# a list of builtin themes.
#
# on_rtd: whether we are on readthedocs.org
import os
on_rtd = os.environ.get('READTHEDOCS', None) == 'True'
if not on_rtd: # only import and set the theme if we're building docs locally
try:
import sphinx_rtd_theme
except ModuleNotFoundError:
html_theme = 'default'
else:
html_theme = 'sphinx_rtd_theme'
html_theme_path = [sphinx_rtd_theme.get_html_theme_path()]
# Add any paths that contain custom static files (such as style sheets) here,
# relative to this directory. They are copied after the builtin static files,
# so a file named "default.css" will overwrite the builtin "default.css".
html_static_path = ['_static']
Get started
===========
Set up a local instance for development
---------------------------------------
.. note::
CoReport is a project in development. The project and this documentation are updated regularly, and these notes
may have changed since the last time you read them.
Clone the codebase:
.. code-block:: bash
git clone git@gitlab.com:what-digital/covid19-report.git
The project requires that you have Docker installed; you'll need:
* Macintosh users: `Docker for Mac <https://docs.docker.com/docker-for-mac/>`_
* Windows users: `Docker for Windows <https://docs.docker.com/docker-for-windows/>`_
* Linux users: `Docker CE <https://docs.docker.com/install/#server>`_ and `Docker Compose
<https://docs.docker.com/compose/install/>`_
``cd`` into the ``covid19-report`` directory and build the application:
.. code-block:: bash
docker-compose build
This will create images based on the services described in the ``docker-compose.yml`` file, including the
``web`` service, The Celery services, the Postgres database and any others.
Migrate the database:
.. code-block:: bash
docker-compose run web python manage.py migrate
This runs the Django ``manage.py migrate`` inside a ``web`` container.
Launch the application:
.. code-block:: bash
docker-compose up
It will take a few moments to start up all the containers, and then you will be able to reach the application at
http://localhost:8000 (this is configured in ``docker-compose.yml``). You should see an **unstyled** login page.
In order to generate the styling for the site, along with all its other frontend components, they will need to be
built manually. This requires Yarn, a Node package, to be installed. (It is beyond the scope of this documentation
to describe that process.)
In a new terminal session in the project directory, run:
.. code-block:: bash
yarn install --pure-lockfile
This installs the components that the project's frontend requires. Then:
.. code-block:: bash
yarn start
This launches the frontend; the frontend assets will be compiled and the served on port 8090. Yarn watches the
frontend source and will re-compile the assets on any change.
When you refresh the login page, the styling should now appear.
Prepare the project for cloud deployment
--------------------------------------------
The CoReport project is ready for deployment on the Divio cloud management platform, and the project already
contains the components required for this. All your repository requires in order to test in a cloud deployment is
to associate it with a Divio cloud project.
Using the Divio Control Panel (you will need an account, it's free), `create a new project
<https://control.divio.com/control/project/create>`_.
Select the defaults:
* *Platform*: ``Python``
* *Type*: ``Django``
You can use Divio's Git server (also the default).
Hit **Skip** in the *Subscription* view.
After a few moments, you'll be presented with its dashboard, which will look something like this:
.. image:: /images/project-undeployed.png
:alt: 'New project in the Dashboard'
Install the `Divio CLI <https://github.com/divio/divio-cli>`_: ``pip install divio-cli``.
Run:
.. code-block:: bash
divio login
This will fetch a token from https://control.divio.com/account/desktop-app/access-token/.
Upload your public key to https://control.divio.com/account/ssh-keys/
Get your project's *slug* from the Control Panel:
.. image:: /images/project-slug.png
:alt: 'Project slug'
:width: 400px
Or you can see it by listing your projects:
.. code-block:: bash
divio project list
Set up the project:
.. code-block:: bash
divio project setup <project slug>
Once the ``divio project setup`` command has finished pulling down the repository, you will find a file in the
newly-created directory:: ``.aldryn``. The file contains the slug and id of the cloud project, and is what the
Divio CLI uses to associate the local project with its counterpart on the cloud.
Move this file into the CoReport ``covid19-report`` project.
We need to do a similar thing with Git. In the local Divio project, find its Git URL:
.. code-block:: bash
git remote show origin
In the ``covid19-report`` project, add a new remote called ``divio``, using this URL, for example:
.. code-block:: bash
git remote add divio git@git.divio.com:my-divio-coreport-project.git
Your ``covid19-report`` project is now associated with the Divio cloud project you just created. You can dispose
of the local project that was created by the ``divio project setup`` command; it's no longer needed.
You can now interact with both the ``origin`` remote (for example, to pull new updates) and push your changes to
the ``divio`` remote (to test it on the cloud).
Working with the project
------------------------
Basic workflow:
.. code-block:: bash
git add <files> # to stage changes for commit
git commit # to commit changes
git push divio # push changes to your cloud project
divio project deploy # deploy the cloud Test server
Also useful:
.. code-block:: bash
divio project pull [or push] db # copy the database to/from the cloud project
You can access the Test and Live sites of your cloud project via the Control Panel. See the Divio developer
handbook for `a quick guide to using the Divio CLI <https://docs.divio.com/en/latest/how-to/local-cli.html>`_.
Whenever you pull or make changes that will require rebuilding the local project (for example, changing
``requirements.in``) re-run:
.. code-block:: bash
docker-compose build
How-to guides
=============
Work in and with the Dockerised environment
----------------------------------------------
Almost any command can be executed inside the Dockerised environment. One way is to precede the command with:
.. code-block:: bash
docker-compose run web
followed by the command. For example, to run Django migrations
.. code-block:: bash
docker-compose run web python manage.py migrate
If you need to run several commands it may be more convenient to open a bash [or fish] shell:
.. code-block:: bash
docker-compose run web bash [or fish]
or a Django shell:
.. code-block:: bash
docker-compose run web python manage.py shell
Update ``requirements.txt``
---------------------------
The project uses the Python dependencies listed in ``requirements.txt``. This is a fully-pinned list of all
dependencies. The list is derived from the project's ``requirements.in``.
The build process uses ``requirements.txt`` if it's present, *not* ``requirements.in``. To update
``requirements.txt``:
.. code-block:: bash
docker-compose run pip-reqs compile
To rebuild the containers with the new requirements:
.. code-block:: bash
docker-compose build
(The actual build process will also execute ``pip-reqs resolve`` (which creates a ``requirements.urls`` file)
followed by ``pip install --no-index --no-deps --requirement requirements.urls`` (which uses the
``requirements.urls`` file).
See `How to pin all of your projects Python dependencies
<https://docs.divio.com/en/latest/how-to/resources-pin-dependencies.html>`_ in the Divio developer handbook for
more information about how this works.
Interact with the local and cloud database
--------------------------------------------------
See `How to interact with your projects database
<https://docs.divio.com/en/latest/how-to/interact-database.html>`_ in the Divio developer handbook.
How to build the documentation
------------------------------
The documentation is written in `Restructured Text
<https://docutils.readthedocs.io/en/sphinx-docs/user/rst/quickstart.html>`_, built with `Sphinx
<https://www.sphinx-doc.org/en/master/>`_ and hosted at `Read the Docs <https://readthedocs.com>`_.
The documentation can be built locally:
.. code-block:: bash
cd docs # navigate to the documentation directory
make install # use the supplied Makefile to set up a virtual environment for building the documentation
make run # builds and serve the documentation
The documentation is served at http://localhost:9090/how-to.html.
.. COVID-19 Report documentation master file, created by
sphinx-quickstart on Fri Mar 20 18:55:51 2020.
You can adapt this file completely to your liking, but it should at least
contain the root `toctree` directive.
COVID-19 Report
===========================================
CoReport is an `open-source <https://gitlab.com/what-digital/covid19-report/-/blob/master/LICENSE>`_ tool that
enables an organisation to collect structured status reports from other organisations at set intervals.
CoReport is currently being deployed by the Swiss canton of Baselland, on hosting sponsored by `Divio
<https://www.divio.com>`_. Divio is also donating further hosting of the tool for other organizations.
The development of CoReport was initiated by `what. <https://what.digital>`_ in collaboration with Divio (see our
:ref:`contributors`). The first iteration of the tool was developed for the Swiss canton of Baselland.
`Project source code <https://gitlab.com/what-digital/covid19-report/>`_
.. toctree::
:maxdepth: 2
:caption: Contents
get-started
how-to
reference
background
@ECHO OFF
pushd %~dp0
REM Command file for Sphinx documentation
if "%SPHINXBUILD%" == "" (
set SPHINXBUILD=sphinx-build
)
set SOURCEDIR=.
set BUILDDIR=_build
if "%1" == "" goto help
%SPHINXBUILD% >NUL 2>NUL
if errorlevel 9009 (
echo.
echo.The 'sphinx-build' command was not found. Make sure you have Sphinx
echo.installed, then set the SPHINXBUILD environment variable to point
echo.to the full path of the 'sphinx-build' executable. Alternatively you
echo.may add the Sphinx directory to PATH.
echo.
echo.If you don't have Sphinx installed, grab it from
echo.http://sphinx-doc.org/
exit /b 1
)
%SPHINXBUILD% -M %1 %SOURCEDIR% %BUILDDIR% %SPHINXOPTS% %O%
goto end
:help
%SPHINXBUILD% -M help %SOURCEDIR% %BUILDDIR% %SPHINXOPTS% %O%
:end
popd
Reference
=========
Key components
-------------------------------------------------------------------------------
* Python 3.6
* Django 2.2
* Webpack 4
* TypeScript 3
Backend (Django) code
---------------------
The root ``backend`` module is reserved for non-project related applications, eg ``auth`` or ``articles``.
Project-specific applications should live in a project module of their own, for example ``backend.dectris``.
Similarly, the ``backend/templates`` directory is only for global templates. For anything, else use app-specific
templates (i.e. do things the standard Django way).
Frontend code
---------------------
For any questions, please refer to Victor Yunenko (victor.yunenko@what.digital)
Project-specific scss styles can be in ``global``; however, JS logic must not be.
When you need to add a script for a new page/block add a new ``entry`` in ``webpack.config.js`` and a page/block
submodule.
For global scripts and styles use the ``global`` entry, and add the respective HTML code in your template.
If you need to add a global variable to JS, add it to the ``backend/templates/default.html#DJANGO`` const and
extend ``frontend/global/ts/django.ts#DJANGO``.
If you need to add a static file, e.g. on URL ``https://localhost/static/img/icon.png``, add it to ``frontend/`` -
everything in there will be accessible under the ``/static/`` URL path.
If you need to add a new font to CSS, use the global path as you normally would, e.g. ``url('~pages/homepage/fonts/frutiger.woff')``.
If you need to add images or other assets, add them under the respective module, e.g. ``global``, ``vendor``, ``pages/homepage``.
We use 24 columns in our Bootstrap 4 configuration.
.. _contributors:
Contributors
-------------
* Victor Yunenko victor.yunenko@what.digital
* Jonathan Stoppani jonathan.stoppani@divio.com
* Anand Patel anand@what.digital
* Elliott White elliott@what.digital
* Mario Colombo mario@what.digital
* Marcus Kuhn marcus@what.digital
divio-docs-theme
pyenchant
sphinx
sphinx-autobuild
sphinx_rtd_theme
sphinxcontrib-spelling
Store project-related applications in a project module, eg `backend.dectris`. The root `backend` module reserve for non-project related apps, eg `auth` or `articles`.
The `backend/templates` directory is only for global templates, for anything else use app specific templates per django guidelines.
needs clarification:
The settings.py file is split into 4 categories, keep them in mind:
- django core
- django packages
The `backend/templates` directory is only for global templates, for anything else use app specific templates per django guidelines.
### envs
### envs - needs clarification
When you're editing the env files add the changes to another inactive changes, this way you're never going to commit those changes to git.