README.md 4.95 KB
Newer Older
Christophe Benz's avatar
Christophe Benz committed
1 2 3 4 5 6
# Open Mobility Indicators – Notebooks

## Installation

Some dependencies like pyrosm or osmnx are more easily installable with `conda` than with `pip`.

Christophe Benz's avatar
Christophe Benz committed
7
Using `conda` without modifying your user environment is possible with `pyenv`.
Christophe Benz's avatar
Christophe Benz committed
8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25

Prerequisites: [pyenv](https://github.com/pyenv/pyenv)

```bash
# Install miniconda3-latest Python version with pyenv:
pyenv install miniconda3-latest

# Create and activate a virtualenv named test-conda-1 from the miniconda3-latest Python miniconda3-latest:
pyenv virtualenv miniconda3-latest test-conda-1
pyenv activate test-conda-1

# Your shell should be prepended by `(test-conda-1)`.

# The `conda` command is now in the PATH of your shell.

# Configure "conda-forge" external source:
conda config --prepend channels conda-forge

Christophe Benz's avatar
Christophe Benz committed
26 27
# Create a new conda environment named "indicator-notebooks":
conda create -n indicator-notebooks --strict-channel-priority
Christophe Benz's avatar
Christophe Benz committed
28

Christophe Benz's avatar
Christophe Benz committed
29
# This "indicator-notebooks" conda env sits in the "test-conda-1" virtualenv.
Christophe Benz's avatar
Christophe Benz committed
30 31 32 33 34

# Source the conda.sh file (once during the shell session) to avoid failing at the next step:
# (OR use `conda init bash` to append a config block to your shell config file)
. ~/.pyenv/versions/miniconda3-latest/etc/profile.d/conda.sh

Christophe Benz's avatar
Christophe Benz committed
35
# Your shell should be prepended by `(indicator-notebooks)` and `(test-conda-1)`.
Christophe Benz's avatar
Christophe Benz committed
36

Christophe Benz's avatar
Christophe Benz committed
37 38
# Activate the "indicator-notebooks" conda environment:
conda activate indicator-notebooks
Christophe Benz's avatar
Christophe Benz committed
39 40

# Install project dependencies:
41
conda env update --file environment.yml
Christophe Benz's avatar
Christophe Benz committed
42

43
# environment.yml contains the dependencies of the project (deep and frozen).
Christophe Benz's avatar
Christophe Benz committed
44 45 46

# To install other packages:
conda install <package_name>
47 48
# Then regenerate environment.yml (if the added package is intended to stay):
conda env export > environment.yml
Christophe Benz's avatar
Christophe Benz committed
49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66
```

Then run jupyter lab:

```bash
jupyter lab
```

When you're done working with the project, you may deactivate the virtualenvs like this:

```bash
# To deactivate the conda env:
conda deactivate

# To deactivate the test-conda-1 virtualenv:
pyenv deactivate
```

67
## Notebooks
Christophe Benz's avatar
Christophe Benz committed
68

69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100
Each notebook is stored in a sub-directory of [`notebooks`](./notebooks) and must follow some conventions regarding its input parameters, output files and other companion files.

Those conventions allow the automatic workflow to compute all the notebooks at once.

### Notebook name

The name of the notebook should be the same as the sub-directory. For example: `./notebooks/omi_score/omi_score.ipynb`.

### Input parameters

The following parameters are common to all the notebooks. They are designed to be overloaded by [Papermill](https://papermill.readthedocs.io/en/latest/) for batch notebook execution.

- `debug` (`bool`): if `True`, the cells starting with `if debug:` will be computed
- `download_data` (`bool`): if `True`, the notebook should start by downloading the data it needs to run
- `location` (`str`): for notebooks that use the Overpass API, defines the location to process
- `output_geojson` (`str`): file path of the generated GeoJSON file

### GeoJSON feature properties

The features of the GeoJSON output file can have special properties that will be interpreted in the frontend.

- `color` (`str`): the color of the feature

### Popup template

The frontend displays a popup when clicking on a GeoJSON feature. The content of the popup can be defined besides the notebook file, in a file called `popup.template` (TODO update the name when the template engine will be chosen).

Available substitution variables:

- every GeoJSON feature property

Example:
Christophe Benz's avatar
Christophe Benz committed
101

102 103
```text
The score is {score}!
Christophe Benz's avatar
Christophe Benz committed
104
```
Christophe Benz's avatar
Christophe Benz committed
105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137

## Architecture

- this repo contains many notebooks
- each notebook comes with its `environment.yml` file containing frozen dependencies for [conda](https://docs.conda.io/en/latest/)

### Conda environments

- conda is used because python geo packages like `osmnx` are not available as a wheel on PyPI
- conda uses channels and we use the [conda-forge channel](https://conda-forge.org/) because the packages we need are published to conda-forge
- caveats:
  - deps are frozen and we don't see the "desired" deps
    - workaround: to declare a new dep `conda install <package_name>` then regenerate `environment.yml` by `conda env export > environment.yml`
  - `prefix` YAML key hard-codes a local path of the dev machine, and it hard-codes also the environement name
    - workaround: the conda env is created first, then `conda env update --file environment.yml` only installs deps

### Local development

Conda must be installed on the dev machine. Pyenv can help installing a specific Python version.

### Pipeline

To compute the notebooks on a large geo area (e.g. PACA) it is required to use a powerful machine.

The process has been modelized in a [Tekton](https://tekton.dev/) pipeline available in the [omi-ops repo](https://gitlab.com/open-mobility-indicators/omi-ops/-/tree/master/k8s/compute-notebooks).

### Limitations

This repo does not make use of container images for now.

As a consequence the environment must be created and deps installed each time the pipeline runs.

With containers, this step would be done only when deps change.