Commit d405bc6b authored by Jorn Baayen's avatar Jorn Baayen

Initial public import of RTC2 Python code.

parents
Jorn Baayen (Deltares)
Matthijs den Toom (Deltares)
Olav van Duin (Deltares)
Tjerk Vreeken (Deltares)
Jesse VanderWees (University of British Columbia)
Alja Vrieling (Vortech)
\ No newline at end of file
This diff is collapsed.
2.0
===
* First release in 2.x series.
\ No newline at end of file
This is RTC-Tools 2.0, a toolbox for control and optimization of environmental systems.
Visit our website at:
https://www.deltares.nl/en/software/rtc-tools/
\ No newline at end of file
# Makefile for Sphinx documentation
#
# You can set these variables from the command line.
SPHINXOPTS =
SPHINXBUILD = sphinx-build
PAPER =
BUILDDIR = _build
# Internal variables.
PAPEROPT_a4 = -D latex_paper_size=a4
PAPEROPT_letter = -D latex_paper_size=letter
ALLSPHINXOPTS = -d $(BUILDDIR)/doctrees $(PAPEROPT_$(PAPER)) $(SPHINXOPTS) .
# the i18n builder cannot share the environment and doctrees with the others
I18NSPHINXOPTS = $(PAPEROPT_$(PAPER)) $(SPHINXOPTS) .
.PHONY: help
help:
@echo "Please use \`make <target>' where <target> is one of"
@echo " html to make standalone HTML files"
@echo " dirhtml to make HTML files named index.html in directories"
@echo " singlehtml to make a single large HTML file"
@echo " pickle to make pickle files"
@echo " json to make JSON files"
@echo " htmlhelp to make HTML files and a HTML help project"
@echo " qthelp to make HTML files and a qthelp project"
@echo " applehelp to make an Apple Help Book"
@echo " devhelp to make HTML files and a Devhelp project"
@echo " epub to make an epub"
@echo " epub3 to make an epub3"
@echo " latex to make LaTeX files, you can set PAPER=a4 or PAPER=letter"
@echo " latexpdf to make LaTeX files and run them through pdflatex"
@echo " latexpdfja to make LaTeX files and run them through platex/dvipdfmx"
@echo " text to make text files"
@echo " man to make manual pages"
@echo " texinfo to make Texinfo files"
@echo " info to make Texinfo files and run them through makeinfo"
@echo " gettext to make PO message catalogs"
@echo " changes to make an overview of all changed/added/deprecated items"
@echo " xml to make Docutils-native XML files"
@echo " pseudoxml to make pseudoxml-XML files for display purposes"
@echo " linkcheck to check all external links for integrity"
@echo " doctest to run all doctests embedded in the documentation (if enabled)"
@echo " coverage to run coverage check of the documentation (if enabled)"
@echo " dummy to check syntax errors of document sources"
.PHONY: clean
clean:
rm -rf $(BUILDDIR)/*
.PHONY: html
html:
$(SPHINXBUILD) -b html $(ALLSPHINXOPTS) $(BUILDDIR)/html
@echo
@echo "Build finished. The HTML pages are in $(BUILDDIR)/html."
.PHONY: dirhtml
dirhtml:
$(SPHINXBUILD) -b dirhtml $(ALLSPHINXOPTS) $(BUILDDIR)/dirhtml
@echo
@echo "Build finished. The HTML pages are in $(BUILDDIR)/dirhtml."
.PHONY: singlehtml
singlehtml:
$(SPHINXBUILD) -b singlehtml $(ALLSPHINXOPTS) $(BUILDDIR)/singlehtml
@echo
@echo "Build finished. The HTML page is in $(BUILDDIR)/singlehtml."
.PHONY: pickle
pickle:
$(SPHINXBUILD) -b pickle $(ALLSPHINXOPTS) $(BUILDDIR)/pickle
@echo
@echo "Build finished; now you can process the pickle files."
.PHONY: json
json:
$(SPHINXBUILD) -b json $(ALLSPHINXOPTS) $(BUILDDIR)/json
@echo
@echo "Build finished; now you can process the JSON files."
.PHONY: htmlhelp
htmlhelp:
$(SPHINXBUILD) -b htmlhelp $(ALLSPHINXOPTS) $(BUILDDIR)/htmlhelp
@echo
@echo "Build finished; now you can run HTML Help Workshop with the" \
".hhp project file in $(BUILDDIR)/htmlhelp."
.PHONY: qthelp
qthelp:
$(SPHINXBUILD) -b qthelp $(ALLSPHINXOPTS) $(BUILDDIR)/qthelp
@echo
@echo "Build finished; now you can run "qcollectiongenerator" with the" \
".qhcp project file in $(BUILDDIR)/qthelp, like this:"
@echo "# qcollectiongenerator $(BUILDDIR)/qthelp/RTC-Tools.qhcp"
@echo "To view the help file:"
@echo "# assistant -collectionFile $(BUILDDIR)/qthelp/RTC-Tools.qhc"
.PHONY: applehelp
applehelp:
$(SPHINXBUILD) -b applehelp $(ALLSPHINXOPTS) $(BUILDDIR)/applehelp
@echo
@echo "Build finished. The help book is in $(BUILDDIR)/applehelp."
@echo "N.B. You won't be able to view it unless you put it in" \
"~/Library/Documentation/Help or install it in your application" \
"bundle."
.PHONY: devhelp
devhelp:
$(SPHINXBUILD) -b devhelp $(ALLSPHINXOPTS) $(BUILDDIR)/devhelp
@echo
@echo "Build finished."
@echo "To view the help file:"
@echo "# mkdir -p $$HOME/.local/share/devhelp/RTC-Tools"
@echo "# ln -s $(BUILDDIR)/devhelp $$HOME/.local/share/devhelp/RTC-Tools"
@echo "# devhelp"
.PHONY: epub
epub:
$(SPHINXBUILD) -b epub $(ALLSPHINXOPTS) $(BUILDDIR)/epub
@echo
@echo "Build finished. The epub file is in $(BUILDDIR)/epub."
.PHONY: epub3
epub3:
$(SPHINXBUILD) -b epub3 $(ALLSPHINXOPTS) $(BUILDDIR)/epub3
@echo
@echo "Build finished. The epub3 file is in $(BUILDDIR)/epub3."
.PHONY: latex
latex:
$(SPHINXBUILD) -b latex $(ALLSPHINXOPTS) $(BUILDDIR)/latex
@echo
@echo "Build finished; the LaTeX files are in $(BUILDDIR)/latex."
@echo "Run \`make' in that directory to run these through (pdf)latex" \
"(use \`make latexpdf' here to do that automatically)."
.PHONY: latexpdf
latexpdf:
$(SPHINXBUILD) -b latex $(ALLSPHINXOPTS) $(BUILDDIR)/latex
@echo "Running LaTeX files through pdflatex..."
$(MAKE) -C $(BUILDDIR)/latex all-pdf
@echo "pdflatex finished; the PDF files are in $(BUILDDIR)/latex."
.PHONY: latexpdfja
latexpdfja:
$(SPHINXBUILD) -b latex $(ALLSPHINXOPTS) $(BUILDDIR)/latex
@echo "Running LaTeX files through platex and dvipdfmx..."
$(MAKE) -C $(BUILDDIR)/latex all-pdf-ja
@echo "pdflatex finished; the PDF files are in $(BUILDDIR)/latex."
.PHONY: text
text:
$(SPHINXBUILD) -b text $(ALLSPHINXOPTS) $(BUILDDIR)/text
@echo
@echo "Build finished. The text files are in $(BUILDDIR)/text."
.PHONY: man
man:
$(SPHINXBUILD) -b man $(ALLSPHINXOPTS) $(BUILDDIR)/man
@echo
@echo "Build finished. The manual pages are in $(BUILDDIR)/man."
.PHONY: texinfo
texinfo:
$(SPHINXBUILD) -b texinfo $(ALLSPHINXOPTS) $(BUILDDIR)/texinfo
@echo
@echo "Build finished. The Texinfo files are in $(BUILDDIR)/texinfo."
@echo "Run \`make' in that directory to run these through makeinfo" \
"(use \`make info' here to do that automatically)."
.PHONY: info
info:
$(SPHINXBUILD) -b texinfo $(ALLSPHINXOPTS) $(BUILDDIR)/texinfo
@echo "Running Texinfo files through makeinfo..."
make -C $(BUILDDIR)/texinfo info
@echo "makeinfo finished; the Info files are in $(BUILDDIR)/texinfo."
.PHONY: gettext
gettext:
$(SPHINXBUILD) -b gettext $(I18NSPHINXOPTS) $(BUILDDIR)/locale
@echo
@echo "Build finished. The message catalogs are in $(BUILDDIR)/locale."
.PHONY: changes
changes:
$(SPHINXBUILD) -b changes $(ALLSPHINXOPTS) $(BUILDDIR)/changes
@echo
@echo "The overview file is in $(BUILDDIR)/changes."
.PHONY: linkcheck
linkcheck:
$(SPHINXBUILD) -b linkcheck $(ALLSPHINXOPTS) $(BUILDDIR)/linkcheck
@echo
@echo "Link check complete; look for any errors in the above output " \
"or in $(BUILDDIR)/linkcheck/output.txt."
.PHONY: doctest
doctest:
$(SPHINXBUILD) -b doctest $(ALLSPHINXOPTS) $(BUILDDIR)/doctest
@echo "Testing of doctests in the sources finished, look at the " \
"results in $(BUILDDIR)/doctest/output.txt."
.PHONY: coverage
coverage:
$(SPHINXBUILD) -b coverage $(ALLSPHINXOPTS) $(BUILDDIR)/coverage
@echo "Testing of coverage in the sources finished, look at the " \
"results in $(BUILDDIR)/coverage/python.txt."
.PHONY: xml
xml:
$(SPHINXBUILD) -b xml $(ALLSPHINXOPTS) $(BUILDDIR)/xml
@echo
@echo "Build finished. The XML files are in $(BUILDDIR)/xml."
.PHONY: pseudoxml
pseudoxml:
$(SPHINXBUILD) -b pseudoxml $(ALLSPHINXOPTS) $(BUILDDIR)/pseudoxml
@echo
@echo "Build finished. The pseudo-XML files are in $(BUILDDIR)/pseudoxml."
.PHONY: dummy
dummy:
$(SPHINXBUILD) -b dummy $(ALLSPHINXOPTS) $(BUILDDIR)/dummy
@echo
@echo "Build finished. Dummy builder generates no files."
Basics
======
.. autoclass:: rtctools.optimization.timeseries.Timeseries
:members: __init__, times, values
:show-inheritance:
.. autoclass:: rtctools.optimization.optimization_problem.OptimizationProblem
:members: bounds, constant_inputs, constraints, control, control_at, delayed_feedback, der, ensemble_member_probability, ensemble_size, get_timeseries, history, initial_time, integral, interpolate, lookup_tables, objective, optimize, parameters, path_constraints, post, pre, seed, set_timeseries, solver_options, state, state_at, states_in, timeseries_at
:show-inheritance:
.. autofunction:: rtctools.util.run_optimization_problem
\ No newline at end of file
This diff is collapsed.
CSV I/O
=======
.. autoclass:: rtctools.optimization.csv_mixin.CSVMixin
:members: min_timeseries_id, max_timeseries_id
:show-inheritance:
\ No newline at end of file
Filling a Reservoir
~~~~~~~~~~~~~~~~~~~
.. insert image here? e.g.
https://commons.wikimedia.org/wiki/File:Badgernet_Clywedog_reservoir_3.JPG
Overview
--------
The purpose of this example is to understand the technical setup of
an RTC-Tools model, how to run the model, and how to interpret the results.
The scenario is the following: A reservoir operator is trying to fill a
reservoir. They are given a six-day forcast of inflows given in 12-hour
increments. The operator wants to save as much of the inflows as possible, but
does not want to end up with too much water at the end of the six days. They
have chosen to use RTC-Tools to calculate how much water to release and when
to release it.
The folder ``<installation directory>\RTCTools2\examples\example_basic``
contains a complete RTC-Tools optimization problem. An RTC-Tools
directory has the following structure:
* ``input``: This folder contains the model input data. These are several files
in comma separated value format, ``csv``.
* ``model``: This folder contains the Modelica model. The Modelica model
contains the physics of the RTC-Tools model.
* ``output``: The folder where the output is saved in the file
``timeseries_export.csv``.
* ``src``: This folder contains a Python file. This file contains the
configuration of the model and is used to run the model .
The Model
---------
The first step is to develop a physical model of the system.
The model can be viewed and editied using the OpenModelica Connection Editor
program. Make sure to load the Deltares libray before loading the example:
#. Load the Deltares library into OpenModelica Conection Editor
* Using the menu bar: *File -> Open Model/Library File(s)*
* Select ``<installation directory>\RTCTools2\mo\Deltares\package.mo``
#. Load the example model into OpenModelica Conection Editor
* Using the menu bar: *File -> Open Model/Library File(s)*
* Select ``<installation
directory>\RTCTools2\examples\example_basic\model\Example.mo``
Once loaded, we have an OpenModelica Connection Editor window that looks like
this:
.. image:: images/simple_storage_openmodelica.png
The model ``Example.mo`` represents a simple system with the following
elements:
* a reservoir, modelled as storage element
``Deltares.ChannelFlow.SimpleRouting.Storage.Storage``,
* an inflow boundary condition
``Deltares.ChannelFlow.SimpleRouting.BoundaryConditions.Inflow``,
* an outfall boundary condition
``Deltares.ChannelFlow.SimpleRouting.BoundaryConditions.Terminal``,
* connectors (black lines) connecting the elements.
You can use the mouse-over feature help to identify the pre-defined models from
the Deltares library. You can also drag the elements around- the connectors will
move with the elements. Adding new elements is easy- just drag them in from the
Deltares Library on the sidebar. Connecting the elements is just as easy- click
and drag between the ports on the elements.
In text mode, the Modelica model looks as follows (with
annotation statements removed):
.. literalinclude:: _build/mo/example_basic.mo
:language: modelica
:lineno-match:
The three water system elements (storage, inflow, and outfall) appear under
the ``model Example`` statement. The ``equation`` part connects these three
elements with the help of connections. Note that ``storage`` extends the partial
model ``QSISO`` which contains the connectors ``QIn`` and ``QOut``.
With ``QSISO``, ``storage`` can be connected on two sides. The ``storage``
element also has a variable ``Q_release``, which is the descision variable the
operator controls.
OpenModelica Connection Editor will automatically generate the element and
connector entries in the text text file. Defining inputs and outputs requires
editing the text file directly. Relationships between the inputs and outputs and
the library elements must also be defined in the ``equation`` section.
In addition to elements, the input variables ``Q_in`` and ``Q_release`` are also
defined. ``Q_in`` is determined by the forecast and the operator cannot contol
it, so we set ``Q_in(fixed = true)``. The actual values of ``Q_in`` are stored
in ``timeseries_input.csv``. In the ``equation`` section, equations are defined
to relate the inputs to the appropriate water system elements.
Because we want to view the water volume in the storage element in the output
file, we also define an output variable ``V_storage``.
The Optimization Problem
------------------------
The python script is created and edited in a text editor. In general, the python
script consists of the following blocks:
* Import of packages
* Definition of the optimization problem class
* Constructor
* Objective function
* Definition of constraints
* Any additional configuration
* A run statement
Importing Packages
''''''''''''''''''
Packages are imported using ``from ... import ...`` at the top of the file. In
our script, we import the classes we want the class to inherit, the
package ``run_optimization_problem`` form the ``rtctools.util`` package, and
any extra packages we want to use. For this example, the import block looks
like:
.. literalinclude:: ../../case-studies/example_basic/src/example.py
:language: python
:lines: 1-5
:lineno-match:
Optimization Problem
''''''''''''''''''''
The next step is to define the optimization problem class. We construct the
class by declaring the class and inheriting the desired parent classes. The
parent classes each perform different tasks realted to importing and exporting
data and solving the optimization problem. Each imported class makes a set of
methods available to the our optimization class.
.. literalinclude:: ../../case-studies/example_basic/src/example.py
:language: python
:pyobject: Example
:lineno-match:
:end-before: """
The next, we define an objective function. This is a class method that returns
the value that needs to be minimized.
.. literalinclude:: ../../case-studies/example_basic/src/example.py
:language: python
:pyobject: Example.objective
:lineno-match:
Constraints can be declared by declairing the ``path_constraints`` method. Path
constraints are constraints that are applied every timestep. To set a constraint
at an individual timestep, we could define it inside the ``constraints`` method.
Other parent classes also declare this method, so we call the ``super()`` method
so that we don't overwrite their behaviour.
.. literalinclude:: ../../case-studies/example_basic/src/example.py
:language: python
:pyobject: Example.path_constraints
:lineno-match:
Run the Optimization Problem
''''''''''''''''''''''''''''
To make our script run, at the bottom of our file we just have to call
the ``run_optimization_problem()`` method we imported on the optimization
problem class we just created.
.. literalinclude:: ../../case-studies/example_basic/src/example.py
:language: python
:lineno-match:
:start-after: # Run
The Whole Script
''''''''''''''''
All together, the whole example script is as follows:
.. literalinclude:: ../../case-studies/example_basic/src/example.py
:language: python
:lineno-match:
Running RTC-Tools
---------------------
TODO:
* An explaination of how to run RTC-Tools goes here.
* So does a description of the terminal output
Extracting Results
------------------
The results from the run are found in ``output/timeseries_export.csv``. Any
CSV-reading software can import it, but this is what the results look like when
plotted in Microsoft Excel:
.. image:: images/basic_example_resultplot.png
This plot shows that the operator is able to keep the water level within the
bounds over the entire time horizon and end with a full reservoir.
Feel free to experiment with this example. See what happens if you change the
max of ``Q_release`` (in the modelica file) or if you make the objective
function negative (in the python script).
Using Forecast Ensembles
~~~~~~~~~~~~~~~~~~~~~~~~
.. note::
This example is an extension of :doc:`example_lookuptable_storage`. It
assumes prior knowledge of goal programming and the lookuptables components
of RTC-Tools. If you are a first-time user of RTC-Tools, see
:doc:`example_basic`.
Then biggest change to RTC-Tools when using ensembles is the structure of the
directory. The folder ``<installation directory>\RTCTools2\examples\example_ensemble``
contains a complete RTC-Tools ensemble optimization problem. An RTC-Tools
ensemble directory has the following structure:
* ``model``: This folder contains the Modelica model. The Modelica model
contains the physics of the RTC-Tools model.
* ``src``: This folder contains a Python file. This file contains the
configuration of the model and is used to run the model.
* ``input``: This folder contains the model input data pertaining to each
ensemble member:
* ``ensemble.csv``: a file where the names and probabilities of the ensembles
are defined
* ``forcast1``
* the file ``timeseries_import.csv``
* the file ``initial_state.csv``
* ``forcast2``
* ``timeseries_import.csv``
* ``initial_state.csv``
* ...
* ``output``: The folder where the output is saved:
* ``forcast1``
* ``timeseries_export.csv``
* ``forcast2``
* ``timeseries_export.csv``
* ...
The Model
---------
.. note::
This example uses the same hydraulic model as the basic example. For a
detalied explaination of the hydraulic model, see :doc:`example_basic`.
In OpenModelica Connection Editor, the model looks like this:
.. image:: images/simple_storage_openmodelica.png
In text mode, the Modelica model is as follows (with annotation statements
removed):
.. literalinclude:: _build/mo/example_ensemble.mo
:language: modelica
:lineno-match:
The Optimization Problem
------------------------
The python script consists of the following blocks:
* Import of packages
* Declaration of Goals
* Declaration of the optimization problem class
* Constructor
* Set ``csv_ensemble_mode = True``
* Declaration of a ``pre()`` method
* Specification of Goals
* Declaration of a ``priority_completed()`` method
* Declaration of a ``post()`` method
* Additional configuration of the solver
* A run statement
Declaring Goals
'''''''''''''''
First, we have a high priority goal to keep the water volume within a minimum
and maximum.
.. literalinclude:: ../../case-studies/example_ensemble/src/example.py
:language: python
:pyobject: WaterVolumeRangeGoal
:lineno-match:
We also want to save energy, so we define a goal to minimize ``Q_release``. This
goal has a lower priority.
.. literalinclude:: ../../case-studies/example_ensemble/src/example.py
:language: python
:pyobject: MinimizeQreleaseGoal
:lineno-match:
Importing Packages
''''''''''''''''''
For this example, the import block is as follows:
.. literalinclude:: ../../case-studies/example_ensemble/src/example.py
:language: python
:lines: 1-11
:lineno-match:
Optimization Problem
''''''''''''''''''''
Next, we construct the class by declaring it and inheriting the desired parent
classes.
.. literalinclude:: ../../case-studies/example_ensemble/src/example.py
:language: python
:pyobject: Example
:lineno-match:
:end-before: """
We turn on ensemble mode by setting ``csv_ensemble_mode = True``:
.. literalinclude:: ../../case-studies/example_ensemble/src/example.py
:language: python
:lines: 50-51
:lineno-match:
The method ``pre()`` is already defined in RTC-Tools, but we would like to add
a line to it to create a variable for storing intermediate results. To do this,
we declare a new ``pre()`` method, call ``super(Example, self).pre()`` to ensure
that the original method runs unmodified, and add in a variable declaration to
store our list of intermediate results. This variable is a dict, reflecting the
need to store results from multiple ensembles.
Because the timeseries we set will be the same for both ensemble members, we
also make sure that the timeseries we set are set for both ensemble members
using for loops.
.. literalinclude:: ../../case-studies/example_ensemble/src/example.py
:language: python
:pyobject: Example.pre
:lineno-match:
Now we pass in the goals:
.. literalinclude:: ../../case-studies/example_ensemble/src/example.py
:language: python
:pyobject: Example.path_goals
:lineno-match:
We define the ``priority_completed()`` method. We ensure that it stores the
results from both ensemble members.
.. literalinclude:: ../../case-studies/example_ensemble/src/example.py
:language: python
:pyobject: Example.priority_completed
:lineno-match:
We output our intermediate results using the ``post()`` method:
.. literalinclude:: ../../case-studies/example_ensemble/src/example.py
:language: python
:pyobject: Example.post
:lineno-match:
Finally, we want to apply some additional configuration, reducing the amount of
information the solver outputs:
.. literalinclude:: ../../case-studies/example_ensemble/src/example.py
:language: python
:pyobject: Example.solver_options
:lineno-match:
Run the Optimization Problem
''''''''''''''''''''''''''''
To make our script run, at the bottom of our file we just have to call
the ``run_optimization_problem()`` method we imported on the optimization
problem class we just created.
.. literalinclude:: ../../case-studies/example_ensemble/src/example.py
:language: python
:lineno-match:
:start-after: # Run
The Whole Script
''''''''''''''''
All together, the whole example script is as follows:
.. literalinclude:: ../../case-studies/example_ensemble/src/example.py
:language: python
:lineno-match:
Running the Optimization Problem
--------------------------------
Following the execution of the optimization problem, the ``post()`` method
should print out the following lines::
Results for Ensemble Member 0:
After finishing goals of priority 1:
Level goal satisfied at 9 of 12 time steps
Integral of Q_release = 45.59
After finishing goals of priority 2:
Level goal satisfied at 9 of 12 time steps