1. 20 Jun, 2020 1 commit
  2. 08 Jun, 2020 2 commits
    • Tjerk Vreeken's avatar
      Style fix for LookupTable class · 3b4ab718
      Tjerk Vreeken authored
      A method in OptimizationProblem was returning a "LookupTable" type, with
      the class not defined anywhere in scope. One of the CI/CD style checkers
      became a bit more aggressive recently and started to catch this as an
      We now properly defined a base LookupTable class, from which the one in
      csv_lookup_table_mixin.py inherits.
    • Tjerk Vreeken's avatar
      Style fix for dubious variable name · 952db03e
      Tjerk Vreeken authored
      One of the CI/CD style checkers became a bit more aggressive recently,
      and started to catch this "l" variable name as an error.
  3. 15 Apr, 2020 10 commits
    • Tjerk Vreeken's avatar
      Simulation/IOMixin: Optimize NaN check on input values · 888d5e56
      Tjerk Vreeken authored
      A call to math's isfinite() is about 20 times faster than calling
      NumPy's isfinite() (~66 ns vs 1.2 us). We also save some attribute
      lookup overhead (20 ns) by importing isfinite() directly, instead of
      calling `math.isfinite()` (the latter takes 85 ns).
    • Tjerk Vreeken's avatar
      SimulationProblem: Optimize set_var by simplifying parameter check · 0b5e90eb
      Tjerk Vreeken authored
      Instead of calling `canonical_state` every time via the use of the
      AliasDict, we can just figure out all parameters and their aliases once.
      That way we can rely on the (very) fast containment check that a native
      Python set() offers.
    • Tjerk Vreeken's avatar
      Simulation/IO: Optimize setting of read variable values · 56bf324e
      Tjerk Vreeken authored
      The IO timeseries do not change in the update loop. Instead of
      repeatedly requesting the timeseries, we can just store the values in an
      internal cache.
    • Tjerk Vreeken's avatar
      SimulationProblem: Optimize copying results to state vector · 952ff491
      Tjerk Vreeken authored
      Going to a NumPy array first is a faster way of achieving the same
    • Tjerk Vreeken's avatar
      Simulation: Use lists to store results · 516cd10f
      Tjerk Vreeken authored
      We do not know the size of the run a priori. Appending to NumPy arrays
      is very slow (~5 us per call), whereas it is very fast for lists (~90
    • Tjerk Vreeken's avatar
      AliasDict: Support storing lists · 945de20c
      Tjerk Vreeken authored
      Most importantly is that AliasDicts need to be able to apply the
      negative operator. A list does not support this operator itself, so we
      have to work around it ourselves.
    • Tjerk Vreeken's avatar
      SimulationProblem: Optimize get_var/set_var calls · 9c98d33c
      Tjerk Vreeken authored
      We can avoid the somewhat expensive `canonical_signed` call. Note that
      this is typically only expensive because we tend to call get_var/set_var
      millions of times.
    • Tjerk Vreeken's avatar
      Simulation: Optimize update/rootfinder call by avoiding vertcat · 6303bd57
      Tjerk Vreeken authored
      Vertcat calls are relatively expensive to do. The vertcat call the
      prepends `dt` was therefore taking a substantial amount of time, equal
      or more than the actual solving of the equations with rootfinder for
      linear models.
      Instead of vertcatting, we can just specify the `dt` as additional
      parameter to the ca.Function that we pass to rootfinder. We can then
      also just pass it separately when calling `__do_step` to do a rootfinder
    • Tjerk Vreeken's avatar
      Simulation/IO: Optimize setting of input variables · ba3ad9cb
      Tjerk Vreeken authored
      The variables in the problem cannot change from update call to update
      call. It therefore makes no sense to make the rather expensive call to
      get all variables time and again. We can save some time by caching the
      return values of these functions.
    • Tjerk Vreeken's avatar
      Storage: Optimize calls to .times_sec property · 90e9f3dc
      Tjerk Vreeken authored
      Calling `self.io.times_sec` would make a copy, to be sure that the user
      could not overwrite the internal array of times. Especially in
      simulation, where we can do many time steps, this copy() call was rather
      expensive. The same intent can be accomplished by disallowing writes to
      the array we return.
      Note that the times in seconds are immutable as soon as they are
      generated (which is usually done when a request is done like
      `get_timeseries_sec`). At that point, they are no longer allowed to
  4. 09 Apr, 2020 5 commits
  5. 08 Mar, 2020 2 commits
    • Tjerk Vreeken's avatar
      Examples: Fix integral call in basic and mixed integer examples · dd548558
      Tjerk Vreeken authored
      The second positional argument to integral() is the time in seconds from
      which to start integrating. It is not the ensemble member, which is
      easier passed as a keyword argument if we want to take the integral over
      the entire optimization horizon.
    • Tjerk Vreeken's avatar
      NetCDF: Fix conversion of imported times to datetimes · 30716146
      Tjerk Vreeken authored
      `NetCDF4` depends on `cftime`, the latter released a version 1.1 which
      changed the behavior of the `num2date` function. It also added a new
      convenience method `num2pydate` which seems to sometimes lose the
      millisecond information.
      We only implicitly depend on `cftime` via `NetCDF4`, but with this
      breaking change on their end we need to make sure that the correct
      version is installed. We therefore add `cftime` to our list of
      (optional) dependencies, and rely on the version we know is correct
      (which excludes 1.1.0).
      See e.g. https://github.com/Unidata/cftime/issues/134 for details on why
      this breaking change was made.
      See https://github.com/Unidata/cftime/issues/144 for the issue on
      num2pydate losing millisecond information.
  6. 12 Feb, 2020 4 commits
    • Tjerk Vreeken's avatar
      Add debug check logging whether constraints are linear/quadratic/other · 37e96f12
      Tjerk Vreeken authored
      It is often useful to make sure that the constraints are linear, such
      that convexity is guaranteed. Depending in the options of for example
      GoalProgrammingMixin, it can be possible that a quadratic objective in
      one priority leads to a quadratic constraint in the next priority, which
      a solver may not be able to solve (or may not even be convex anymore).
      Eventually this check might be rewritten such that it (also) logs
      whether a problem is (MI)LP/QP/QCQP/NLP, and also say something about
      the convexity.
    • Tjerk Vreeken's avatar
      Add debug check for linear independence of constraints · 58fe2da1
      Tjerk Vreeken authored
      Interior point solvers need the linear independence constraint
      qualification (LICQ) to be satisfied. Because a solver like IPOPT is
      callback based, it cannot detect even the simplest of dependencies that
      violate this qualification.
      This commit adds some checks on typical linear dependencies that arise.
      Currently these checks consist of:
      1. Duplicate constraints that set a variable to a certain value.
         For example the constraint `x = 1` twice.
      2. Setting a equality constraint on a variable that has equal bounds.
         For example having a constraint `x = 1` when x has bounds [1, 1].
    • Tjerk Vreeken's avatar
      Add debug check on jacobian coefficients at x0 · 49626120
      Tjerk Vreeken authored
      Many solvers (IPOPT, CPLEX, etc) would prefer the coefficients in the
      Jacobian to be in the order of [1E-2, 1E2], and also for the range of
      coefficients for a certain variable or in a certain constraint to be
      rather limited (say 1E3). The check added in this commit gives feedback
      to the user on what constraints/variables do not meet these targets.
      Note that not satisfying these limits/ranges does not mean that the
      problem will not solve or even be slower, because solvers tend to be
      quite smart in how they themselves rescale equations or evade numerical
      instabilities. That said, the larger (more variables, more constraints)
      a problem gets, the more sensitive solvers tend to become to bad
    • Tjerk Vreeken's avatar
      Add flag to run optional debug checks (private API) · 8a9b27b6
      Tjerk Vreeken authored
      When setting up or debugging a model it can be useful to do typical
      checks to see if you made a mistake. These checks can be rather
      computationally expensive, which is why it should be easy to disable
      them when running operationally. A (for now private) flag
      `_debug_check_level` is added to OptimizationProblem/SimulationProblem
      that can be used to enable/disable certain checks.
      Foreseeing quite a few of these checks, it should be easy to filter
      checks based on verbosity and how expensive they are to perform. A
      typical way of doing this is with an enum, just like logging levels
      (DEBUG, INFO, WARNING, etc). To this end the value of
      `_debug_check_level` can be set to the value of a `DebugLevel` enum.
      This enum has with 5 levels, with room for future refinement. The
      default check level is in the middle (MEDIUM), with two available below
      (NONE, LOW) and two above (HIGH, VERYHIGH).
      With more of these checks, and considering how expensive and useful they
      might be for a certain type of issue, it is also possible to filter more
      precise by setting the value of `_debug_check_level` to a function. That
      way one can filter a specific test on its level and fully qualified
      For some checks it might make sense to accept some arguments. For
      example, a certain tolerance value above which messages should be
      logged. For this purpose the `_debug_check_options` class variable of
      OptimizationProblem/SimulationProblem can be used. This is a dictionary
      that maps a fully specified check name to (a dictionary of) additional
      keyword arguments.
      Note that it is likely that some details of these debug checks will
      change, which is why it is only part of the private API for now.
      Closes #1042
  7. 29 Jan, 2020 1 commit
  8. 28 Jan, 2020 1 commit
    • Teresa Piovesan's avatar
      CSVMixin: Allow naming of input files · bb872625
      Teresa Piovesan authored
      Allow users to specify the name of the input files used to extract that
      initial states, parameters and ensemble information. Note that this was
      already possible when using PIMixin.
  9. 20 Jan, 2020 1 commit
    • Tjerk Vreeken's avatar
      csv: Fix reading empty columns as float · 4b770409
      Tjerk Vreeken authored
      We use np.genfromtxt with dtype=None, which means that it will guess. If
      a column does not have any value specified, it will guess the dtype
      boolean with values `False`. This would eventually be turned into a
      Timeseries of `0.0`. We however want a Timeseries full of `np.nan`.
      It is difficult to make np.genfromtxt read in floats, when the first
      column can sometimes be a string. Therefore, we detect the case where
      the dtype of a column is boolean, and force a conversion to float.
      Basically, we therefore only support reading in floats (and a optionally
      string for the first column).
      Note that CSVMixin (both in optimization and simulation) forces a
      conversion to float already anyway, so there is no freedom lost.
      Closes #1128
  10. 10 Jan, 2020 1 commit
  11. 14 Nov, 2019 1 commit
  12. 07 Nov, 2019 1 commit
    • Teresa Piovesan's avatar
      HomotopyMixin: Fix overwriting of seed · c138a581
      Teresa Piovesan authored
      It is necessary to overwrite the seed within the HomotopyMixin class
      whenever the results of the latest optimization run are stored here.
      Note that a necessary requirement for this is that theta must be greater
      than zero. Moreover, there are only two situations in which we need to
      overwrite the seed: when the class GoalProgrammingMixin is not used, or
      whenever it is the first run of the goal programming loop. To detect
      this, and to make it clear that we internally rely on this variable, we
      rename __first_run to _gp_first_run.
  13. 06 Nov, 2019 4 commits
    • Tjerk Vreeken's avatar
      ModelicaMixin: Fix skipping of non-specified start attribute · 7b20bb4b
      Tjerk Vreeken authored
      If the start attribute is the default of zero, we assume that it has not
      been explicitly set. The type of the start attribute can however be more
      than just a float; it can also be a bool, int (depending on the Python
      type of the variable). We reformulate the check such that it catches all
      these cases.
      Note that the default seed value is zero anyway in CollInt, so in
      practice the result will often be the same. The only difference is that,
      previously, failing check on zero (e.g. because the start attribute was
      in int) would _explicitly_ set the seed to zero. This could trigger code
      that checks if a seed for a certain variable has been specified.
    • Teresa Piovesan's avatar
      CascChanExample: Fix degeneracy of Jacobian matrix · 01a88d74
      Teresa Piovesan authored
      In the cascading channel example, one of the water level initial
      conditions was overdetermined, which lead to a degenerate Jacobian
      matrix. We remove one of these initial conditions.
      Closes #1099
    • Tjerk Vreeken's avatar
      Tests/CSVLookupTableMixin: Do not generate plots · 7aa647e2
      Tjerk Vreeken authored
      The generation of plots requires matplotlib, which is not an explicit
      dependency of our test environment. Instead of adding matplotlib to our
      test dependencies, we can just disable the generation of plots, as it
      does not add anything to the tests.
      Note that the generation of plots might make sense when debugging why
      the test is failing, but that is a different use case for which the user
      can easily just install matplotlib.
    • Tjerk Vreeken's avatar
      ModelicaMixin: Pass expand_mx to Pymoca for eliminable expressions · 7f70632e
      Tjerk Vreeken authored
      Pymoca requires (as a workaround for a CasADi shortcoming) that
      `expand_mx` is set when using `eliminable_variable_expression`.
  14. 31 Oct, 2019 2 commits
    • Tjerk Vreeken's avatar
      SinglePassGPMixin: Fix priority of applied constraint_relaxation · f79664d3
      Tjerk Vreeken authored
      The behavior in GoalProgrammingMixin with keep_soft_constraints=True is
      that the constraint_relaxation of priority 2 is applied to the objective
      of priority 2. This is because we call `goal_programming_options` in
      GoalProgrammingMixin at the end of priority 2 to make this constraint.
      In SinglePassGoalProgrammingMixin, we would get the
      constraint_relaxation by calling `goal_programming_options` at the start
      of the next priority, so the constraint_relaxation of priority 3 would
      be applied to the objective of priority 2. This is not necessarily
      wrong, but does differ from GoalProgrammingMixin. We therefore fix the
      discrepancy by making SinglePassGoalProgrammingMixin behave equal in
      this sense to GoalProgrammingMixin; the constraint relaxation at
      priority 2 is applied to the objective of priority 2.
    • Tjerk Vreeken's avatar
  15. 29 Oct, 2019 2 commits
    • Tjerk Vreeken's avatar
      SimulationProblem: Optimize state lookup · d0d21987
      Tjerk Vreeken authored
      The naive implementation until this commit was O(N), which means that
      state lookup could take up a significant portion of the computational
      time when there were many variables/parameters. We implement a O(1)
      state index lookup for SimulationProblem using a dictionary, just like
      fb6bbf08 did for optimization's CollocatedIntegratedOptimizationProblem.
      Closes #1052
    • Tjerk Vreeken's avatar
      pi.py: Optimize validation of series timesteps · 4af0ea86
      Tjerk Vreeken authored
      If there are many timeseries and/or a lot of reference times, we can
      save quite some time by converting only the times of the current series
      to a set. Imagine a case where a lot of timeseries are only specified at
      t0 (for the initial state) or are empty, but the total length of all
      times is a series of length 100,000+.
  16. 24 Oct, 2019 1 commit
    • Tjerk Vreeken's avatar
      Simulation/PIMixin: Allow specifying ensemble member to read · b6d5f665
      Tjerk Vreeken authored
      Although simulation mode only makes sense for a single ensemble member,
      PI-XML files can contain multiple ensemble members. PIMixin in
      simulation would always read the first ensemble member before this
      commit. Instead of forcing the user to write a separate file for each
      ensemble member, we now allow a new attribute `pi_ensemble_member` that
      specifies what ensemble member should be read from the PI-XML file.
  17. 18 Oct, 2019 1 commit
    • Teresa Piovesan's avatar
      GPMixin: Fix mu_reinit option · 8b0be114
      Teresa Piovesan authored
      Setting the goal programming option 'mu_reinit' to False would trigger a
      CasADi RuntimeError, as the `gather_stats` option no longer exists.
      CasADi instead just always collects stats, so we can simply remove the
      `gather_stats` option.
      Note that the `gather_stats` option was deprecated by CasADi about 4
      years ago at time of writing this commit.