S

StOpt

Library for stochastic control

The STochastic OPTimization library (StOpt) aims at providing tools in C++ for solving some
stochastic optimization problems encountered in finance or in the industry.
A python binding is available for some C++ objects provided permitting to easily solve an optimization problem by regression.
Different methods are available :
- dynamic programming methods based on Monte Carlo  with regressions (global, local and  sparse regressors), for underlying states following
  some uncontrolled Stochastic Differential Equations (python binding provided).
- Semi-Lagrangian methods for Hamilton Jacobi Bellman general equations for underlying states following some controlled  Stochastic Differential Equations (C++ only)
- Stochastic Dual Dynamic Programming methods to deal with stochastic stocks management problems in high dimension. A SDDP module in python is provided.
  To use this module, the transitional optimization problem has to written in C++ and  mapped to python (examples provided).
Besides somes methods are provided to solve by Monte Carlo some problems where the underlying stochastic state is controlled.
For each method, a framework is provided to optimize the problem and then simulate  it out of the sample using the optimal commands previously calculated.
Parallelization methods based on OpenMP and MPI are provided in this framework permitting to solve high dimensional problems on clusters.
The library should be flexible enough to be used at different levels depending on the user's willingness.

#######################################################################################################
GENERAL DOCUMENTATION AVAILABLE AT :

https://hal.archives-ouvertes.fr/hal-01361291

#######################################################################################################

########################################################################################################
Licence : The library is published under the GNU LESSER GENERAL PUBLIC LICENSE
(see LICENCE.txt , COPYING, COPYING.LESSER files)

########################################################################################################

########################################################################################################
The library can be built on Linux/Mac/Windows using some recent compilers. It has been tested with
- gcc, clang, intel icc on linux,
- clang on mac
- visual studio  on windows
########################################################################################################

##################
## Prerequisite ##
##################
StOpt library uses some external library.
Two are included  in the package   so do no need any installation :
- geners library  for generic serialization for C++ : http://geners.hepforge.org/
- boost numpy  used to expose easily c++ array to python numpy : https://github.com/ndarray/Boost.NumPy
The following libraries need to be installed :
- zlib, (geners serialization)
- bzip2, (geners serialization)
- eigen library  version 3 : http://http://eigen.tuxfamily.org/ (choose version 3.2.4 for windows)
- boost library : http://www.boost.org/ , version 1.55 is enough
- python  ( 2.7 , 3.5 tested) with numpy package. Scipy package needed for test cases. Mpi4py package needed for test with mpi. 
Other libraries are optional :
- mpi  http://www.open-mpi.org/   for MPI calculation
- COIN OSI, COIN CLP, COIN UTILS  only for SDDP test  (http://www.coin-or.org/)

StOpt relies on cmake for compilation http://www.cmake.org/ and permits to generate projects (Kdevelopp or Visual Studio for example).

###############
## Important ##
###############

StOpt uses multithread : so OMP_NUM_THREADS environnement variable has to be set to a number of threads to use.
Python test case are located in StOpt/test/python
To use python the PYTHONPATH should be updated to include the  directory containing the  StOpt (.so, .pyd)  library.

####################################
## INSTALLATION, TESTS  LINUX/MAC ##
####################################
####################################

Install package for zlib, bzip2, python, scipy (http://www.scipy.org/),  eigen,  boost (including boost python, boost mpi if necessary), cmake
and cmake gui (ccmake)
Go to StOpt directory (containing directories boostpython, geners-1.10.0, StOpt ...)

   $ cd ..
   $ mkdir BUILD
   $ cd BUILD
   $ cmake ../StOpt
   $ cmake ../StOpt

(Rerun cmake so that the python libs can be found..)
If packages for libraries are not in usual place, use ccmake (ccmake ../StOpt) to defines include directories and libraries needed.
In cmake gui define :
- if  python bindings should be compiled (default no)
- if the library should use mpi (default true)
- if the library should compile sddp test (default false (COIN CLP needed))
This gui can be called by

   $ ccmake ../StOpt
   
Then :

   $ make
   
Test cases are run by:

   $ ctest
   
A package can be build using cpack :
- for debian :
  cpack -G DEB
- for Red Hat :
  cpack -G RPM
-  a GZIP compression :
  cpack -G TGZ

Installation in /usr/local

   $ make install
   
For installation in given directory /home/toto/test  (installation will be achieved in /home/toto/test/usr/local)

   $ make DESTDIR=/home/toto/test install
   
It is also possible to have installation in /home/toto/test such that the directory tree looks like /home/toto/test/lib, /home/toto/test/include...

   $ cmake -DCMAKE_INSTALL_PREFIX=/home/toto/test ../StOpt
   $ make install

##########################################################################
##    Python test                                                        #
##########################################################################
To run the python tests  if library compiled with python (don't forget to update PYTHONPATH in your .bashrc):

   $ cd ../StOpt/test/python
   $ python -m unittest discover

In StOpt/test/python/functional  python version of C++ test cases are available.
Two kind of mappings are available.
- A first mapping is achieved at the grids and conditional expectation level.
- A second one is achieved at a time step resolution level (file *HighLevel*.py)
In order to test python mpi  :

   $ mpirun -np x python test*HighLevelMpi.py
   
with "x" number of mpi processus generated.

# special case for Mac User #
The boost instalaltion can be achieved with a package manager : homebrew for exemple will install the missing dependencies (openmpi ...)
In case of a boost compîlation from source, check that "boost mpi" is compiled and installed. If not : In the directory where the boost file  is decompressed,

    $ ./boostrap.sh
Edit a file user-config.jam and add (dont' forget the space between mpi and ;)
"
using mpi ;
"./bjam --user-config=user-config.jam install

##################################
## INSTALLATION, TESTS WINDOWS :##
##################################
##################################
# Minimal Requirement

- Win 7
- Visual Studio 2013 (because of C++11 use)

Step-by-step procedure :

We call 'StOpt' the git dir you've just cloned

###################################################################################
## WINDOWS : Set a proper directory because you will need it when using CMake     ##
####################################################################################

For the purpose of this tutorial, let us create C:\local

1) unzip StOpt\utils\bzip2-master.zip in C:\local

2) unzip StOpt\utils\zlib-master.zip in C:\local

#####################################################################################
## WINDOWS :  Download CMake for Windows                                           ##
#####################################################################################

3) install it whenever you want

#####################################################################################
##  WINDOWS : Build Bzip2                                                          ##
#####################################################################################

4) build BZip2

  open CMake-gui

  in 'where is the source code' put C:\local\bzip2-master

  in 'where to build the binaries' put C:\local\bzip2

  click on Configure

  select Microsoft Visual Studio 12 2013 Win64
    (check that radio button 'Use default native compilers' is selected
    and click Finish)

  wait for Configuring done

  click on Generate

  close Cmake

  open C:\local\bzip2\bzip2.sln in Visual Studio 2013

  build the solution in both DEBUG and RELEASE

  In Release and Debug directory rename bz2.dll in libbz2.dll

####################################################################################
##  WINDOWS : Build Zlib                                                          ##
####################################################################################

5) build Zlib

  open CMake-gui

  in 'where is the source code' put C:\local\zlib-master

  in 'where to build the binaries' put C:\local\zlib-master

  click on Configure

  select Microsoft Visual Studio 12 2013 Win64
    (check that radio button 'Use default native compilers' is selected
    and click Finish)

  wait for Configuring done

  click on Generate

  close Cmake

  open C:\local\zlib-master\zlib.sln in Visual Studio 2013

  build the solution in both DEBUG and RELEASE


#####################################################################################
##  WINDOWS : Download Anaconda Python2.7 Win64                                    ##
#####################################################################################

5) install it in C:\Anaconda

###################################################################################
##  WINDOWS :Download eigen library                                               ##
####################################################################################

6) unzip in C:\local

####################################################################################
##  WINDOWS : Download win64 pre-built boost libraries for windows                 ##
#####################################################################################

7) install it in C:\local\boost_1_xx_x


##########################################################################
##  WINDOWS : IF EVERYTHING IS CORRECTLY DONE,                          ##
## your C:\local should look like this :                                ##
##########################################################################

DIR bzip2-master
DIR bizp2
DIR zlib-master
DIR eigen-eigen-xxxxxx
DIR boost_1_xx_x

##########################################################################
##  WINDOWS : Configure and Generate with CMAKE for StOpt             ##
##########################################################################

IT IS A TRIAL AND ERROR PROCESS SO DO NOT BE AFRAID OF SEEING ERRORS
Because some dependencies are not found, you will need to specify them

We try to be as comprehensive as we can in the following lines

8)Configure and generate a Visual Studio Solution from StOpt source files

  Open CMake-gui

  in 'where is the source code' put StOpt

  in 'where to build the binaries' put C:\local\build_StOpt

  CLICK ON CONFIGURE
    select Microsoft Visual Studio 12 2013 Win64
    (check that radio button 'Use default native compilers' is selected
    and click Finish)

  ...you see some errors

    First be sure that (supposing you don't need MPI neither SDDP)
    BUILD_PYTHON    is TICKED
    BUILD_MPI   is UNTICKED
    BUILD_SDDP  is UNTICKED

    Second, fill the
    BOOST_INCLUDE_DIR = C:/local/boost_1_xx_x

    Make sure Advanced tick box (on the upper right of cmake) is ticked

    +add entry
      NAME = BOOST_LIBRARY_DIR
      TYPE = PATH
      VALUE =  C:/local/boost_1_xx_x/lib64-msvc-12.0
      click OK


    +add entry
      NAME = BOOST_LIBRARYDIR
      TYPE = PATH
      VALUE = C:/local/boost_1_xx_x/lib64-msvc-12.0
      click OK

    Fill
    EIGEN3_INCLUDE_DIR = C:\local\eigen-eigen-xxxxxx


  CLICK ON CONFIGURE...  you see some errors

    Fill
    ZLIB_INCLUDE_DIR = C:/local/zlib-master
    ZLIB_LIBRARY = C:/local/zlib-master/Release/zlib.lib

    +add entry
      NAME = ZLIB_DLL_DIR
      TYPE = PATH
      VALUE = C:/local/zlib-master/Release


  CLICK ON CONFIGURE...  you see some errors

    Fill
    BZIP2_INCLUDE_DIR = C:\local\bzip2-master
    BZIP2_LIBRARY_DEBUG = C:/local/bzip2/Debug/bz2.lib
    BZIP2_LIBRARY_RELEASE = C:/local/bzip2/Release/bz2.lib

    +add entry
      NAME = BZIP2_DLL_DIR
      TYPE = PATH
      VALUE = C:/local/bzip2/Release


  CLICK ON CONFIGURE...  you see some errors

    Fill
    PYTHON_EXECUTABLE = C:\Anaconda\python.exe

  CLICK ON CONFIGURE... you see NO error and you see 'Configuring done'

  Now you can click on Generate

  Congratulations ! You have a VISUAL STUDIO SLN of the project

#########################################################################
##   Alternative compilation  without CMake Gui                         ##
########################################################################

In a command windows with VS 2015 for example

cmake -G "Visual Studio 14 2015 Win64" ../ 
-DBUILD_MPI=OFF 
-DBOOST_ROOT="PATH to boost directrory"
-DBOOST_LIBRARYDIR="PATH to boost lib"
-DEIGEN3_INCLUDE_DIR="PATH to Eigen"
-DPYTHON_EXECUTABLE="Python executable used for python mapping"
-DZLIB_INCLUDE_DIR="Zlib PATH"
-DZLIB_LIBRARY="zlib.lib library used"
-DBZIP2_INCLUDE_DIR="bzip2 path"
-DBZIP2_LIBRARY_DEBUG="bzip2.lib debug library"
-DBZIP2_LIBRARY_RELEASE="bz2.lib release library"
-DBZIP2_DLL_DIR="Bzip2 library directory with dll"
-DZLIB_DLL_DIR="Zlib library with dll"

##########################################################################
##  WINDOWS : Build the solution LIBRARY_STOPT.sln                       ##
##########################################################################

9) Build Solution

In VISUAL STUDIO 2013, open C:\local\build_StOpt\LIBRARY_STOPT.sln

BUILD Solution in RELEASE

##########################################################################
##  WINDOWS : Check for tests                                           ##
##########################################################################

10) we use ctest, so notice where the cmake.exe is
  if you do not know, check for Properties of the shortcut Cmake-gui

let us say that it is in "C:\Program Files (x86)\CMake\bin\ctest.exe"

now go to C:\local\build_StOpt

create a ctest.bat with the following content

'ctest.bat content--------------------------'

set OMP_NUM_THREADS=1
PATH=%PATH%;C:\local\boost_1_xx_x\lib64-msvc-12.0;C:\local\bzip2\Release;C:\local\zlib\Release
"C:\Program Files (x86)\CMake\bin\ctest.exe" -C Release


'-------------------------------------------'

now execute ctest.bat
all tests should be passed

TUTORIAL TESTED ON :
Win7 64bits SP1
Intel64 Family 6 Model 42 Stepping 7


###########################################################################
##   OPTIONAL WINDOWS : Generate an installer for the library            ##
###########################################################################
Install first Null Soft a available at  http://nsis.sourceforge.net/
In  a command tool go to C:\local\build_StOpt

    $ cpack -R Release
    
command will generate a windows installer.


###############################################################################
## Helper for cmake compilation  for projects using StOpt                    ##
###############################################################################
in utils directory , a file FindStOpt.cmake  and a file FindGeners.cmake are provided
to help compilation in projects using cmake