[TOC]



# Introduction

The Data Server is implemented in Java and has only one function: to read, when asked, all the messages sent between Magnum Energy components on an ME network and present the data as one large JSON file with a simple set of key/value pairs.

Users can augment the server,  using a plug-in,  to include data from other devices such as third party charge controllers or other devices. The author of this software has developed, separately, a plug-in to include readings from a MorningStar charge controller and a thermometer.

The GETTING_STARTED.md document covers the details of getting the server connected and running. This will sumarize the programs that are part of the server suite and also provide help for building more software.

When your server is up and running you can consider the following:

  - Customize the server [runtime settings](#customizing-runtime)

  - Setup your platform to ensure the server and logger are always  running. 
      The recommended way is to start the programs as a system service (recommended).
      Refer to the separate document named RUNNING_AS_SERVICE.md for details on how to make the processes run continuously. 
      **NOTE**: You cannot use command line override for options when using the service version.

  - What are you going to do with the data?

      - Put it in a database?
        
      - Use spreadsheets?
        
      - Adjust the filter to provide less and reordered data?

  - Build a web interface to show online status and statistics.

      - Here the sky is the limit. 
        An interesting tool to consider is [Grafana](https://grafana.com/) to present everything as graphs.

# Customizing Runtime

The data server program references a configuration file for settings. This files is named `conf/magnum_server.options`. This file can be edited by you to change the defaults settings as needed. One use is to change the default USB serial device name.

When running the command line version of the programs, you can also override all options at the command line by including the name=value parameter when invoking the command. If specifying an override value on the command line and there are spaces or special characters in the value, then surround the value with quotation marks. Like  `option="this value has space"`.  

**Server options**

| Option Name               | Default value       | Comment                                                      |
| ------------------------- | ------------------- | ------------------------------------------------------------ |
| datareaderdevice          | /dev/ttyUSB0        | See section in the [GETTING_STARTED](GETTING_STARTED.md) guide about changing this |
| dataserverport            | 19450               | Port the server listens on                                   |
| dateformat                | yyyy-MM-dd HH:mm:ss | Format required for Date value in log records. The string must conform to the convention [described here](https://docs.oracle.com/javase/7/docs/api/java/text/SimpleDateFormat.html). |
| timezone                  |                     | Sets the timestamp from the server to specific time zone. Examples are GMT, GMT+3, America/Toronto etc. |
|                           |                     | The following are advanced settings                          |
| packetstride              | 50                  | Number of packets collected for one log record. This normally does not need changing. |
| server_readers            |                     | If none are defined the default is MagnumReader. Known readers are MagnumReader, PacketFileReader, JSONReader. <br />You can add your own but it must include the full classname. |
| jsonreader_filename       |                     | filename of JSON file if JSONReader included as a reader.    |
| packetfilereader_filename |                     | filename of packet record file if PacketFileReader is included as a reader. This file is the output from testrs485. |


# What the Programs Do

These are the scripts provide to start various server side programs

| Program | Description  |
| ----------- | ---------------------------------------------------------------------------- |
| findserialport | Run during initial install to identify serial ports                        |
| startserver | Starts data server as background task. This will restart a running server. |
| stopserver  | Stops background server task                                               |
| showserver  | Displays the server status. Running or not.                                 |
| showstatus  | Display the status of server and logger on current system.                  |
| runserver | Runs the server in foreground. Useful for testing. |
| enableserver | Enables the Magnum reader as a system service. (refer to RUNNING_AS_SERVICE document) |
| disableserver | Disables the Magnum reader as a system service. |
| testrs485   | Run during initial install to validate Serial connection                   |

**Notes**:

  - startserver and startlogger will also restart the programs. That is,  if they are running they will be stopped and then started again.

## Problems, Questions or Comments

If you want report a bug, ask a question or comment on the software, please create an issue by going to the project page on GitLab <https://gitlab.com/Magnum_Energy/distribution/issues> and add an issue. 
You must be a registered user of GitLab but this is simple and cost free process. 

Or write the author magnum@godwin.ca

# The Web Interface

The server is accessed via a web interface. The server has only one function, to return a JSON log record based on current information on a Magnum Energy network.

That's all the server does. All other functionality such as CSV, SQL formatting, logging,  database updates and field filtering is performed by client side utilities.

These examples will assume the default server host name and port.

## Retrieve a JSON log record

The default web URL is http://localhost:19450/  
This will return a full JSON log record of all elements available in your network.

## Extending the Interface to Include Other Data Points

There is sometimes a need to add other data points to the log record such as third party charge controller data. There are tow ways to this:

  - Use the included plugin named ShellReader to invoke a shell command or script to provide JSON data to the server, or
  - Use the provided [plugin interface](#plugins) to write a Java extension to extract the data, or
  - [Wrap this server](#using-this-server-with-other-data-collection-programs) in  another data collection program.

Select the links for more details. 

There are unpublished extensions that are available on request.

- MorningStar MPPT Charge Controller using ModBus interface. This a Java plugin.
- SunSpec Alliance devices such as Outback Energy AXS. This is a python program.
- DS18B20 thermal sensor. This is a python program.
- An python example of reading the data server.

### Shell Reader

This plugin allows for quickly enabling a way to contribute data to the server. There is an example python program named ../examples/python/read_ds18b20.py that will read a DS18B20 sensor and provide, as JSON, the temperature. Refer to the file for technical details.

If you want to write your own script it only has to meet these criteria.

- It runs as a single callable command. If multiple steps are required, put it in a bash script.
- It must return a well formed JSON string consisting of only name-value pairs. This is an associative array on PHP and a dictionary in Python.
- It must return a non-zero exit value to indicate failure, and zero for success.
- The command string must contain complete full pathnames for the command and any file names. This is the same as the criteria for crontab jobs.
- If you want to enable multiple providers, separate the commands with a ;
- The value for shellreader_commands MUST be surrounded by quotes.

To enable a script set the following parameters in conf/magnum-server.conf

`server_readers=MagnumReader,ShellReader`
`shellreader_commands="<fully qualified command>"`

For example, this is used for the example program.

`shellreader_commands="/usr/bin/python3 /home/pi/magnum/python/read_ds18b20.py"`

You should test your command script at a command line before enabling it.

An example python program to return a temperature is on GitLab as examples/python/read_ds18b20.py.

### Plugins

Plugins need to be developed in Java to allow extra data points to appear in the generated output. Examples include third party charge controllers and temperature sensors.

Adding extra data is straight forward. You need to create a java class that implements ca.godwin.magnum.DataReader interface. This class is then dynamically loaded at run-time using the server_readers= option. You can define multiple plugins by coma delimiting their class names in the option. This class is responsible for returning an LinkedHashMap array of name / value pairs. For a sample of a simple plugin look at file named JSONReader.java in the supplied source.

How the data is collected in these classes is implementation specific. For example, the MorningStar controller has a ModBus interface so a plugin would communicate using their published interface, extract the data wanted, populate the object so that the two methods provide data.
The values are appended after the magnum fields in the order that the plugins are defined.  

There is a supplied set of files to help you set up an environment to build java plugins without the need to download and build the entire project. Refer to a document named PLUGIN_BUILD.md.

**NOTE **Any extra libraries used by a plugin must be added to the lib sub directory.  For example I had to  add jars for Modbus support.  If your runtime java invocation needs to be customized, I recommend you create your own, separate, script to prevent a future override of the standard script.

### Using This Server With Other Data Collection Programs

Since this server is a simple web interface it can provide all the its data to another program by using that program's equivalent of the curl command . For example, in PHP just use 
`$json = file_get_contents("http://localhost:19450");` to get the json string. In Python import the `requests` package (this may need to be installed) and use `me_data = requests.get("http://localhost:19450").json()`. This function has equivalents in many languages.

Then this program can combine the ME data with other collected data to produce a log record. The advantage of this method is there is no need implement any code using Java.

A Python example is provided in in GitLab examples/python/read_server.py

## Simulation modes

The server normally runs by listening to the USB serial interface to read the data packets sent out on the ME network. However, the server can also run with test data in simulation mode. There are two ways of doing this.

  - Using JSON log records created by the server,

  - Using mock data records generated by the testrs485 routine.

### JSON Alternate Data

If you want to read stored JSON records saved by the datalogger the load the JSON record reader plugin. This plugin will load any JSON data from a file as long as the data using a simple name=value type of data, similar to what is produced by the server. Add these items to your conf/magnum-server.conf file or the startserver command.

`server_readers=JSONReader`  
`jsonreader_filename=<file name with JSON records>`

To use this you will need to collect JSON data on a live system. With the server running and reading real information from an ME network, start the logger and ensure these two options are set:

`logger_json=true`  
`logger_json_filter=false`

Let it run for a long time, I waited 2 days, and then combine the daily json log files and use this as input. You'll have lots of testing data.

### Test Data Packets

The server normally runs by listening to the USB serial interface to read the data packets sent out on the ME network. However, the server can also run with mock data in simulation mode. The program testrs485, used to test the interface, produces a file with all the read packets displayed with the packets contents in Hexadecimal notation. Add these two lines to the magnum_server.options file or include them as parameters when using startserver.

`server_readers=PacketFileReader`  
`packetfilereader_filename=<filename of output from testrs485>`

When running in simulation mode, the server grabs the next 50 packets and presents them as a log record. When it reaches the end of the file it starts over, and so on and so on. If you want a larger sample for testing you can run the testrs485 when attached to the magnum network to get more records. Just use the command:

`testrs485 packets=<number of packets you want>`

I normally pick packets=1000, which takes about 45 seconds to generate.