DATASERVER.md 12.7 KB
Newer Older
Charles Godwin's avatar
Charles Godwin committed
1 2 3 4 5
[TOC]



# Introduction
Charles Godwin's avatar
Charles Godwin committed
6 7 8

The Data Server is implemented in Java and has only one function: to read, when asked, all the messages sent between Magnum Energy components on an ME network and present the data as one large JSON file with a simple set of key/value pairs.

Charles Godwin's avatar
Charles Godwin committed
9
Users can augment the server,  using a plug-in,  to include data from other devices such as third party charge controllers or other devices. The author of this software has developed, separately, a plug-in to include readings from a MorningStar charge controller and a thermometer.
Charles Godwin's avatar
Charles Godwin committed
10 11 12 13 14 15 16 17

The GETTING_STARTED.md document covers the details of getting the server connected and running. This will sumarize the programs that are part of the server suite and also provide help for building more software.

When your server is up and running you can consider the following:

  - Customize the server [runtime settings](#customizing-runtime)

  - Setup your platform to ensure the server and logger are always  running. 
Charles Godwin's avatar
Charles Godwin committed
18
      The recommended way is to start the programs as a system service (recommended).
Charles Godwin's avatar
Charles Godwin committed
19
      Refer to the separate document named RUNNING_AS_SERVICE.md for details on how to make the processes run continuously. 
Charles Godwin's avatar
Charles Godwin committed
20
      **NOTE**: You cannot use command line override for options when using the service version.
Charles Godwin's avatar
Charles Godwin committed
21 22 23 24 25 26 27 28 29 30 31 32 33 34

  - What are you going to do with the data?

      - Put it in a database?
        
      - Use spreadsheets?
        
      - Adjust the filter to provide less and reordered data?

  - Build a web interface to show online status and statistics.

      - Here the sky is the limit. 
        An interesting tool to consider is [Grafana](https://grafana.com/) to present everything as graphs.

Charles Godwin's avatar
Charles Godwin committed
35
# Customizing Runtime
Charles Godwin's avatar
Charles Godwin committed
36 37 38 39 40 41 42 43 44 45 46

The data server program references a configuration file for settings. This files is named `conf/magnum_server.options`. This file can be edited by you to change the defaults settings as needed. One use is to change the default USB serial device name.

When running the command line version of the programs, you can also override all options at the command line by including the name=value parameter when invoking the command. If specifying an override value on the command line and there are spaces or special characters in the value, then surround the value with quotation marks. Like  `option="this value has space"`.  

**Server options**

| Option Name               | Default value       | Comment                                                      |
| ------------------------- | ------------------- | ------------------------------------------------------------ |
| datareaderdevice          | /dev/ttyUSB0        | See section in the [GETTING_STARTED](GETTING_STARTED.md) guide about changing this |
| dataserverport            | 19450               | Port the server listens on                                   |
Charles Godwin's avatar
Charles Godwin committed
47
| dateformat                | yyyy-MM-dd HH:mm:ss | Format required for Date value in log records. The string must conform to the convention [described here](https://docs.oracle.com/javase/7/docs/api/java/text/SimpleDateFormat.html). |
Charles Godwin's avatar
Charles Godwin committed
48
| timezone                  |                     | Sets the timestamp from the server to specific time zone. Examples are GMT, GMT+3, America/Toronto etc. |
Charles Godwin's avatar
Charles Godwin committed
49
|                           |                     | The following are advanced settings                          |
Charles Godwin's avatar
Charles Godwin committed
50 51 52 53 54
| packetstride              | 50                  | Number of packets collected for one log record. This normally does not need changing. |
| server_readers            |                     | If none are defined the default is MagnumReader. Known readers are MagnumReader, PacketFileReader, JSONReader. <br />You can add your own but it must include the full classname. |
| jsonreader_filename       |                     | filename of JSON file if JSONReader included as a reader.    |
| packetfilereader_filename |                     | filename of packet record file if PacketFileReader is included as a reader. This file is the output from testrs485. |

Charles Godwin's avatar
Charles Godwin committed
55

Charles Godwin's avatar
Charles Godwin committed
56
# What the Programs Do
Charles Godwin's avatar
Charles Godwin committed
57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75

These are the scripts provide to start various server side programs

| Program | Description  |
| ----------- | ---------------------------------------------------------------------------- |
| findserialport | Run during initial install to identify serial ports                        |
| startserver | Starts data server as background task. This will restart a running server. |
| stopserver  | Stops background server task                                               |
| showserver  | Displays the server status. Running or not.                                 |
| showstatus  | Display the status of server and logger on current system.                  |
| runserver | Runs the server in foreground. Useful for testing. |
| enableserver | Enables the Magnum reader as a system service. (refer to RUNNING_AS_SERVICE document) |
| disableserver | Disables the Magnum reader as a system service. |
| testrs485   | Run during initial install to validate Serial connection                   |

**Notes**:

  - startserver and startlogger will also restart the programs. That is,  if they are running they will be stopped and then started again.

Charles Godwin's avatar
Charles Godwin committed
76
## Problems, Questions or Comments
Charles Godwin's avatar
Charles Godwin committed
77 78 79 80 81 82

If you want report a bug, ask a question or comment on the software, please create an issue by going to the project page on GitLab <https://gitlab.com/Magnum_Energy/distribution/issues> and add an issue. 
You must be a registered user of GitLab but this is simple and cost free process. 

Or write the author magnum@godwin.ca

Charles Godwin's avatar
Charles Godwin committed
83
# The Web Interface
Charles Godwin's avatar
Charles Godwin committed
84 85 86 87 88 89 90

The server is accessed via a web interface. The server has only one function, to return a JSON log record based on current information on a Magnum Energy network.

That's all the server does. All other functionality such as CSV, SQL formatting, logging,  database updates and field filtering is performed by client side utilities.

These examples will assume the default server host name and port.

Charles Godwin's avatar
Charles Godwin committed
91
## Retrieve a JSON log record
Charles Godwin's avatar
Charles Godwin committed
92 93 94 95 96 97 98 99

The default web URL is http://localhost:19450/  
This will return a full JSON log record of all elements available in your network.

## Extending the Interface to Include Other Data Points

There is sometimes a need to add other data points to the log record such as third party charge controller data. There are tow ways to this:

Charles Godwin's avatar
Charles Godwin committed
100
  - Use the included plugin named ShellReader to invoke a shell command or script to provide JSON data to the server, or
Charles Godwin's avatar
Charles Godwin committed
101 102 103
  - Use the provided [plugin interface](#plugins) to write a Java extension to extract the data, or
  - [Wrap this server](#using-this-server-with-other-data-collection-programs) in  another data collection program.

Charles Godwin's avatar
Charles Godwin committed
104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137
Select the links for more details. 

There are unpublished extensions that are available on request.

- MorningStar MPPT Charge Controller using ModBus interface. This a Java plugin.
- SunSpec Alliance devices such as Outback Energy AXS. This is a python program.
- DS18B20 thermal sensor. This is a python program.
- An python example of reading the data server.

### Shell Reader

This plugin allows for quickly enabling a way to contribute data to the server. There is an example python program named ../examples/python/read_ds18b20.py that will read a DS18B20 sensor and provide, as JSON, the temperature. Refer to the file for technical details.

If you want to write your own script it only has to meet these criteria.

- It runs as a single callable command. If multiple steps are required, put it in a bash script.
- It must return a well formed JSON string consisting of only name-value pairs. This is an associative array on PHP and a dictionary in Python.
- It must return a non-zero exit value to indicate failure, and zero for success.
- The command string must contain complete full pathnames for the command and any file names. This is the same as the criteria for crontab jobs.
- If you want to enable multiple providers, separate the commands with a ;
- The value for shellreader_commands MUST be surrounded by quotes.

To enable a script set the following parameters in conf/magnum-server.conf

`server_readers=MagnumReader,ShellReader`
`shellreader_commands="<fully qualified command>"`

For example, this is used for the example program.

`shellreader_commands="/usr/bin/python3 /home/pi/magnum/python/read_ds18b20.py"`

You should test your command script at a command line before enabling it.

An example python program to return a temperature is on GitLab as examples/python/read_ds18b20.py.
Charles Godwin's avatar
Charles Godwin committed
138 139 140

### Plugins

Charles Godwin's avatar
Charles Godwin committed
141
Plugins need to be developed in Java to allow extra data points to appear in the generated output. Examples include third party charge controllers and temperature sensors.
Charles Godwin's avatar
Charles Godwin committed
142 143 144 145 146 147 148 149

Adding extra data is straight forward. You need to create a java class that implements ca.godwin.magnum.DataReader interface. This class is then dynamically loaded at run-time using the server_readers= option. You can define multiple plugins by coma delimiting their class names in the option. This class is responsible for returning an LinkedHashMap array of name / value pairs. For a sample of a simple plugin look at file named JSONReader.java in the supplied source.

How the data is collected in these classes is implementation specific. For example, the MorningStar controller has a ModBus interface so a plugin would communicate using their published interface, extract the data wanted, populate the object so that the two methods provide data.
The values are appended after the magnum fields in the order that the plugins are defined.  

There is a supplied set of files to help you set up an environment to build java plugins without the need to download and build the entire project. Refer to a document named PLUGIN_BUILD.md.

Charles Godwin's avatar
Charles Godwin committed
150
**NOTE **Any extra libraries used by a plugin must be added to the lib sub directory.  For example I had to  add jars for Modbus support.  If your runtime java invocation needs to be customized, I recommend you create your own, separate, script to prevent a future override of the standard script.
Charles Godwin's avatar
Charles Godwin committed
151 152 153

### Using This Server With Other Data Collection Programs

Charles Godwin's avatar
Charles Godwin committed
154 155
Since this server is a simple web interface it can provide all the its data to another program by using that program's equivalent of the curl command . For example, in PHP just use 
`$json = file_get_contents("http://localhost:19450");` to get the json string. In Python import the `requests` package (this may need to be installed) and use `me_data = requests.get("http://localhost:19450").json()`. This function has equivalents in many languages.
Charles Godwin's avatar
Charles Godwin committed
156 157 158

Then this program can combine the ME data with other collected data to produce a log record. The advantage of this method is there is no need implement any code using Java.

Charles Godwin's avatar
Charles Godwin committed
159 160
A Python example is provided in in GitLab examples/python/read_server.py

Charles Godwin's avatar
Charles Godwin committed
161 162 163 164 165 166 167 168 169 170 171 172
## Simulation modes

The server normally runs by listening to the USB serial interface to read the data packets sent out on the ME network. However, the server can also run with test data in simulation mode. There are two ways of doing this.

  - Using JSON log records created by the server,

  - Using mock data records generated by the testrs485 routine.

### JSON Alternate Data

If you want to read stored JSON records saved by the datalogger the load the JSON record reader plugin. This plugin will load any JSON data from a file as long as the data using a simple name=value type of data, similar to what is produced by the server. Add these items to your conf/magnum-server.conf file or the startserver command.

173
`server_readers=JSONReader`  
Charles Godwin's avatar
Charles Godwin committed
174 175 176 177
`jsonreader_filename=<file name with JSON records>`

To use this you will need to collect JSON data on a live system. With the server running and reading real information from an ME network, start the logger and ensure these two options are set:

178 179
`logger_json=true`  
`logger_json_filter=false`
Charles Godwin's avatar
Charles Godwin committed
180 181 182 183 184 185 186

Let it run for a long time, I waited 2 days, and then combine the daily json log files and use this as input. You'll have lots of testing data.

### Test Data Packets

The server normally runs by listening to the USB serial interface to read the data packets sent out on the ME network. However, the server can also run with mock data in simulation mode. The program testrs485, used to test the interface, produces a file with all the read packets displayed with the packets contents in Hexadecimal notation. Add these two lines to the magnum_server.options file or include them as parameters when using startserver.

187
`server_readers=PacketFileReader`  
Charles Godwin's avatar
Charles Godwin committed
188
`packetfilereader_filename=<filename of output from testrs485>`
Charles Godwin's avatar
Charles Godwin committed
189 190 191

When running in simulation mode, the server grabs the next 50 packets and presents them as a log record. When it reaches the end of the file it starts over, and so on and so on. If you want a larger sample for testing you can run the testrs485 when attached to the magnum network to get more records. Just use the command:

Charles Godwin's avatar
Charles Godwin committed
192
`testrs485 packets=<number of packets you want>`
Charles Godwin's avatar
Charles Godwin committed
193 194

I normally pick packets=1000, which takes about 45 seconds to generate.