|
|
|
Overview
|
|
|
|
========
|
|
|
|
|
|
|
|
The OutputManager service is responsible for aggregating output data at a single process and then write this data to a Protocol Buffer file. Data aggregation is performed every N ticks, where N can be specified in the configuration. For local buffering of output data, Agents write these to an OutputBuffer, which is then read out by the OutputManager.
|
|
|
|
|
|
|
|
Goals & Non-Goals
|
|
|
|
=================
|
|
|
|
|
|
|
|
Provide a convenient as possbile way, fully encapsuled for "Agents" to store ouput data into a single output file. Within child classes of Agents, only two things need to be specified: "Which output columns exist", "What values to store for given columns"
|
|
|
|
|
|
|
|
An implementation is working correctly, if 100% of the data stored by Agents are written to the output file and all values are associated with the correct columns for the correct time. In addition, all input data given to the simulation shall be written to the output file, as well.
|
|
|
|
|
|
|
|
Current Solution
|
|
|
|
================
|
|
|
|
|
|
|
|
At the moment nothing is implemented, however AMIRIS uses a quite similar system. Output data are written to output "channels" which are then read out every N ticks and written to files, where each agent type has its own file. In Agent child classes, only an output channel needs to be instantiated, which is then used to store data to.
|
|
|
|
|
|
|
|
Proposed Solution
|
|
|
|
=================
|
|
|
|
|
|
|
|
<img src="images/OutputManagerMpi.png" width="900" height="320" />
|
|
|
|
<img src="images/OutputManagerLegend.png" width="475" height="250" />
|
|
|
|
|
|
|
|
In a first step, all input data are written to the output file.
|
|
|
|
|
|
|
|
Agent write their output data to an individual Buffer. Every Nth tick: The OutputManager collects all data from all local output Buffers and sends the output data collection to one process responsible for writing output to file. On this process: The data is given to a FileWriter, which saves them to the file system in Protocol Buffer format.
|
|
|
|
|
|
|
|
<img src="images/OutputManagerCollect.png" width="600" height="421" />
|
|
|
|
|
|
|
|
Agents may ask the static BufferManager for an individual, local output Buffer. All output data is stored to this buffer. The BufferManager keeps track of indices, corresponding simulation times and column names. The Output Manager may ask the BufferManager for a complete list of all currently buffered data, which it then transmits to one specified process writing the data to the file system.
|
|
|
|
|
|
|
|
<img src="images/OutputManagerDummyBuffer.png" width="757" height="250" />
|
|
|
|
|
|
|
|
Agents may request an individual OutputBuffer from the BufferManager. For this an agent sends its className and columnNames to the BufferManager. The simulation Configuration may activate / deactivate output from certain agent classes. This is managed by the BufferManager as well. If output is deactivated for an agent class, a DummyBuffer is
|
|
|
|
returned instead which will not store any output data.
|
|
|
|
|
|
|
|
<img src="images/OutputManagerRuntime.png" width="852" height="250" />
|
|
|
|
|
|
|
|
Alternative Solutions
|
|
|
|
=====================
|
|
|
|
|
|
|
|
The output data format was discussed and evaluated in depth. Alternative output formats like .csv were favoured at first, but Protocol buffers are an ideal way to transfer output data to the "evaluation stage" of ARCMod written in Python: they are faster and need less file size than .csv outputs. In comparison to .hdf5 and .netcdf file formats, for protobuf an individual file format can be designed, and a very good integration in Java, C++ and Python exists allowing a smoother integration and reduces overhead for using the respective library.
|
|
|
|
|
|
|
|
Cross-Team Impact
|
|
|
|
=================
|
|
|
|
|
|
|
|
<table style="width:100%;">
|
|
|
|
<colgroup>
|
|
|
|
<col style="width: 42%" />
|
|
|
|
<col style="width: 57%" />
|
|
|
|
</colgroup>
|
|
|
|
<thead>
|
|
|
|
<tr class="header">
|
|
|
|
<th>Impactor</th>
|
|
|
|
<th>Answers</th>
|
|
|
|
</tr>
|
|
|
|
</thead>
|
|
|
|
<tbody>
|
|
|
|
<tr class="odd">
|
|
|
|
<td>What maintenance effort to keep element running?</td>
|
|
|
|
<td>low - medium: changes in the output file format may be necessary</td>
|
|
|
|
</tr>
|
|
|
|
<tr class="even">
|
|
|
|
<td>What latency caused to the system?</td>
|
|
|
|
<td>medium: writing outputs to file & aggregation of output data may consume a non-negligible amount of processing time</td>
|
|
|
|
</tr>
|
|
|
|
<tr class="odd">
|
|
|
|
<td>Tightly coupled to:</td>
|
|
|
|
<td>Protobuf output messages</td>
|
|
|
|
</tr>
|
|
|
|
<tr class="even">
|
|
|
|
<td>Negative consequences?</td>
|
|
|
|
<td>Outputs may only be double values</td>
|
|
|
|
</tr>
|
|
|
|
</tbody>
|
|
|
|
</table>
|
|
|
|
|
|
|
|
Discussion
|
|
|
|
========== |
|
|
|
\ No newline at end of file |
