|
|
|
Overview
|
|
|
|
========
|
|
|
|
The OutputManager service is responsible for aggregating output data at a single process.
|
|
|
|
It writes this data to a Protocol Buffer file.
|
|
|
|
Data aggregation is performed every N ticks, where N can be specified by the user.
|
|
|
|
Agents buffer their output data in an individual OutputBuffer first, which is then read out by the OutputManager.
|
|
|
|
|
|
|
|
The OutputManager service is responsible for aggregating output data at a single process and then write this data to a Protocol Buffer file. Data aggregation is performed every N ticks, where N can be specified in the configuration. For local buffering of output data, Agents write these to an OutputBuffer, which is then read out by the OutputManager.
|
|
|
|
## Goals & Non-Goals
|
|
|
|
Provide a convenient way to fully encapsulate Agents output management.
|
|
|
|
Store data into a single output file.
|
|
|
|
Child classes of Agent shall only need to specify: "Which output columns exist", "What values to store for given columns".
|
|
|
|
|
|
|
|
Goals & Non-Goals
|
|
|
|
=================
|
|
|
|
An implementation is working correctly, if 100% of the data stored by Agents are written to the output file and all values are associated with the correct columns and the correct time.
|
|
|
|
In addition, all input data given to the simulation shall be written to the output file, as well.
|
|
|
|
|
|
|
|
Provide a convenient as possbile way, fully encapsuled for "Agents" to store ouput data into a single output file. Within child classes of Agents, only two things need to be specified: "Which output columns exist", "What values to store for given columns"
|
|
|
|
|
|
|
|
An implementation is working correctly, if 100% of the data stored by Agents are written to the output file and all values are associated with the correct columns for the correct time. In addition, all input data given to the simulation shall be written to the output file, as well.
|
|
|
|
|
|
|
|
Current Solution
|
|
|
|
================
|
|
|
|
|
|
|
|
At the moment nothing is implemented, however AMIRIS uses a quite similar system. Output data are written to output "channels" which are then read out every N ticks and written to files, where each agent type has its own file. In Agent child classes, only an output channel needs to be instantiated, which is then used to store data to.
|
|
|
|
|
|
|
|
Proposed Solution
|
|
|
|
=================
|
|
|
|
|
|
|
|
<img src="images/OutputManagerMpi.png" width="900" height="320" />
|
|
|
|
<img src="images/OutputManagerLegend.png" width="475" height="250" />
|
|
|
|
## Solution
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
In a first step, all input data are written to the output file.
|
|
|
|
|
|
|
|
Agent write their output data to an individual Buffer. Every Nth tick: The OutputManager collects all data from all local output Buffers and sends the output data collection to one process responsible for writing output to file. On this process: The data is given to a FileWriter, which saves them to the file system in Protocol Buffer format.
|
|
|
|
|
|
|
|
<img src="images/OutputManagerCollect.png" width="600" height="421" />
|
|
|
|
|
|
|
|
Agents may ask the static BufferManager for an individual, local output Buffer. All output data is stored to this buffer. The BufferManager keeps track of indices, corresponding simulation times and column names. The Output Manager may ask the BufferManager for a complete list of all currently buffered data, which it then transmits to one specified process writing the data to the file system.
|
|
|
|
|
|
|
|
<img src="images/OutputManagerDummyBuffer.png" width="757" height="250" />
|
|
|
|
|
|
|
|
Agents may request an individual OutputBuffer from the BufferManager. For this an agent sends its className and columnNames to the BufferManager. The simulation Configuration may activate / deactivate output from certain agent classes. This is managed by the BufferManager as well. If output is deactivated for an agent class, a DummyBuffer is
|
|
|
|
returned instead which will not store any output data.
|
|
|
|
|
|
|
|
<img src="images/OutputManagerRuntime.png" width="852" height="250" />
|
|
|
|
|
|
|
|
Alternative Solutions
|
|
|
|
=====================
|
|
|
|
|
|
|
|
The output data format was discussed and evaluated in depth. Alternative output formats like .csv were favoured at first, but Protocol buffers are an ideal way to transfer output data to the "evaluation stage" of ARCMod written in Python: they are faster and need less file size than .csv outputs. In comparison to .hdf5 and .netcdf file formats, for protobuf an individual file format can be designed, and a very good integration in Java, C++ and Python exists allowing a smoother integration and reduces overhead for using the respective library.
|
|
|
|
|
|
|
|
Cross-Team Impact
|
|
|
|
=================
|
|
|
|
|
|
|
|
<table style="width:100%;">
|
|
|
|
<colgroup>
|
|
|
|
<col style="width: 42%" />
|
|
|
|
<col style="width: 57%" />
|
|
|
|
</colgroup>
|
|
|
|
<thead>
|
|
|
|
<tr class="header">
|
|
|
|
<th>Impactor</th>
|
|
|
|
<th>Answers</th>
|
|
|
|
</tr>
|
|
|
|
</thead>
|
|
|
|
<tbody>
|
|
|
|
<tr class="odd">
|
|
|
|
<td>What maintenance effort to keep element running?</td>
|
|
|
|
<td>low - medium: changes in the output file format may be necessary</td>
|
|
|
|
</tr>
|
|
|
|
<tr class="even">
|
|
|
|
<td>What latency caused to the system?</td>
|
|
|
|
<td>medium: writing outputs to file & aggregation of output data may consume a non-negligible amount of processing time</td>
|
|
|
|
</tr>
|
|
|
|
<tr class="odd">
|
|
|
|
<td>Tightly coupled to:</td>
|
|
|
|
<td>Protobuf output messages</td>
|
|
|
|
</tr>
|
|
|
|
<tr class="even">
|
|
|
|
<td>Negative consequences?</td>
|
|
|
|
<td>Outputs may only be double values</td>
|
|
|
|
</tr>
|
|
|
|
</tbody>
|
|
|
|
</table>
|
|
|
|
|
|
|
|
Discussion
|
|
|
|
========== |
|
|
\ No newline at end of file |
|
|
|
Agent write their output data to an individual Buffer.
|
|
|
|
Every N-th tick: The OutputManager collects all data from all local output Buffers and sends the output data collection to one process responsible for writing output to file.
|
|
|
|
On this central output process: The data is given to a FileWriter, which saves them to the file system in Protocol Buffer format.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Agents may ask the static BufferManager for an individual, local output Buffer.
|
|
|
|
All output data is stored to this buffer.
|
|
|
|
The BufferManager keeps track of local Buffers and column names.
|
|
|
|
The OutputManager may ask the BufferManager for a complete list of all currently buffered data.
|
|
|
|
This data is then transmits to one specified process writing the data to the file system.
|
|
|
|
|
|
|
|
|
|
|
|
Agents request an individual OutputBuffer from the BufferManager by specifying their className and columnNames.
|
|
|
|
In the Simulation configuration, output from certain classes can be deactivated.
|
|
|
|
For classes with deactivated output, BufferManager returns a DummyBuffer which will not store any output data.
|
|
|
|
|
|
|
|
## Alternative Solutions
|
|
|
|
Format of output data was discussed and evaluated in depth.
|
|
|
|
Alternative output formats like .csv were favoured at first, but Protocol buffers are an ideal way to transfer output data across platforms and programming languages.
|
|
|
|
In comparison to .hdf5 and .netcdf file formats, protobuf offers straightforward integration for Java, C++ and Python.
|
|
|
|
|
|
|
|
## Impacts
|
|
|
|
|Aspect|Impact|
|
|
|
|
|--------|-------|
|
|
|
|
|What maintenance effort to keep element running?|low - medium: changes in the output file format may be necessary|
|
|
|
|
|What latency caused to the system?|medium: writing outputs to file & aggregation of output data may consume a non-negligible amount of processing time|
|
|
|
|
|Tightly coupled to:|protobuf|
|
|
|
|
|Negative consequences?|for simple outputs: may only be double values & agents can only store one value per column and tick| |
|
|
\ No newline at end of file |
