After more than 25 years of computer business we still lack of consistent performance monitoring between different operating systems, each system deploying its own type of monitoring and data collection. UNIX systems try to stay a bit close with each other since all are POSIX systems and follow similar industry standards, like The Open Group. Other systems, like Windows, use different data collection techniques.
It is very difficult to have a consistent data recording across many operating systems without purchasing separately additional software or install 3rd parties software. Even more, the recorded format data varies from system to system making difficult the collection and analysis.
If we step back and we look other industries, how are they doing it, we see a completely different picture, efforts being made towards standardization and common ways to record and offer data for analysis:
Airplanes use some sort of data recorders, usually found as a device called flight data recorder FDR, built to store aircraft data parameters. Such unit is found by default on many airplanes nowadays and its usage is regulated by governments and federal administrations, example FAA in United States. This device sometimes is referred as the black box.
Ships, boats or other type of vessels use some sort of recorder, called voyager data recorder VDR, used to store vessel data parameters. Similar to aerospace industry such devices are required when a certain vessel must comply with international standards, International Convention for the Safety of Life at Sea, SOLAS. Used mainly for accident investigation the VDR can serve as preventive maintenance, performance efficiency monitoring, heavy weather damage analysis and accident avoidance. This device sometimes is referred as the black box.
Automobiles use some sort of device used to store vehicle parameters, called event data recorder EDR. EDRs are not enforced by any standard organizations and are not really required by law so their usage varies from vendor to vendor. National Highway Traffic Safety Administration, NHTSA, proposed a series of changes to standardize and enforce mandatory EDR installation and usage by vendors. Around 2010 over 85% of all vehicles in US would already have some sort of EDR installed.
Computer systems have no such data recording device, installed. Manufacturers are not interested in standardizing this effort since they prefer selling additional software packages which can perform such recording features for an extra cost. The lack of standardisation and agreements between vendors makes computing business complex and difficult to handle performance data. Currently, there are hundreds of performance monitoring solutions.
All recorded observations we call them raw data. Raw data is produced by a monitoring agent, running on each host we plan to record data from. This set of data is not modified, altered or changed in any way and it is entirely the way we collected from the computer system. Its format is simple, as already mentioned, having its parameters collected separated by a character like , or :. Each recorder will write and store all collected parameters under such raw data file for the entire duration of its execution.
By default, the raw data file has the extension called, krd, recorder datafile. Each krd file will have the following format, described below:
Raw Data File timestamp: parameter: parameter: parameter: ... parameter timestamp: parameter: parameter: parameter: ... parameter ... timestamp: parameter: parameter: parameter: ... parameter timestamp: parameter: parameter: parameter: ... parameter