Eight round of improvements in 2.X series

In this release together with upgrade to PostgreSQL 13 aggregated column indices finally come to fruition and we are able to significantly push down search times of IP address and range based queries from tens of seconds to (usually) subsecond speed. Together with overlapping range aggregation for storage of source/target heavy events it seems we have finally finished our performance goal, which started by switching from MongoDB to PostgreSQL. Toast time. (Even though Mentat 2.8 will run on older PostgreSQL instances, to take advantage on those improvements you have to upgrade to PostgreSQL 13.)

Timeline aggregation framework now prominently replaces Hosts view in the main menu and is now better integrated with Event search.

Also, usual set of bugfixes, UI, API, documentation and framework cleanup went in.

Mentat specific namespace within events was originally _CESNET. As a means to shed company dependencies, the key is now renamed to _Mentat. It is advisable to review the Inspector configuration for rules related to the keys in this namespace and to review possible related homegrown code.

Multiple instances of Inspector are now folded into one with a default ruleset merged in pursue for a simpler default configuration. If you use the default configuration, you can just use new default Inspector and Controller configuration (which installation from Debian packages does for you). If you've made local changes, you might review new distribution configuration files and also decide to merge.

Please, visit the issue tracker for list of related issues:

   https://homeproj.cesnet.cz/versions/103

Seventh round of improvements in 2.x series

 This release brings major improvements in our Timeline search module. It now enables greater search condition customization capabilities, the same as our Events search module. The search forms are almost identical, which provides users ability to quickly jump from one results page to another with same search conditions. We have also improved search performance of the Timeline module by performing data aggregations and calculations in database instead of in the application. Sadly, not all of the previous aggregation calculations are now supported, because the data were not available directly in our database model. We might bring them back in the future.

This release also lays the groundwork for future abandonment of including report data as email attachments. In the future email reports will contain only links, which can be used to obtain full data, or the users will have the option to use web version of our reports (link is also included in the email). We are encountering issues with misconfigured mailers or too aggressive email filters, which prevent our reports from being successfully delivered. Also the email format is very restrictive and we are unable to present all necessary information in clear form. You should, after all, use only 80 characters per line, and that is not much. Also the use of CSV format for data attachments is now deprecated and will be removed in one of the future releases.

We have also focused on squishing some annoying bugs and a lot of invisible man hours went into writing better tests for our web interface codebase, so that we can have some peace of mind and produce better releases.

Please visit our ticket tracking system for more in-depth information about this release:

https://homeproj.cesnet.cz/versions/97

Fifth round of improvements in 2.x series

This release brings in major improvements in reporting component. Online reports are now more interactive and integrated into other parts of the system, there are context actions available for each node. On top of that there is a simple feedback button available for each address in each report section, so that users may provide their feedback more comfortably. System Mentat is now capable of enriching displayed information with data from third party services like DNS, PassiveDNS, NERD, WHOIS and GeoIP. There is a new module available currently for system administrators that attempts to display all available information for single IP address. Group membership management was simplified to enable group managers to more easily add or remove members and even activate new user accounts. Additionally we have also managed to squash quite a few bugs.

It is also worth noting, that this version also attempts to speed up the database searching by using aggregated IP ranges to narrow down the number of searched rows even more.

Please visit our ticket tracking system for more in-depth information about this release:

https://homeproj.cesnet.cz/versions/86

Sixth round of improvements in 2.x series

This release brings in further improvements to our reporting component.
Reports are now templated according to the classification of each
reported event to provide recipients with most important information
relevant to that event class. This new feature is fully configurable for
administrators of Mentat system, soon user manual will be provided.

Additionally lot of work went into database optimizations. First the
PostgreSQL was upgraded to latest version 12. Next we have increased the
amount of possible paralel queries by separating stored IDEA BSON to
different table. We have implemented basic DoS prevention mechanism by
limiting number of queries each user may execute at any given time. We
have also increased the speed in which IDEA events are stored into
database by using bulk inserts.

The MaxMind IP geolocation service recently changed its policies for
accessing their free databases, so we have addressed this issue as well
as the change with access policies to CESNET PassiveDNS service.

Full list of resolved issues can be found in our ticket tracking system:

	https://homeproj.cesnet.cz/versions/93

v2.4.0          Fourth round of improvements in 2.x series

    This release completely changes the installation procedures when installing from Debian packages and also attempts to simplify the necessary bootstrap procedures for novice developers. The Debian packages now preconfigure custom Python virtual environment and the whole Mentat system is then installed into that environment using native Python package management. This approach greatly simplifies the installation procedure, we can now install more recent Python packages for you without breaking your system. Additionally lot of work went into making the whole project executable from within the cloned git repository, which should simplify the development process for novice developers. Additionally we have also managed to squash quite a few bugs.

    It is also worth noting, that this release removes the support for automated database migration from MongoDB to PostgreSQL!

    Full list of resolved issues can be found in our ticket tracking system:

        https://homeproj.cesnet.cz/versions/85

Third round of improvements in 2.x series

After quite long gestation period this release brings brand new timeline visualisations for event and reporting dashboards. For system administrators there is now a new module available called Timeline, which provides results similar to that of event dashboards with the difference that the result is calculated directly from event database. These calculations are very expensive, so this feature should be considered as experimental for now and that is the reason why it is currently accessible only to the administrators. Depending on the size of the selected network, time window and result set the calculations may take minutes.

There are also some improvements under the hood. The event database migration mechanism was implemented to enable further database schema improvements. Also the JavaScript charting library underwent first part of major design overhaul.

Full list of resolved issues can be found in our ticket tracking system:

	https://homeproj.cesnet.cz/versions/83

Second round of improvements in 2.x series

This release brings two major improvements. First there is the much better integration of changelogs into the Hawat web interface components, which enables administrators better monitoring of user changes. The other major improvement is the grunt work for implementing API interface has been done and the event search form is the first part of the interface that provides the JSON API. To enable access to the API from arbitrary scripts and applications a new authentication mechanism based on API keys was implemented.

Full list of resolved issues can be found in our ticket tracking system:

	https://homeproj.cesnet.cz/versions/82

First round of improvements in 2.x series

This release focuses on resolving most important bugs and issues, that
were discovered after production deployment. There are some database
query performance optimizations and couple of new features were
implemented as well, the most visible being the web interface dashboard
for system administrators and better integration of item changelog
within the web interface.

Full list of resolved issues can be found in our ticket tracking system:

	https://homeproj.cesnet.cz/versions/81

This tag marks the first production release of Mentat system 2.0.0. After lots of work we have finally successfully abandoned the original implementation written in Perl and using MongoDB as database backend.

This commit also marks the end of an era of frantic coding. Until now some of the guidelines and best practices for coding and committing were violated in favor of faster development. This was also possible thanks to the fact that there was mostly only single developer working on the project, so there were no possible conflicts. From now on however things will be back to normal state.

 base for pilot 1 phase 1