Commit 742d4df3 authored by Botond Botyanszki's avatar Botond Botyanszki

update mongodb-input

parent 76af41c9
#!/bin/sh
# Generate HTML format
asciidoctor mongodb-perl-input.adoc
# Generate PDF format
asciidoctor-pdf mongodb-perl-input.adoc
:addon-name: mongodb-input
[id="addon-{addon-name}"]
[desc="Pull data from a MongoDB database"]
= MongoDB
include::../../_asciidoctor/public.adoc[]
The MongoDB add-on provides an example Perl script for collecting data from a
MongoDB database with NXLog. The script must be modified to correspond with
the data to be collected.
. Copy the `mongodb-input.pl` script to a location that is accessible by
NXLog.
. Edit the script as required for connecting to the database and according to
the data to be collected.
. Configure NXLog. Use the <<im_perl,im_perl>> module to execute the Perl
script, which in turn pulls data from the MongoDB database and generates
events.
When new documents are available in the database, the script will sort
them by ObjectId and process them sequentically, passing them to NXLog by
calling `Log::Nxlog::add_input_data()`. Finally, the document is deleted from
the collection. The script will poll the database continuously with
`Log::Nxlog::set_read_timer()`. In the event that the MongoDB server is
unreachable, the timer delay will be increased to attempt reconnection later.
WARNING: The script deletes files from MongoDB after download. This might not
be the desired behavior for your particular scenario. For instance,
the original files may be need to be preserved for compliance or
forensic reasons.
.Collecting Data From MongoDB
====
For this example, a JSON data set of US ZIP (postal) codes was used. The data
set was fed to MondoDB using the following command:
[source,console]
----
$ mongoimport -d zips -c zips --file zips.json
----
.nxlog.conf
[source,config]
----
include::snippets/nxlog.conf[tag=doc_include]
----
.Input Sample
[source,json]
----
include::snippets/sample.json[]
----
.Output Sample
[source,log]
----
include::snippets/sample.out[]
----
====
This diff is collapsed.
[[mongodb_perl_input]]
= MongoDB integration using im_perl
This script can be used with NXLog to pull data from a MongoDB database.
This section describes how to use NXLog and generate logs from a MongoDB input.
[[mongodb_perl_input_config]]
== Configuring NXLog
The im_perl module is used to execute the perl script that will pull the
data from the MongoDB database. For simplicity, in the configuration below
the output is saved to a file.
[source,config]
----
include::../nxlog.conf[]
----
=== Test case
I this test case, a JSON data set of US zip (postal) codes was used. It was
downloaded from http://jsonstudio.com/resources/. The data set can be fed
to MondoDB using the following command:
[source,bash]
----
$ mongoimport -d zips -c zips --file zips.json
----
The perl script is constructed in such a way that when new documents are available
in the database, it will first sort them by ObjectId, process them sequentically
and pass them to NXLog by calling `Log::Nxlog::add_input_data()`. Finally, the
document is deleted from the collection. The script will poll the database
continuously by setting a timer `Log::Nxlog::set_read_timer()`. In the event that
the MongoDB server is unreachable, ( hence an exception is thrown while trying to
manipulate the data ) the timer is set to a different value so that the connection
can be reestablished at a later time.
==== Sample Input and Output
This is sample input and output using the US zip data set.
[source,json]
----
include::sample.json[]
----
[source,output]
----
include::sample.out[]
----
include common.conf
# tag::doc_include[]
<Input perl>
Module im_perl
PerlCode mongodb-input.pl
</Input>
<Output file>
Module om_file
File '/tmp/output.log'
</Output>
# end::doc_include[]
{ "_id" : "01001", "city" : "AGAWAM", "loc" : [ -72.622739, 42.070206 ], "pop" : 15338, "state" : "MA" }
{ "_id" : "01002", "city" : "CUSHMAN", "loc" : [ -72.51564999999999, 42.377017 ], "pop" : 36963, "state" : "MA" }
{ "_id" : "01005", "city" : "BARRE", "loc" : [ -72.10835400000001, 42.409698 ], "pop" : 4546, "state" : "MA" }
{ "_id" : "01007", "city" : "BELCHERTOWN", "loc" : [ -72.41095300000001, 42.275103 ], "pop" : 10579, "state" : "MA" }
{ "_id" : "01008", "city" : "BLANDFORD", "loc" : [ -72.936114, 42.182949 ], "pop" : 1240, "state" : "MA" }
{ "_id" : "01010", "city" : "BRIMFIELD", "loc" : [ -72.188455, 42.116543 ], "pop" : 3706, "state" : "MA" }
{ "_id" : "01011", "city" : "CHESTER", "loc" : [ -72.988761, 42.279421 ], "pop" : 1688, "state" : "MA" }
{ "_id" : "01012", "city" : "CHESTERFIELD", "loc" : [ -72.833309, 42.38167 ], "pop" : 177, "state" : "MA" }
{ "_id" : "01013", "city" : "CHICOPEE", "loc" : [ -72.607962, 42.162046 ], "pop" : 23396, "state" : "MA" }
{ "_id" : "01020", "city" : "CHICOPEE", "loc" : [ -72.576142, 42.176443 ], "pop" : 31495, "state" : "MA" }
ID: 01001 City: AGAWAM Loc: -72.622739,42.070206 Pop: 15338 State: MA
ID: 01002 City: CUSHMAN Loc: -72.51565,42.377017 Pop: 36963 State: MA
ID: 01005 City: BARRE Loc: -72.108354,42.409698 Pop: 4546 State: MA
ID: 01007 City: BELCHERTOWN Loc: -72.410953,42.275103 Pop: 10579 State: MA
ID: 01008 City: BLANDFORD Loc: -72.936114,42.182949 Pop: 1240 State: MA
ID: 01010 City: BRIMFIELD Loc: -72.188455,42.116543 Pop: 3706 State: MA
ID: 01011 City: CHESTER Loc: -72.988761,42.279421 Pop: 1688 State: MA
ID: 01012 City: CHESTERFIELD Loc: -72.833309,42.38167 Pop: 177 State: MA
ID: 01013 City: CHICOPEE Loc: -72.607962,42.162046 Pop: 23396 State: MA
ID: 01020 City: CHICOPEE Loc: -72.576142,42.176443 Pop: 31495 State: MA
<Input perl>
Module im_perl
PerlCode /opt/nxsec/bin/perl-input.pl
</Input>
<Output file>
Module om_file
File '/tmp/output.log'
</Output>
<Route perl_to_file>
Path perl => file
</Route>
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment