rrdtool log parser

I need a script to parse logs and generate graphs using rrdtool.

You will create the script (ideally as a single file in perl) and provide instructions to install it. The script will be callable via cron.

If you require perl modules (or anything else) this will need to be detailed at the time you quote for the work so that I can ensure they don't conflict with anything we're doing already.

The graphs generated will be in the form of a viewable html page with static graphics. Users who visit the page will not trigger regeneration of the graphs, this must be done via the script each time it is called using cron.

## Deliverables

POP3 / IMAP logging

==============

Here is an example of pop3 and imap logins as recorded in my [url removed, login to view] file:

Dec 30 07:18:56 dylan cyrus/pop3d[21364]: login: [url removed, login to view][[url removed, login to view]] username1 APOP User logged in

Dec 30 07:21:38 dylan cyrus/pop3d[21256]: badlogin: [url removed, login to view][[url removed, login to view]] NTLM authentication failure

Dec 30 07:21:41 dylan cyrus/pop3d[21256]: badlogin: [url removed, login to view][[url removed, login to view]] DIGEST-MD5 authentication failure

Dec 30 07:21:42 dylan cyrus/pop3d[21256]: login: [url removed, login to view][[url removed, login to view]] username2 plaintext?

Dec 30 07:21:49 dylan cyrus/imapd[21297]: login: [url removed, login to view][[url removed, login to view]] username3 plaintext?

Dec 30 07:21:56 dylan cyrus/pop3d[20799]: login: [url removed, login to view][[url removed, login to view]] username4 CRAM-MD5 User logged in

As we can see pop3 logins are listed as 'cyrus/pop3d'

As we can see imap logins are listed as 'cyrus/imapd'

We support? CRAM-MD5, DIGEST-MD5, NTLM, plaintext, and APOP (I think thats the full list)

I need to graph logins with rrdtool much as you can see for smtp here:

[url removed, login to view]

(note there are multiple graphs for weeks, months, etc - I'll need much the same)

I want to show successful pop3 logins, and imap logins on a graph.

I also want to show unsuccessful logins (authentication failures) either on the same graph as the logins or on a new graph under the successful logins graph.

A third graph will list successful logins to both imap and pop broken into authentication types (eg pop3/cram-md5 or imap/plaintext)

You will be responsible for ensuring all cyrus log data can be parsed, I don't know if all cyrus logs look like those above, they probably do, but this is your responsibility to check and ensure they can all be parsed.

Data will likely gathered (by your script) once per day from /var/log/mail.log.0

SMTP / spam / virus logging

==========

We use DSPAM.

DSPAM stores its logs in /usr/local/var/dspam/[url removed, login to view]

This file is being rotated very infrequently (every few months) but this may change.?

You will need to parse data from this file and ensure that you do not duplicate input data when graphing.

I'm not familiar with the format of the [url removed, login to view] file for DSPAM but as we're not using it for anything else I will change the format if you post clear instructions of how this can be done.

Data for incoming / outgoing email is being recorded by postfix in syslog.?

Data will be gathered once per day from /var/log/mail.log.0

I need an output much as you can see here:

[url removed, login to view]

My link goes to output form [url removed, login to view] which doesn't (currently) parse my logs, it also doesn't (currently) work to the schedule that I want.

Schedule

=======

MailGraph runs as a daemon in the background,? I do not want a daemon running in the background.

I want a single script (ideally in perl) that can be called via cron when the server isn't busy.

The intent is that the script will be called once per day and with the exception of [url removed, login to view] for dspam (which is a live log file) it will operate with data from a static unchanging /var/log/mail.log.0 file and split out a static html page with static graphs, data will be stored in an rrdtool database.

It occurs to me that if I were to save the 'last read' line in input files the script could be called as many times as I wanted and not just once per day (start reading at 'last read' line, parse to end of file, record new 'last read' line) and this would also make the script work on live log file.

Additionally this would allow for detection of log rotation (the file being read has less lines than the 'last read' line value) and the scripts could then automatically switch to using the rotated file ($filename'.0'), finish it, and return to the new file.

Preference would be given to anyone who will implement this functionality - please indicate if you would include this functionality.

More information

===========

If more information is required please ask before submitting a quote.

I would much prefer to have a proper agreement as to the work being done before a quote ;)

You will be provided with a non-root user account on our server, and a copy of current [url removed, login to view], mail.log.0, and [url removed, login to view] files.

You will put the script on the server and demonstrate that it runs correctly, we would expect the output html files, rrdtool files, and static graphics (for the graphs) to be created in your 'home' account area and not the correct location when tests are performed.

We are unwilling to provide you with root access to our server and will install the script after you demonstrate that it works correctly.

We do have root access if there is anything you require, rrdtool is already installed.

Ingegneria Linux MySQL PHP Gestione Progetti Architettura Software Testare Software

Rif. progetto: #3053412

Info sul progetto