Nagios plugin guide attached.

Feel free to ping/grab me etc for more!

Note: statistics are actually collected slightly after 0:00 UTC, so if you do go with the "more than a day has passed since the listed update date" method I suggested in IRC, I'd suggest you use something like 24.5-25 hours. If you're just going with a "statistics should have been collected during today-in-UTC", don't check until at least 20 after. For reference, today's stats were available at about 15 minutes after the hour.

Just figured I should mention to prevent some false positives.

Attached is the statistics-collected script. It requires Perl DBI and DBD::Pg modules, but otherwise just needs Perl.

I don't know where this check will actually be ran, so I have not depended on musicbrainz-server or anything else. If it can actually be ran by astro or something, then I'd prefer to get this checked in to the musicbrainz-server project and simplify it to use DBDefs.

Thoughts?

Running it on astro seems like a good idea (assuming you're only talking for now about monitoring production). I suggest you add it to musicbrainz-server, presumably it can make use of settings etc so it knows how to connect to totoro - ideally you can just run something like

/path/to/codebase/bin/nagios/check-statistics-collected

(with no arguments, i.e. it Just Works)

then check that the same thing works when run as nagios, i.e.

sudo -u nagios /path/to/codebase/bin/nagios/check-statistics-collected

then once we've got that far we'll look into integrating it into Nagios.

Other questions to ponder:

• how often should it be checked (once every 5 mins? 4 hours? 24 hours?)
• if it goes warning/critical, how often should it be checked? (if different)
• whom, if anyone, should Nagios notify when there's a problem? (currently notifications take the form of emails)
• if there's a problem, should Nagios notify immediately, or (the default) wait for it to report a problem several times in a row?

In code review at http://codereview.musicbrainz.org/r/2080/

Re-opening: the nagios script is now in master, but I need to talk to Dave about what to do next (ie, actually getting it ran).

Ok, this works fine if I do:

sudo -u nagios /home/musicbrainz/musicbrainz-server/admin/nagios/statistics-collected

I would like this to be checked:

• At 5am, in whatever timezone astro is in.
• In warning/critical please check every 2 hours
• I would like to be notified of any change in status
• I am happy with Nagios reporting either immediately or after a few checks. Go with whatever is easiest.

djce@dudley gateway$ git commit -a
[master 08cbd43] MBH-259 Add a Nagios check to make sure statistics are being collected
1 files changed, 1 insertions, 0 deletions

Also
djce@astro:~$ cat /etc/nagios/nrpe.d/musicbrainz-server.cfg
command[check_mbserver_stats_collected]=/home/musicbrainz/musicbrainz-server/admin/nagios/statistics-collected
djce@astro:~$
which maybe should be part of the musicbrainz-server codebase?

djce@dudley nagios$ git commit -a
[master 83ce853] MBH-259 Add a Nagios check to make sure statistics are being collected
2 files changed, 28 insertions, 0 deletions

https://mail.eorbit.net/nagios/cgi-bin/extinfo.cgi?type=2&host=astro.musicbrainz.org&service=mbs+stats+collection

I don't have access to that URL, so I'm going to take your word. I think having nagios configuration outside the server is not a problem, so this all looks good now.