NAME

VBTK - Virtual Brent Toolkit - A generic toolkit for system monitoring

DESCRIPTION

VBTK is a collection of modules which can be used to build a complex monitoring system completely written in perl. It's system monitoring abilities are, at the moment, mostly limited to solaris unix servers. But it also supports monitoring through SNMP, HTTP, SMTP, POP3, etc., and the framework is easily expanded. More modules will be added over time. See the sections below on File System Layout, Environment Variables, Server and Control Modules, and Data-Gathering Modules.

Note that the VB toolkit is just a group of modules. You write the final perl program which runs them yourself. But don't let that deter you, everything is defaulted and quite simple, so the final perl program is usually very simple. Also, there are many examples and templates to copy from.

SERVER AND CONTROL MODULES

The following modules are used in the VBTK collection to define the VB Server, and control the various monitoring processes.

VBTK::Server

This module allows the definition of a VB Server daemon process. This process is responsible for gathering, monitoring, and displaying the statuses and associated text of of all test objects. It has an embedded web server which is the access point for client processes to submit statuses, as well as for users to view statuses. See VBTK::Server for more details.

VBTK::Actions

This module allows the definition of actions which can be triggered by the VB server based on status changes.

There are also sub-classes of the VBTK::Actions module. These provide convenient defaults for two common types of actions - email and paging. See VBTK::Actions for more details.

VBTK::Controller

This module is used to start/stop all the various client and server processes which make up the VBTK monitoring system. See VBTK::Controller for more details.

DATA-GATHERING MODULES

The following modules are run as part of client processes which gather data, process it, and report it to the VB server. Each module is setup with common defaults so that it can be used with minimal configuration. Some modules have sub-classes which provide additional groups of common defaults, to further simplify configuration. See each module's man page for more details.

VBTK::Wrapper

This module is the heart of most of the monitoring scripts. It is a simple command-line-execution wrapper. It runs a command on the command line and then parses the output. It can also tail the output of a log file. If multiple VBTK::Wrapper objects are defined in the same perl process, the VBTK engine will time-share between them. It passes it's collected data on to the VBTK::Parser module.

There are also sub-classes of the VBTK::Wrapper module. These provide convenient defaults for various monitoring configurations, including running vmstat, df, metatstat (DiskSuite), vxprint (VXVM), ping, etc. (Mostly Solaris utilities) See VBTK::Wrapper for more details.

VBTK::Snmp

This module allows the monitoring of systems through Snmp. It retrieves a specified list of SNMP oid's on a specified interval and passes them on to the VBTK::Parser module.

There are also sub-classes of the VBTK::Snmp module. These provide convenient defaults for various monitoring configurations, including monitoring network I/O, Dynamo, NT cpu utilization, etc. See VBTK::Snmp for more details.

VBTK::Http

This module allows the monitoring of a web server. It retrieves HTML from specified URL's on a specified interval. It can establish an HTML baseline and compare the retrieved HTML to the original baseline with each pass. It also records the response time of the HTML retrieval. See VBTK::Http for more details.

VBTK::Tcp

The module allows the monitoring of a TCP listener. It simply attempts to connect to a specified host and port and then logs the success or failure. It's good for monitoring any TCP listener. It records the reponse time of the connection attempt. See VBTK::Tcp for more details.

VBTK::DBI

This module allows the monitoring of databases through the use of the DBI module. It executes pre-defined SQL on a specified interval, and passes the results on to the VBTK::Parser module.

There are also sub-classes of the VBTK::DBI module. These provide convenient defaults for various monitoring configurations, including monitoring oracle tablespaces, io rates, and blocking processes.

VBTK::Smtp

This module allows the monitoring of an SMTP server. It sends an email through the specified SMTP server on a specified interval, measures the response time, and passes the result on to the VBTK::Parser module.

VBTK::Pop3

This module allows the monitoring of a POP3 server. It connects to the POP3 server on a specified interval, measures the response time, and passes the result on to the VBTK::Parser module. It can also be configured to retrieve email with a specified Subject, and measure and store the round-trip time. This is usually used together with the VBTK::Smtp module to test the full mail cycle.

VBTK::Log

This module allows the monitoring of ascii text logs. It will tail the specified log or logs, searching for patterns, and setting VBObject statuses accordingly.

VBTK::Parser

This module is never called directly from a monitoring script. It is the engine responsible for parsing through the data retrieved by the data-gathering modules and filtering, ignoring, or re-arranging the data as specified. It makes the decisions about what status to set, and handles contacting the VB server to report the status, update graphs, store text, etc. See VBTK::Parser for more details.

ENVIRONMENT VARIABLES

The VB toolkit can make use of several environment variables to override the defaults entered when VBTK was first installed. If you go with the defaults, it makes things much simpler.

VBHOME

This is the root directory of the VBTK files. (Defaults to /usr/vbtk).

VBPORT

The TCP port on which the VBServer web server will be available and which the client processes will report their statuses into the VBServer. (Defaults to 4712).

VBURI

The URI which the client processes will use to contact the VBServer when reporting their statuses. (Defaults to 'http://vbserver:$VBPORT') Note that if you just setup a DNS alias for 'vbserver', then you don't have to change the default.

VBPSWD

The path to a file where passwords can be stored for automated access by client processes. This is currently only used by the VBTK::DBI module.

VBLOGS

The directory where the logs of STDOUT from each of the VBServer and client processes will be stored. (Defaults to '$VBHOME/logs')

VBOBJ

The directory where the object database will be stored. Don't put anything else into this directory, it should just be for the object database. (Defaults to '$VBHOME/vbobj') You can override this in the call to new VBTK::Server with the ObjectDir parameter if you want.

RRDBIN

The location of the rrdtool binary. Defaults to '/usr/local/bin/rrdtool'.

FILE SYSTEM LAYOUT

By default, the VB toolkit installs into a single directory as designated by the environment variable VBHOME. Sub-directories are then setup as follows. See the section on Environment Variables for details on how to relocate these from their default settings.

bin

Location of some administrative scripts used for installing and replicating the VBTK software.

conf

Location of user-defined configuration scripts

examples

A directory containing example files which can be copied into the 'conf' directory and then customized.

logs

Location of STDOUT log files from the VBServer and client processes.

mib

Directory in which MIB files can be installed for use by the VBTK::Snmp module.

perf

Directory in which performance related log files are usually stored by default. These log files contain the data gathered by the client monitoring processes. Their format and location is determined by the 'LogFile', 'LogHeader', and 'LogDetail' parameters passed in the contructors of the data-gathering modules.

scripts

Directory in which can be placed shell or other scripts used by the data-gathering processes to collect their data.

vbobj

Directory in which the VBServer stores it's VB Object database. For each VB object created, the VBServer will create a sub-directory using the objects full name. All history data, RRD databases, meta data, etc will be stored in this sub-directory.

web

Document root directory for the VBServer's built-in web server. The '.phtml' and images files in this directory are all user-customizable, so feel free to customize the web interface as you like. See the section on Customizing the Web Interface in the VBTK::Server documentation for more details.

DEPENDENCIES

The VB toolkit depends on several other perl modules, which are readily available from CPAN.

REQUIRED DEPENDENCIES

The following modules are required. They are available from CPAN - http://www.cpan.org/modules.

Bundle::LWP
Date::Manip
Algorithm::Diff
File::Find
Storable
Mail::Sendmail
Mail::POP3Client

The simplest way to install these is to just use 'perl -MCPAN -e shell'. Once in the shell, just type 'install Date::Manip', 'install Storable', etc.

OPTIONAL DEPENDENCIES

In addition, the VB toolkit can make use of the following modules if installed.

rrdtool

Allows for creation of RRD databases and dynamic generation of graphs if the rrdtool is installed along with it's dependencies. You can find the source at http://www.rrdtool.com/index.html . I strongly recommend installing this!

SNMP

Allows for SNMP monitoring if the SNMP module is installed along with it's dependencies. This requires requires the Net-SNMP toolkit library available from http://sourceforge.net/projects/net-snmp .

DBI

Allows for database monitoring via the DBI module, if the DBI module is installed along with it's dependencies.

INITIAL CONFIGURATION

Follow the README instructions for doing the usual Perl module 'make', 'make test' and 'make install'. Once that's done, getting a vanilla system up and running is fairly easy.

ARCHITECTURE DESIGN

Before we start configuring anything, you need to plan your monitoring architecture. Draw a map of this on paper first, taking into consideration the following areas. Just start with something simple at first.

Decide where to run the Master VBServer

Decide where you'll run the master VBServer process. This is the instance where you'll edit the config files for all other hosts and from which those config files will be replicated when you use the 'vbc sync' command.

Note that the VBServer process can become quite intensive if you have a lot of objects to monitor, so choose a machine which you can dedicate to just monitoring. Also, make sure it's a stable machine (I guess that rules out Windows...) which isn't going to get un-plugged by the janitor or crash regularly.

Decide what hosts you will monitor

List out your hosts and what services they run. Decide what you'll want to monitor. (ie: disk space, cpu utilization, web server, ftp server, mail server, etc.)

Decide where to monitor from

Some monitoring is done locally, such as disk space, cpu utilization, disk volume status, log tailing, etc. Other monitoring is usually done remotely, such as ping and testing of smtp, snmp, pop3, and other TCP services. It's usually easiest to run all the remote tests from one or two machines. Also you may want to run the same test from multiple locations. For example, you might want to test your HTTP server from both inside and outside the network. That way you could better differentiate between a network failure and a HTTP server failure.

Decide where to run Slave VBServers

If you have hosts you want to monitor which are not on the same LAN as the master VBServer, it's best to setup a slave VBServer on the remote LAN. The clients send a lot of data to their VBServer and you don't want this all going over the WAN. In addition, the master VBServer and slave VBServer can be configured to heartbeat each other, so that in case one dies, the other can warn you.

Decide where you'll access the web interface from

You can access the web interface from the Master node or from any Slave node. They will all show all the objects in one interface. But don't expose this interface to the public internet, since I'm not sure about vulnerabilities.

VBSERVER CONFIGURATION

Designate one host as the place to run your master VBServer. This will be where you make changes to the config files. You can then use the 'vbc sync' command on each of the other hosts to download the changes.

Note that you don't have to use the 'sync' functionality. You could just use rsync or something similar to copy the config files around, or you could just edit the config files independently on each host. I prefer to have one master and sync out from there, because it makes it easier to keep track of all the config differences.

Initial Setup

Follow the 'PER-SERVER CONFIGURATION' instructions below for just the VBServer host. When you run the installer, answer (M)aster when prompted for which type of processes you'll be running on this host.

Edit 'vbc' Configuration File

Look in the $VBHOME/examples directory and you'll find example config files. These are actually perl programs, so it will help if you know some perl. The 'vbc' config file will have already been copied over to the $VBHOME/conf directory. Open up the 'vbc' config file. This is the controller file which defines which instances will be running on each host. You should already see the master VBServer host listed there. Add in all your other hosts, and specify any slave servers.

Edit 'vbserver' Configuration File

Copy the 'vbserver' config file from the 'examples' directory into the 'conf' directory and edit it. Just read the comments.

Start the 'vbserver' process.

Run 'conf/vbc start vbserver' and then check the log file 'logs/vbserver...' for errors. Now try to connect with a web browser to 'http://<hostname>:4712'.

PER-SERVER INITIAL CONFIGURATION

Now that you've drawn your monitoring map, let's get the various servers setup. The software install is identical regardless of what you're going to run on the host. If you have hosts which won't be running any local tests, then you obviously don't need to install anything there.

Create a user and group 'vbtk'.

If you're running this on a unix machine, you don't want to run it as superuser, so create a user 'vb' and group 'vb' to run all the processes under.

    useradd vbtk 
    groupadd vbtk
Create a home directory.

Create a home directory for the VBTK software. I suggest '/usr/vbtk', which is the default. Change this directory to be owned by the 'vbtk' user and to be the 'vbtk' user's home directory.

    mkdir /usr/vbtk; chown vb:vb /usr/vbtk

If you're going to use a directory other than '/usr/vbtk', then make sure you set the environment variable $VBHOME to the new location. Make sure this is always set before you run any scripts.

Run the installer.

This will check the VBHOME directory and prompt you for either the location of the VBTK install directory or a VBServerURI from which it can sync config files. It also does some setup in the $VBHOME directory.

    perl -MVBTK -e install

ON-GOING CONFIGURATION

Once you have the initial VBServer running, read through the config files in the examples directory and copy ones you want to use over to the conf directory. Remember you have to list these config files in the 'vbc' config file in order to start them with the 'vbc' command.

You can also look through the various modules in the Data Gathering Modules section of this document for help on creating new config files.

SECURITY

Don't expose the web interface to the public internet! I make no claims about how secure it is. There may be vulnerabilities which I don't know about.

FAQ

How do I delete an object?

Before you can delete an object, make sure that there isn't a client process which is still sending sending a status for that object. Otherwise it will just get recreated with the next status submission. If you're not sure which client process is setting the status, look under the 'Info' tab for that object for the 'Script Name' and 'Running From' values. This shows you which script is setting the status. Once you think you've fixed the script, check the object history to see if it's still being updated. If not, go to the 'vbobj' directory, on the Master or Slave server to which this object is reported, and delete the sub-directory which matches the object name. After about a minute, the object will disappear from the web interface.

There are error messages in the VBServer log about the 'rrdupdate' command failing.

If you've changed the number of entries being passed in the RrdColumns entry, or if you've changed the CF specification, then you probably need to rebuild the corresponding RRD database. Look in the VBHOME/vbobj directory for a directory named the same as the object and delete the RRD.db file. Then restart the client process which monitors the object and the RRD.db file will be regenerated.

OPTIONAL CUSTOMIZATION

The VBTK toolkit is written such that additional data-gathering modules can be added on to it fairly easily. See the 'examples' sub directory under the VBTK home directory for examples of how to write your own data-gathering modules, or look at the code of the data-gathering modules listed above.

KNOWN PROBLEMS

@delta counter roll-over

The @delta array used in the 'VBDetail' and 'LogDetail' parms of data-gathering arrays doesn't yet take into account 32-bit and 64-bit roll-over. So you may get an occasional very large negative number when counters roll-over. This will be fixed in a future release.

Won't run on Windows

I did some initial testing running this under Windows, but had too many problems with the 'fork' emulation. It just didn't work quite right. Also, windows doesn't support 'alarm', which I use quite a bit. So it's not going to work. It runs great on Cygwin however, so this is probably the best way to go if you want to run it on a windows platform.

The VBServer engine needs performance tuning

The VBServer engine uses more resources than I would have liked, but seems to run okay. On my Sun Netra T1 440MHz server, with about 100 objects reporting in, it runs at about 3-5% all the time. I'll work on getting that down lower in a future release.

Date::Manip timezone determination

The VBServer process will default to GMT if Date::Manip can't determine the timezone. See "Timezones" in Date::Manip for details on how to set the Timezone.

FUTURE ENHANCEMENTS

I have several things in mind, but feel free to send suggestions.

Action History Log and improved Actions

There needs to be a single place to go to see a log of all the actions fired recently. In addtion, I would like to add the option of requiring actions to be acknowledged. That way if your sysadmin misses the first page, it will keep paging every n minutes, until they log in and acknowledge the problem.

I'd like to add the option of saving the cookie database somewhere on disk, so that it doesn't get wiped out every time you restart.

Admin page in web interface

I'd like to add an 'Admin' page in the web interface to allow administrative actions such as deleting an object, resetting the RRD database, etc.

MaxRepeat option for object history

I'd like to add a 'MaxRepeat' option for object history.

SEE ALSO

VBTK::Server
VBTK::Actions
VBTK::RmtServer
VBTK::Templates
VBTK::Controller
VBTK::Wrapper
VBTK::Snmp
VBTK::Http
VBTK::Tcp
VBTK::DBI
VBTK::Smtp
VBTK::Pop3
VBTK::Log
VBTK::Parser

SPECIAL THANKS

And a special thanks to all the great perl authors who's libraries I used to build this. I couldn't have done it without you! CPAN is great!

AUTHOR

Brent Henry, vbtoolkit@yahoo.com

COPYRIGHT

Copyright (C) 1996-2002 Brent Henry

This program is free software; you can redistribute it and/or modify it under the terms of version 2 of the GNU General Public License as published by the Free Software Foundation available at: http://www.gnu.org/copyleft/gpl.html

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.