Log Monitor

ProTop has built-in monitoring for database log files and can additionally monitor any generic log file such as admserv.log, ubroker logs or any custom application logs. Follow the three simple steps below. [UNIX/Linux only]

NOTE:

• The log file monitor includes self-monitoring of tmp/*.debug files by default. Uncomment the logmon.sh scheduled task in your etc/schedule.*.cfg to enable, you don't have to do anything else. If you are already using logmon, simply add the lines from the latest etc/logmon.cfg to your localized (etc/logmon.*.cfg) configuration file to enable monitoring of tmp/*.debug.
•  Log files for databases monitored with ProTop are already scanned by default and do not need to be configured in your etc/logmon.*.cfg file.

Quick Setup Example

1. copy $PROTOP/etc/logmon.cfg to logmon.[custid].cfg
2. copy $PROTOP/etc/log_text.cfg to log_text.[custid].cfg
3. edit logmon.[custid].cfg and add the log files you want to monitor, one log file per line. Give it a unique ID, the type of log file (text is most common), the explicit path (or relative to $PROTOP) to the log file followed by a 0 (more on the last value below) e.g.:
   
   007 text /data/mb117/pt/log/dbmonitor.log 0
4. edit log_text.[custid].cfg and add the string to search for, one per line. Give it a unique id, a start position, an operator, the string to search for, an action, and a nag frequency. For example:
   
   007 0 contains 0000 alert 300
   
   For this example, this will search for the string "0000" in the entire dbmonitor.log file and send an alert every 5 minutes when found.
5. Uncomment the logmon.sh line in your $PROTOP/etc/schedule.[custid].cfg. If you do not have one, copy it from the schedule.cfg.
   
   0,15,30,45 * * * * logmon.sh >> ${PTTMP}/logmon.err 2>&1 [NOALERT]
6. Watch for the alerts in your feed on the portal.

NOTICE: These alerts are sent by the ProTop client to the portal with a resource of `hostname`, i.e., custid.testserver.  You will see these in an unfiltered view of the alert feed. To see all of these alerts, filter on: alert.subject matches *text*. If you filter on a site, you will not see them (at the moment).

What you will see on the portal:

Read on for the nitty gritty detail...

Included monitor types

Protop is installed with 3 types of log monitors. See below for instructions on adding new types or customizing these.

The log monitor configuration file: etc/logmon.*.cfg

To have protop monitor log files, you need to edit your etc/logmon.*.cfg to specify which files you want to monitor and with what type of monitor. Each configuration file line contains four fields separated by spaces explained in the table below. 

By way of example, here is the current default etc/logmon.cfg:

# etc/logmon.cfg
#
# log files to monitor
#
# id# type pathname backup
#
# 01 ubrk /u1/wrk/asesb/logs/restbroker1.broker.*.log 0
#
# 51 ubrk /u1/wrk/asesb/logs/restbroker1.server.*.log 0
# text example that looks for errors in $PROTOP/tmp/*.debug files
#
# enable the following in etc/schedule.cfg:
#
#    5,20,35,50 * * * * logmon.sh >> ${PTTMP}/logmon.err 2>&1 [NOALERT]

990 text tmp/pt3agent.*.debug 0
999 text tmp/dbmonitor.debug 0

The log monitor "type" configuration file: etc/log_text.*.cfg

Each monitor type is composed of two files:

“xxxx” is the monitor type, as specified in etc/logmon.cfg above.

The syntax of the config files can be different for each monitor type. The 4GL procedure (.p) implements the necessary logic to parse these rules and apply an action.

For example, here is the default generic text monitor config file, etc/log_text.cfg. The only active line in this file is the one that begins 4glerr:

# etc/log_text.cfg
#
# uniqId startPos operation target action nagFrequency
#
# mixing radically different rule sets is a bad idea
#  - think about it, i'm sure you will agree.
#  - if a set of rules go together then that is probably a log file "type"
#  - so clone the text monitor and create a new "type" specific to those rules
#
#001 17 begins   trax       alert 3600                  # starting at position 17 if the line begins "trax"
#002  0 index    "audit "   alert 3600                  # starting at position 0 if the line contains "audit "
#003  0 contains " 2*rhsmd" alert 3600                  # starting at position 0 if the line matches "* 2*rhsmd*"
#
#
#900   0 contains "abl*(453)" alert 300
#901  32 begins   "P"         alert 60
#902   0 index    "(7129)"    alert 3600
#903   0 ends     "batch."    alert 0
# an example for detecting errors in protrace or *.debug files
#

4glerr 0 index " --> " alert 0   # starting at position 0 if the line contains " --> " throw a "4glerr" alert

# make sure that we end the file with a linefeed

log_text.cfg expects these columns:

Possible actions

Here are some examples of etc/log_text.cfg entries

001 17 begins trax alert 3600 # starting at position 17 if the line begins "trax"
002 0 index "audit " alert 3600 # starting at position 0 if the line contains "audit "
003 0 contains " 2*rhsmd" alert 3600 # starting at position 0 if the line matches "* 2*rhsmd*"
903 0 ends "batch." alert 0 # starting at position 0 if the line ends with "batch."

Once your etc/logmon.*.cfg and etc/log_text.cfg files are customized, enable logmon.sh in your ProTop scheduler by removing the # at the beginning of this line in your etc/schedule.*.cfg: 

# 
# 5,20,35,50 * * * * logmon.sh >> ${PTTMP}/logmon.err 2>&1 [NOALERT] 

Or, add this line to your crontab on Linux or Unix to run logmon.sh every 15 minutes. Replace [PROTOPDIR] with your ProTop installation directory e.g. /usr/var/protop:

*/15 * * * * [PROTOPDIR]/bin/logmon.sh >> [PROTOPDIR]/log/logmon.log 2>&1

Creating your own simple log monitor type

It is a good practice to logically separate rule sets for a specific topic in different monitor types. You can create a custom log type simply by following these steps:

1. Choose a name for your new monitor type. It should be a short word with no spaces. It will represent a new monitor type. Example: “edi”.
2. Make a copy etc/log_text.cfg and util/log_text.p in their respective folder, renaming the files by replacing “text” with your new name. e.g. etc/log_edi.cfg and util/log_edi.p
3. Edit log_edi.cfg to add your own rules.

4. Register your new type by adding a line in util/logmon.p at the end of the existing section containing these lines:

   run util/log_oedb.p persistent. run util/log_ubrk.p persistent. run util/log_text.p persistent. run util/log_edi.p persistent.

5. Enable the new log monitor by editing etc/logmon.cfg. Add a line specifying the filename that you want to monitor and your new monitor type name:

   10 edi /edi/outgoing/logs/out.log 0

Sample custom monitor for database Logs

Note: Log files for databases monitored by ProTop are already scanned. You can use this monitor for other OpenEdge databases not registered with ProTop.

There is a custom database log named etc/log_oedb.cfg that can be used or extended to scan OpenEdge database log files. It takes advantage of the structured log format to scan for specific message number and perform an action. In addition to matching a message number, it records the full message text, the process id and the user number. This monitor is a starting point to perform more complex processing when scanning database logs.

The configuration file structure is described below:

Sample content of etc/log_oedb.cfg

# etc/log_oedb.cfg
#
# msg# action nagFrequency

452 login 0
453 logout 60
7129 ignore 0