Monitor logfiles and command output on AIX using multitail.

Most applications, like operating systems, have their own logs files. Sometimes, we need to see them at the same time to troubleshoot issues. Here’s when multitail (http://www.vanheusden.com/multitail/) can help us.
This tool can be installed on an AIX box. You can download it from http://www.perzl.org/aix/index.php?n=Main.Multitail

First, check that you need some packages dependencies:

Package dependencies:
    ncurses >= 5.6

Once we have downloaded multitail and its dependencies packages, we’re going to installed them:

(lpar):[root] /tmp -> ls -ltr *.rpm
-rw-r--r--    1 root     system      3511978 Nov 18 11:12 ncurses-5.9-1.aix5.1.ppc.rpm
-rw-r--r--    1 root     system       387633 Nov 18 11:12 ncurses-devel-5.9-1.aix5.1.ppc.rpm
-rw-r--r--    1 root     system       182153 Nov 18 11:17 multitail-6.2.1-1.aix5.1.ppc.rpm
(lpar):[root] /tmp -> rpm -Uvh ncurses-5.9-1.aix5.1.ppc.rpm
ncurses                     ##################################################
(lpar):[root] /tmp -> rpm -Uvh ncurses-devel-5.9-1.aix5.1.ppc.rpm
error: failed dependencies:
        /opt/freeware/bin/bash is needed by ncurses-devel-5.9-1

Wait a minute, ncurses-devel failed because needs ‘bash’ packege as dependencies, but in our environment we don’t use bash, so we can ignore this error executing again the same command with ‘–nodeps’ options:

(lpar):[root] /tmp -> rpm -Uvh ncurses-devel-5.9-1.aix5.1.ppc.rpm --nodeps
ncurses-devel               ##################################################

Now, let’s install multitail package:

(lpar):[root] /tmp -> rpm -Uvh  multitail-6.2.1-1.aix5.1.ppc.rpm
multitail                   ##################################################

To verify the installation:

(lpar):[root] /tmp -> rpm -qa|egrep 'ncurses-|multitail'
multitail-6.2.1-1
ncurses-devel-5.9-1
ncurses-5.9-1

Once it has finished let’s play with multitail 😉
Multitail has its own conf file on /etc/multitail.conf. Its configuration is not easy to understand at first time, so take your time to read some docs on http://www.vanheusden.com/multitail/

One feature of multitail, besides filter and merge, is you can colorize logs files.

Which color scheme has multitail? Below, you can see all colorcheme:

HINT: You can create your owns colorcheme for your applications and later execute multitab with options ‘-cS’ name_own_colorcheme

(lpar):[root] /tmp -> cat /etc/multitail.conf|grep "colorscheme:"
# colorscheme:
colorscheme:syslog:kernel and unsorted messages
colorscheme:ssh:www.openssh.org
colorscheme:powerdns:www.powerdns.com
colorscheme:logcat
colorscheme:liniptfw:Linux IPtables (2.6.x kernel)
colorscheme:postfix:www.postfix.org
colorscheme:apache:default Apache logging (webserver)
colorscheme:apache_error:default Apache error logging
colorscheme:rsstail:RSSTail output (RSS feed reader)
colorscheme:acctail:(BSD-) process accounting reader
colorscheme:wtmptail:www.vanheusden.com/wtmptail/
colorscheme:squid:http proxy server
colorscheme:asterisk:software PBX
colorscheme:sendmail
colorscheme:mailscanner:wrapper around sendmail/clamav/spamassassin
colorscheme:spamassassin
colorscheme:clamav:clamd logging
colorscheme:samba
colorscheme:audit
colorscheme:exim
colorscheme:httping:ping for HTTP
colorscheme:netstat:see www.vanheusden.com/multitail/examples.html
colorscheme:tcpdump
colorscheme:dhcpd
colorscheme:bind
colorscheme:smartd
colorscheme:kerberos
colorscheme:oracle
colorscheme:ntpd
colorscheme:nagtail:www.nagios.org status viewer
colorscheme:websphere:WebSphere error-log
colorscheme:nntpcache
colorscheme:vnetbr:Veritas Netbackup backup/restore logs
colorscheme:procmail
colorscheme:checkpoint:Checkpoint Firewall-1
colorscheme:pppd:PPP daemon
colorscheme:inn
colorscheme:netscapeldap:Netscape Directory server (LDAP)
colorscheme:vmstat:vmstat is part of sysstat
colorscheme:mpstat:mpstat is part of systat
colorscheme:log4j
colorscheme:lambdamoo:MUD/MOO server http://www.moo.mud.org/
colorscheme:boinctail:BOINCTail http://www.vanheusden.com/boinctail/
colorscheme:p0f:p0f http://lcamtuf.coredump.cx/p0f.shtml
colorscheme:portsentry:http://sourceforge.net/projects/sentrytools/
colorscheme:strace:strace is the truss of Linux
colorscheme:argus:Argus http://qosient.com/argus/
colorscheme:ii:ii IRC client http://www.suckless.org/wiki/tools/irc
colorscheme:snort:Intrusion detector
colorscheme:motion:Security camera software
colorscheme:errpt:AIX error reporting tool
colorscheme:mysql:MySQL error log
colorscheme:boinc:BOINC http://boinc.berkeley.edu/
colorscheme:acpitail:Show temperature/battery/etc info
colorscheme:qmt-clamd
colorscheme:qmt-smtp
colorscheme:qmt-send
colorscheme:qmt-spamassassin
colorscheme:qmt-sophie

Let’s see examples.
1- See apache/httpd logs

(lpar):[root] /tmp ->  multitail -cS apache /var/log/httpd/access_log
(lpar):[root] /tmp ->  multitail -cS apache_error /var/syslog/httpd/error_log

2- See syslog from other AIX’s boxes

(lpar):[root] /tmp ->  multitail -l 'ssh lpar1 "tail -f /var/syslog/syslog.log"' -l 'ssh lpar2 "tail -f /var/syslog/syslog.log"'

3- Tail two logs at the same time with a label at the beginnig:

(lpar):[root] /tmp ->  multitail --label " SYSLOG ->" -i /var/syslog/syslog.log --label "   CRON ->" -i /var/adm/cron/log

4- Tail syslog.log and output of ‘errpt’ command.

(lpar):[root] /tmp -> multitail -R 2 -l "errpt" -i /var/syslog/syslog.log

5- More help

(lpar):[root] multitail -h

And on internet of course. 😉

Just thanks if the post was helpful 🙂

Advertisements

About igalvarez

More than 20 years experiences on IT industry. 20+ years in Unix experience : IBM-AIX, HMC/SDMC/IVM, SVC, Protectier, PureFlex Systems, VIOservers, IBM Bladecenters, IBM System Power, RedHat Linux, SuSE Linux, Debian/Ubuntu Linux. Solaris, SCO Unix, Tru64 Unix, Linux Virtualization: XEN, KVM, databases sb2, oracle, postgreSQL, Find more information here: http://en.gravatar.com/igalvarez
This entry was posted in AIX, multitail, syslog and tagged , , . Bookmark the permalink.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s