snmptrapd traphandle configuration example

This article is going to show the basic configuration of snmptrapd and it's traphandle command.

Assumptions:
snmptrapd is running on a linux host named "test-centos";
The host sending snmptrap messages in this example is named "test-zfs-host"

Now first we're going to set snmptrapd up on the snmptrap server side:

###Server side
[root@test-centos snmp]# cat /etc/snmp/snmptrapd.conf
#traphandle default /bin/mail -s "snmpdtrapd messages" <put your mail address here>
traphandle default /root/lognotify
authCommunity log,execute,net public

[root@test-centos snmp]# service snmptrapd restart

[root@test-centos snmp]# cat /root/lognotify
#!/bin/bash
read host
read ip
vars=

while read oid val
do
if [ "$vars" = "" ]
then
vars="$oid = $val"
else
vars="$vars, $oid = $val"
fi
done
echo trap: $host $ip $vars >/var/tmp/snmptrap.out

And to test whether snmptrapd is working as expected:

###On client side
snmptrap -v2c -c public test-centos:162 "" SNMPv2-MIB::sysDescr SNMPv2-MIB::sysDescr.0 s "test-zfs-host test-zfs-host.ip this is a test snmptrap string"

And after running this command, you can have a check of /var/tmp/snmptrap.out on the snmptrapd server side(test-centos):

[root@test-centos ~]# cat /var/tmp/snmptrap.out

PS:
If you're using sun zfs head, you can set snmptrap destinations in zfs BUI(configuration -> SNMP), here's the snapshot(click to see the larger image):

perl script for monitoring sun zfs memory usage

On zfs's aksh, I can check memory usage with the following:

test-zfs:> status memory show
Memory:
Cache 719M bytes
Unused 15.0G bytes
Mgmt 210M bytes
Other 332M bytes
Kernel 7.79G bytes

So now I want to collect this memory usae information automatically for SNMP's use. Here's the steps:

cpan> o conf prerequisites_policy follow
cpan> o conf commit

Since the host is using proxy to get on the internet, so in /etc/wgetrc:

http_proxy = http://www-proxy.us.example.com:80/
ftp_proxy = http://www-proxy.us.example.com:80/
use_proxy = on

Now install the Net::SSH::Perl perl module:

PERL_MM_USE_DEFAULT=1 perl -MCPAN -e 'install Net::SSH::Perl'

And to confirm that Net::SSH::Perl was installed, run the following command:

perl -e 'use Net::SSH::Perl' #no output is good, as it means the package was installed successfully

Now here goes the perl script to get the memory usage of sun zfs head:

[root@test-centos ~]# cat /var/tmp/mrtg/zfs-test-zfs-memory.pl
#!/usr/bin/perl
use strict;
use warnings;
use Net::SSH::Perl;
my $host = 'test-zfs';
my $user = 'root';
my $password = 'password';

my $ssh = Net::SSH::Perl->new($host);
$ssh->login($user,$password);
my ($stdout,$stderr,$exit) = $ssh->cmd("status memory show");
$ssh->cmd("exit");
if($stderr){
print "ErrorCode:$exit\n";
print "ErrorMsg:$stderr";
} else {
my @std_arr = split(/\n/, $stdout);
shift @std_arr;
foreach(@std_arr) {
if ($_ =~ /.+\b\s+(.+)M\sbytes/){
$_=$1/1024;
}
elsif($_ =~ /.+\b\s+(.+)G\sbytes/){
$_=$1;
}
else{}
}
foreach(@std_arr) {
print $_."\n";
}
}
exit $exit;

PS:
If you get the following error messages during installation of a perl module:

[root@test-centos ~]# perl -MCPAN -e 'install SOAP::Lite'
CPAN: Storable loaded ok
CPAN: LWP::UserAgent loaded ok
Fetching with LWP:
ftp://ftp.perl.org/pub/CPAN/authors/01mailrc.txt.gz
LWP failed with code[500] message[LWP::Protocol::MyFTP: connect: Connection timed out]
Fetching with Net::FTP:
ftp://ftp.perl.org/pub/CPAN/authors/01mailrc.txt.gz

Trying with "/usr/bin/links -source" to get
ftp://ftp.perl.org/pub/CPAN/authors/01mailrc.txt.gz
ELinks: Connection timed out

Then you may have a check of whether you're using proxy to get on the internet(run cpan > o conf init to re-configure cpan; later you should set /etc/wgetrc: http_proxy, ftp_proxy, use_proxy).

 

resolved - yum returned Segmentation fault error on centos

The following error messages occurred while running yum list or yum update on a centos/rhel host:

[root@test-centos ~]# yum list
Loaded plugins: rhnplugin, security
This system is not registered with ULN.
ULN support will be disabled.
Segmentation fault

As always, I did a strace on this:

[root@test-centos ~]# strace yum list
open("/var/cache/yum/el5_ga_base/primary.xml.gz", O_RDONLY) = 6
lseek(6, 0, SEEK_CUR) = 0
read(6, "\37\213\10\10\0\0\0\0\2\377/u01/basecamp/www/el5_"..., 8192) = 8192
--- SIGSEGV (Segmentation fault) @ 0 (0) ---
+++ killed by SIGSEGV +++

And here's the error messages from /var/log/messages:

[root@test-centos ~]# tail /var/log/messages
Jan 6 07:07:44 test-centos kernel: yum[5951]: segfault at 3500000000 ip 000000350cc79e0a sp 00007fff05633b78 error 4 in libc-2.5.so[350cc00000+14e000]

After some googling, I found the yum "Segmentation fault" was caused by the conflict between zlib and yum. To resolve this problem, we need use the older version of zlib. Here's the detailed steps:

[root@test-centos ~]# cd /usr/lib
[root@test-centos lib]# ls -l libz*
-rw-r--r-- 1 root root 125206 Jul 9 07:40 libz.a
lrwxrwxrwx 1 root root 22 Aug 2 07:10 libz.so -> /usr/lib/libz.so.1.2.7
lrwxrwxrwx 1 root root 22 Aug 2 07:10 libz.so.1 -> /usr/lib/libz.so.1.2.7
-rwxr-xr-x 1 root root 75028 Jun 7 2007 libz.so.1.2.3
-rwxr-xr-x 1 root root 99161 Jul 9 07:40 libz.so.1.2.7

[root@test-centos lib]# rm libz.so libz.so.1
rm: remove symbolic link `libz.so'? y
rm: remove symbolic link `libz.so.1'? y
[root@test-centos lib]# ln -s libz.so.1.2.3 libz.so
[root@test-centos lib]# ln -s libz.so.1.2.3 libz.so.1

After these steps, you should now able to run yum commands without any issue.

Also, after using yum, you should change back zlib to the newer version, and here's the steps:

[root@test-centos ~]# cd /usr/lib
[root@test-centos lib]# rm libz.so libz.so.1
rm: remove symbolic link `libz.so'? y
rm: remove symbolic link `libz.so.1'? y
[root@test-centos lib]# ln -s libz.so.1.2.7 libz.so
[root@test-centos lib]# ln -s libz.so.1.2.7 libz.so.1

PS:

  • public-yum-ol7.repo has issue which can lead to the error, you can remove it and try epel. You can run "strace yum list" then search for "No such file".
  • Sometimes, you need update "yum" package itself to resolve this. Downloaded from yum repo, and use rpm to upgrade it.

zfs iops on nfs iscsi disk

On zfs storage 7000 series BUI, you may found the following statistic:

This may seem quite weird as you can see that, NFSv3(3052) + iSCSI(1021) is larger than Disk(1583). As iops for protocal NFSv3/iSCSI finally goes to Disk, so why iops for the two protocals is larger than Disk iops?

Here's the reason:

Disk operations for NFSv3 and iSCSI are logical operations. These logical operations are then combined/optimized by sun zfs storage and then finally go to physical Disk operations.

PS:

1.When doing continuous access to disks(like VOD), disk throughputs will become the bottleneck of performance rather than IOPS. In constract, IOPS limits disk performance when random access is going on disks.

2.For NAS performance analytic, here are two good articles(in Chinese) http://goo.gl/Q2M7JE http://www.storageonline.com.cn/storage/nas/the-nas-performance-analysis-overview/

3.You may also wonder why Disk iops can be as high as 1583. As this number is the sum of all disk controllers of the zfs storage system. Here's some ballpark numbers for HDD iops:

 

zfs shared lun stoage set up for oracle RAC - using iscsiadm

  • create iSCSI Target Group

Open zfs BUI, navigate through "Configuration" -> "SAN" -> "iSCSI Targets". Then create new iSCSI Target by clicking plus sign. Give it an alias, and then select the Network interface(may be bond or LACP) you want to use(check it from "Configuration" -> "Network" and "Configuration" -> "Cluster"). After creating this iSCSI target, drag the newly created target to the right side "iSCSI Target Groups" to create one iSCSI Target Group. You can give that iSCSI target group an name too. Note down the iSCSI Target Group's iqn, this is important for later operations.(Network interfaces:use NAS interface. You can select multiple interfaces)

  • create iSCSI Initiator Group

Before going on the next step, we need first get the iSCSI initiator IQN for each hosts we want LUN allocated. On each host, execute the following command to get the iqn for iscsi on linux platform(You can edit this file before read it, for example, make iqn name ended with` hostname` so it's easier for later operations on LUN<do a /etc/init.d/iscsi restart after your modification to initiatorname.iscsi>. You should first ensure package iscsi-initiator-utils is installed):

[root@test-host ~]# cat /etc/iscsi/initiatorname.iscsi
InitiatorName=<your host's iqn name>

Now go back to zfs BUI, navigate through "Configuration" -> "SAN" -> "Initiators". On the left side, click "iSCSI Initiators", then click plus sign on it. Enter IQN you get from previos step and give it an name.(do this for each host you want iSCSI LUN allocated). After this, drag the newly created iSCSI initiator(s) from left side to form new iSCSI Initiator Groups on the right side(drag two items from the left to the same item on the right to form an group).


  • create shared LUNs for iSCSI Initiator Group

After this, we need now create LUNs for iSCSI Initiator Group(so that shared lun can be allocated, for example, oracle RAC need shared storage). Click on diskette sign on the just created iSCSI Initiator Group,select the project you want the LUN allocated from, give it a name, and assign the volume size. Select the right target group you created before(you can also create a new one e.g. RAC in shares).

PS:You can also now go to "shares" -> "Luns" and create Lun(s) using the target group you created and use default Initiator group. Note that one LUN need one iSCSI target. So you should create more iSCSI targets and add them to iSCSI target group if you want more LUNs.

  • scan shared LUNs from hosts

Now we're going to operate on linux hosts. On each host you want iSCSI LUN allocated, do the following steps:

iscsiadm -m discovery -t st -p <ip address of your zfs storage>(use cluster's ip if there's zfs cluster) #Discover available targets from a discovery portal
iscsiadm -m node -T <variable, iSCSI Target Group iqn> -p <ip address of your zfs storage> -l #Log into a specific target. Or use output from above command($1 is --portal, $3 is --targetname, -l is --login). Use -u to log out for a specified record.

If there's error when handling with multipath devices (more info here <udevadm>):

iscsiadm: default: 1 session requested, but 1 already present.
iscsiadm: Could not log into all portals

Then:

#iscsiadm -m session --rescan

#iscsiadm -m node --loginall=all

#iscsiadm -m discovery -t sendtargets -p <ZFS IP>

#iscsiadm -m node --loginall=all

iscsiadm -m session --rescan

#iscsiadm -m node --loginall=all

After these steps, you host(s) should now see the newly allocated iSCSI LUN(s), you can run fdisk -l to confirm.

PS:

  • To remove discovered targets on one specific portal(or ZFS appliance):

iscsiadm -m discovery -t st -p 192.168.25.175

ls -l /var/lib/iscsi/nodes/

iscsiadm -m node -T <iscsi target group iqn> -p 192.168.25.175 -u

iscsiadm -m discovery -o show

iscsiadm -m node -o delete -p 192.168.25.175:3260

ls -l /var/lib/iscsi/nodes/

  • Here's more about iscsiadm:

iscsiadm -m session -P 1 #Display list of all current sessions logged in, -P for level(1 to 3). From here, we can know which target are the local disks mapped to(e.g. /dev/sde is mapped to target iqn.1986-03.com.sun:02:f82d223a-ca68-e9ac-87d5-95b079b2ba05, then we can know the NAS ZFS appliance name. On ZFS appliance, after got the target group name, we can check which LUNs use that target group, thus we know the mapping between local iscsi disk and ZFS LUN)

iscsiadm -m session --rescan #rescan all sessions

iscsiadm -m session -r SID --rescan #rescan a specific session #SID can be got from iscsiadm -m session -P 1

iscsiadm -m node -T targetname -p ipaddress -u #Log out of a specific target

If you want to remove specific iSCSI LUN from system, then do the following:

cd /sys/class/iscsi_session/session<SID>/device/target<scsi host number>:0:0/<scsi host number>:0:0:<Lun number>
echo 1 > delete

iscsiadm -m node -T targetname -p ipaddress #Display information about a target
iscsiadm -m node -s -T targetname -p ipaddress #Display statistics about a target

iscsiadm -m discovery -o show #View iSCSI database regarding discovery
iscsiadm -m node -o show #View iSCSI database regarding targets to log into
iscsiadm -m session -o show #View iSCSI database regarding sessions logged into
multipath -ll #View if the targets are multipathed (MPIO)

You can find more information about iscsi disk in /sys/class/{scsi_device, scsi_disk, scsi_generic,scsi_host} and /sys/block/ after get the info from iscsiadm -m session -P 3.

If you want to set snapshot on all shares, then try below:

cd /
shares select project1 select share1 snapshots automatic create
set frequency=day hour=21 minute=49 keep=7
commit -- there's no extra empty lines here
cd /
shares select project1 select share1 snapshots automatic create
set frequency=day hour=21 minute=49 keep=7
commit

Here's the CMD equivalent of this article(aksh of oracle ZFS appliance):

shares project rac186187
set mountpoint=/export/rac186187
set quota=1T
set readonly=true
set default_permissions=777

set default_user=root

set default_group=root
set sharenfs="sec=sys,ro=test.example.com,rw=@10.240.72.0/21:@10.240.22.0/21,root=@10.240.72.0/21"
#get reservation
#get pool
#get snapdir #snapdir = visible
#get default_group #default_group = root
#get default_user #default_user = root
#get exported #exported = true
commit

configuration net interfaces list
configuration net datalinks list

configuration san iscsi targets create rac186187
set alias=rac186187
set interfaces=aggr93001
commit

configuration san iscsi targets list
configuration san iscsi targets groups create rac186187
set name=rac186187
set targets=iqn.1986-03.com.sun:02:0a3dec0f-b830-c17a-c957-dd2dc1755a16
commit

[cat /etc/iscsi/initiatorname.iscsi] -> iqn.1988-12.com.oracle.testhost186, iqn.1988-12.com.oracle.testhost187
configuration san iscsi initiators create testhost186
set alias=testhost186
set initiator=iqn.1988-12.com.oracle.testhost186
commit

configuration san iscsi initiators create testhost187
set alias=testhost187
set initiator=iqn.1988-12.com.oracle.testhost187
commit

configuration san iscsi initiators list
configuration san iscsi initiators groups create rac186187
set name=rac186187
set initiators=iqn.1988-12.com.oracle.testhost186,iqn.1988-12.com.oracle.testhost187
commit

shares select rac186187 #project must set readonly=false
lun rac186187
set volsize=500G
set targetgroup=rac186187
set initiatorgroup=rac186187
commit

And if you want to create one share, here's the way:

shares select <project name> #to show current properties, run "show"
filesystem <new share name>
set quota=100G
set quota_snap=false
set reservation=50G
set reservation_snap=false
set root_permissions=777
set root_user=root
set root_group=root
commit
cd /

To modify role permissions:

zfs02:configuration roles devroot authorizations> create

zfs02:configuration roles devroot auth (uncommitted)> set scope= <tab>
ad appliance dataset keystore net schema stmf update workflow
alert cluster hardware nas role stat svc user worksheet

zfs02:configuration roles devroot auth (uncommitted)> set scope=alert
scope = alert

zfs02:configuration roles devroot auth (uncommitted)> show
Properties:
scope = alert
allow_configure = false

zfs02:configuration roles devroot auth (uncommitted)> set allow_configure=true
allow_configure = true (uncommitted)

zfs02:configuration roles devroot auth (uncommitted)> show
Properties:
scope = alert
allow_configure = true (uncommitted)

zfs02:configuration roles devroot auth (uncommitted)> commit

To search for logs, you can try below:

ssh root@testzfs|tee log.txt

testzfs:> maintenance logs select alert list -a
testzfs:> maintenance logs select audit list -a
testzfs:> maintenance logs select fltlog list -a
testzfs:> maintenance logs select scrk list -a
testzfs:> maintenance logs select system list -a

Then grep <entry> log.txt

PPS:

Suppose we know that there is one session logged in to target iqn.1986-03.com.sun:02:a7a92623-dfc2-eab0-cf84-be0b549dc5c9, then how can we add LUNs or change the size of existed LUN on ZFS?

First, to change the size of existing LUN:

[root@testhost1 ~]# iscsiadm -m session
tcp: [2] 10.240.79.34:3260,2 iqn.1986-03.com.sun:02:a7a92623-dfc2-eab0-cf84-be0b549dc5c9

Log on ZFS UI, go to Configuration, SAN, iSCSI, Targets, search for "iqn.1986-03.com.sun:02:a7a92623-dfc2-eab0-cf84-be0b549dc5c9" which is the target name, then you'll find the target and target group it belongs to. Note down the target group name, e.g. RAC01, then go to Shares, LUNs, click on the LUN and change the size as needed.

To add new LUN to the host:

First, find the iscsi initiator name on this host testhost1-p:

[root@testhost1 ~]# cat /etc/iscsi/initiatorname.iscsi
InitiatorName=iqn.1988-12.com.oracle.testhost1

Log on ZFS UI, go to Configuration, SAN, iSCSI, Initiators, search "iqn.1988-12.com.oracle.testhost1", you'll find the initiator and initiator group. From here you can click the diskette icon and add new LUN. Make sure to select the right target group you got from previous step.

SUN zfs storage 7320 monitoring using net-snmp and mrtg

This article is going to talk about zfs storage 7320 monitoring using net-snmp and mrtg. Although the monitored system is sun zfs storage 7320, you'll find the main idea of this article can be applied to many different system monitoring, including but not limited to cpu usage/network/bandwidth/disk/temperature of cisco switches, other linux systems and even windows systems.

As net-snmp extending agent functionality is not supported on sun zfs storage 7320 which is solaris 11 express system, so I'm going to monitor sun zfs storage through using of one linux snmp client by writing monitoring scripts on that linux client rather than on zfs itself.

Now, here goes the steps:

Part 1 - set up snmp client and mrtg on a linux host

yum -y install gcc-* gd-* libpng-* zlib-* httpd
yum -y install net-snmp* net-snmp-libs lm_sensors lm_sensors-devel
yum -y install mrtg
cp /etc/snmp/snmpd.conf{,.bak}
echo "rocommunity public" > /etc/snmp/snmpd.conf
mkdir -p /etc/mrtg
chkconfig --level 2345 snmpd on
service snmpd start

Ensure snmpd is listening:

netstat -tunlp |grep snmp
tcp 0 0 127.0.0.1:199 0.0.0.0:* LISTEN 26427/snmpd
udp 0 0 0.0.0.0:161 0.0.0.0:* 26427/snmpd

And let's have a test to make sure snmp client is working as expected:

[root@test-centos mrtg]# snmpwalk -v2c -c public localhost interface
IF-MIB::ifNumber.0 = INTEGER: 4
IF-MIB::ifIndex.1 = INTEGER: 1
IF-MIB::ifIndex.2 = INTEGER: 2
IF-MIB::ifIndex.3 = INTEGER: 3
IF-MIB::ifIndex.4 = INTEGER: 4
IF-MIB::ifDescr.1 = STRING: lo
IF-MIB::ifDescr.2 = STRING: eth0
IF-MIB::ifDescr.3 = STRING: eth1
......
......

Now it's time to configure mrtg on linux:

[root@test-centos mrtg]# cat /etc/httpd/conf.d/mrtg.conf
Alias /mrtg /var/www/mrtg
<Location /mrtg>
Order deny,allow
Allow from all
</Location>

Now, do a httpd restart:

[root@test-centos ~]# apachectl restart

Part 2 - configure snmp on SUN zfs storage 7320 web UI

Log on SUN zfs storage 7320 web UI, navigate through "Configuration" -> "SNMP", and configure as the following:

After this, you'll need enable/restart SNMP service on zfs.
Now do a snmpwalk from snmp linux client to SUN zfs storage:

[root@test-centos ~]# snmpwalk -v2c -c public test-zfs-host interface
IF-MIB::ifNumber.0 = INTEGER: 6
IF-MIB::ifIndex.1 = INTEGER: 1
IF-MIB::ifIndex.4 = INTEGER: 4
IF-MIB::ifIndex.5 = INTEGER: 5
IF-MIB::ifIndex.6 = INTEGER: 6
IF-MIB::ifIndex.7 = INTEGER: 7
IF-MIB::ifIndex.8 = INTEGER: 8
IF-MIB::ifDescr.1 = STRING: lo0
......
......

  • Part 3 - extending snmp mibs/oids on snmp client(the linux host)

As stated at the beginning of this article, net-snmp extending agent functionality is not supported on sun zfs storage 7320. So I'm going to monitor sun zfs storage through using of one linux snmp client by writing monitoring scripts on that linux client rather than on zfs itself.

To avoid ssh asking for password when connecting from the linux host, you need add the linux host's pubkey to sun zfs storage:
On snmp Linux client:

[root@test-centos ~]# ssh-keygen -t rsa #if you already have pubkey, skip this step
[root@test-centos ~]# cat /root/.ssh/id_rsa.pub |awk '{print $2}'
AAAAB3NzaC1yc2EAAAABIwAAAQEAxYd97A/V5RwdkfzbkmYBqF189pTLOlbYt0dZzO395dfU0Sp/Ykrk+sOJO0bJZEtytuTcCz/bVutWB7vLzeQPxIToRUQnZX7ZoMsjyaFk3LhtAgFhYIycOw2FQL8Qvb5yMBASB2/KthsqaiNqOP/2Vy5e0aCFFIV5DlKQTp/3eceSMq8kTx+e801lZow++yT70rp3p+5WtriN/NKYI0B3cpSQY/36D/TcOF9v5IaqQokp/mLRoc1MLOhN0sy0ipCdT+0bbkZ4Lh8bEeQO48UGKEOnYrYto33tay4mZk8HPWFK4w/TQGxBLthiuPQ4oZzG3gVpQUS4GRwI9zZoGtgELQ==

On Sun zfs storage:
Click "Configuration", then add the snmp client's public key(ensure RSA is selected):

Do a ssh connection from snmp linux client to the sun zfs storage host, it should now not asking for password.(ensure to do this, as there's possibility that sun zfs's key is not on the linux client as you may never have connected from that linux client to the sun zfs storage system)

Now we're to the most important part. Assume we want to monitoring space usage on the SUN zfs storage system, to do this, you'll need do the following on the snmp linux client:

[root@test-centos mrtg]# cat /etc/snmp/snmpd.conf
rocommunity public
extend .1.3.6.1.4.1.2021.31 zfs-test-zfs-host-total /bin/bash /var/tmp/mrtg/zfs-test-zfs-host-total.sh
extend .1.3.6.1.4.1.2021.32 zfs-test-zfs-host-used /bin/bash /var/tmp/mrtg/zfs-test-zfs-host-used.sh

[root@test-centos ~]# cat /var/tmp/mrtg/zfs-test-zfs-host-used.sh
#!/bin/bash
_used=`ssh test-zfs-host "status ls"|grep Used|awk '{print gensub('/T/',"","g",$2)}'`
echo $_used;

[root@test-centos ~]# cat /var/tmp/mrtg/zfs-test-zfs-host-total.sh
_used=`ssh test-zfs-host "status ls"|grep Used|awk '{print gensub('/T/',"","g",$2)}'` #I trimed 'T', you may need modify this to meet your environment
_avail=`ssh test-zfs-host "status ls"|grep Avail|awk '{print gensub('/T/',"","g",$2)}'` #I trimed 'T', you may need modify this to meet your environment
_all=`echo $_used + $_avail|bc`
echo $_all;

[root@test-centos ~]# chmod +x /var/tmp/mrtg/zfs-test-zfs-host-used.sh
[root@test-centos ~]# chmod +x /var/tmp/mrtg/zfs-test-zfs-host-total.sh

Now, let's do a snmp restart on snmp linux client and then test the newly added OIDs:

[root@test-centos ~]# service snmpd restart

[root@test-centos ~]# snmpwalk -v2c -c public localhost .1.3.6.1.4.1.2021.32
UCD-SNMP-MIB::ucdavis.32.1.0 = INTEGER: 1
UCD-SNMP-MIB::ucdavis.32.2.1.2.19.122.102.115.45.115.108.99.101.49.48.115.110.48.49.45.117.115.101.100 = STRING: "/bin/bash"
UCD-SNMP-MIB::ucdavis.32.2.1.3.19.122.102.115.45.115.108.99.101.49.48.115.110.48.49.45.117.115.101.100 = STRING: "/var/tmp/mrtg/zfs-test-zfs-host-used.sh"
UCD-SNMP-MIB::ucdavis.32.2.1.4.19.122.102.115.45.115.108.99.101.49.48.115.110.48.49.45.117.115.101.100 = ""
UCD-SNMP-MIB::ucdavis.32.2.1.5.19.122.102.115.45.115.108.99.101.49.48.115.110.48.49.45.117.115.101.100 = INTEGER: 5
UCD-SNMP-MIB::ucdavis.32.2.1.6.19.122.102.115.45.115.108.99.101.49.48.115.110.48.49.45.117.115.101.100 = INTEGER: 1
UCD-SNMP-MIB::ucdavis.32.2.1.7.19.122.102.115.45.115.108.99.101.49.48.115.110.48.49.45.117.115.101.100 = INTEGER: 1
UCD-SNMP-MIB::ucdavis.32.2.1.20.19.122.102.115.45.115.108.99.101.49.48.115.110.48.49.45.117.115.101.100 = INTEGER: 4
UCD-SNMP-MIB::ucdavis.32.2.1.21.19.122.102.115.45.115.108.99.101.49.48.115.110.48.49.45.117.115.101.100 = INTEGER: 1
UCD-SNMP-MIB::ucdavis.32.3.1.1.19.122.102.115.45.115.108.99.101.49.48.115.110.48.49.45.117.115.101.100 = STRING: "9.31"
UCD-SNMP-MIB::ucdavis.32.3.1.2.19.122.102.115.45.115.108.99.101.49.48.115.110.48.49.45.117.115.101.100 = STRING: "9.31"
UCD-SNMP-MIB::ucdavis.32.3.1.3.19.122.102.115.45.115.108.99.101.49.48.115.110.48.49.45.117.115.101.100 = INTEGER: 1
UCD-SNMP-MIB::ucdavis.32.3.1.4.19.122.102.115.45.115.108.99.101.49.48.115.110.48.49.45.117.115.101.100 = INTEGER: 0
UCD-SNMP-MIB::ucdavis.32.4.1.2.19.122.102.115.45.115.108.99.101.49.48.115.110.48.49.45.117.115.101.100.1 = STRING: "9.31"
[root@test-centos ~]# snmpwalk -v2c -c public localhost .1.3.6.1.4.1.2021.31
UCD-SNMP-MIB::ucdavis.31.1.0 = INTEGER: 1
UCD-SNMP-MIB::ucdavis.31.2.1.2.20.122.102.115.45.115.108.99.101.49.48.115.110.48.49.45.116.111.116.97.108 = STRING: "/bin/bash"
UCD-SNMP-MIB::ucdavis.31.2.1.3.20.122.102.115.45.115.108.99.101.49.48.115.110.48.49.45.116.111.116.97.108 = STRING: "/var/tmp/mrtg/zfs-test-zfs-host-total.sh"
UCD-SNMP-MIB::ucdavis.31.2.1.4.20.122.102.115.45.115.108.99.101.49.48.115.110.48.49.45.116.111.116.97.108 = ""
UCD-SNMP-MIB::ucdavis.31.2.1.5.20.122.102.115.45.115.108.99.101.49.48.115.110.48.49.45.116.111.116.97.108 = INTEGER: 5
UCD-SNMP-MIB::ucdavis.31.2.1.6.20.122.102.115.45.115.108.99.101.49.48.115.110.48.49.45.116.111.116.97.108 = INTEGER: 1
UCD-SNMP-MIB::ucdavis.31.2.1.7.20.122.102.115.45.115.108.99.101.49.48.115.110.48.49.45.116.111.116.97.108 = INTEGER: 1
UCD-SNMP-MIB::ucdavis.31.2.1.20.20.122.102.115.45.115.108.99.101.49.48.115.110.48.49.45.116.111.116.97.108 = INTEGER: 4
UCD-SNMP-MIB::ucdavis.31.2.1.21.20.122.102.115.45.115.108.99.101.49.48.115.110.48.49.45.116.111.116.97.108 = INTEGER: 1
UCD-SNMP-MIB::ucdavis.31.3.1.1.20.122.102.115.45.115.108.99.101.49.48.115.110.48.49.45.116.111.116.97.108 = STRING: "16.32"
UCD-SNMP-MIB::ucdavis.31.3.1.2.20.122.102.115.45.115.108.99.101.49.48.115.110.48.49.45.116.111.116.97.108 = STRING: "16.32"
UCD-SNMP-MIB::ucdavis.31.3.1.3.20.122.102.115.45.115.108.99.101.49.48.115.110.48.49.45.116.111.116.97.108 = INTEGER: 1
UCD-SNMP-MIB::ucdavis.31.3.1.4.20.122.102.115.45.115.108.99.101.49.48.115.110.48.49.45.116.111.116.97.108 = INTEGER: 0
UCD-SNMP-MIB::ucdavis.31.4.1.2.20.122.102.115.45.115.108.99.101.49.48.115.110.48.49.45.116.111.116.97.108.1 = STRING: "16.32"

From the output, we can see that OIDs UCD-SNMP-MIB::ucdavis.32.3.1.1.19.122.102.115.45.115.108.99.101.49.48.115.110.48.49.45.117.115.101.100 (used space) and UCD-SNMP-MIB::ucdavis.31.3.1.1.20.122.102.115.45.115.108.99.101.49.48.115.110.48.49.45.116.111.116.97.108 (total space) are the two OIDs we want.

  • Part 4 - do the mrtg drawing

As we got the OIDs we want, it's now easier for us to do the mrtg drawing. On the snmp linux host, do the following steps:

[root@test-centos ~]# cat /etc/mrtg/test-zfs-host.cfg
#LoadMIBs: /usr/share/snmp/mibs/UCD-SNMP-MIB.txt,/usr/share/snmp/mibs/TCP-MIB.txt
workdir: /var/www/mrtg/
Title[zfs_space_test-zfs-host]: Percentage used space on zfs
PageTop[zfs_space_test-zfs-host]: <h1>Percentage used space on zfs</h1>
Target[zfs_space_test-zfs-host]: .1.3.6.1.4.1.2021.32.3.1.1.19.122.102.115.45.115.108.99.101.49.48.115.110.48.49.45.117.115.101.100&.1.3.6.1.4.1.2021.31.3.1.1.20.122.102.115.45.115.108.99.101.49.48.115.110.48.49.45.116.111.116.97.108:public@localhost
Options[zfs_space_test-zfs-host]: growright,gauge,transparent,nopercent
Unscaled[zfs_space_test-zfs-host]: ymwd
MaxBytes[zfs_space_test-zfs-host]: 100
YLegend[zfs_space_test-zfs-host]: UsedSpace %
ShortLegend[zfs_space_test-zfs-host]: T
LegendI[zfs_space_test-zfs-host]: Used
LegendO[zfs_space_test-zfs-host]: Total
Legend1[zfs_space_test-zfs-host]: Percentage used space on zfs
Legend2[zfs_space_test-zfs-host]: Percentage all space on zfs

PS:
You need replace "UCD-SNMP-MIB::ucdavis" with .1.3.6.1.4.1.2021 or you'll get error messages like the following:

[root@test-centos ~]# env LANG=C TZ=Asia/Shanghai /usr/bin/mrtg /etc/mrtg/test-zfs-host.cfg
Argument "v4only" isn't numeric in int at /usr/bin/../lib64/mrtg2/SNMP_Session.pm line 183.
backoff (v4only) must be a number >= 1.0 at /usr/bin/../lib64/mrtg2/SNMP_util.pm line 465

Let's continue:

[root@test-centos ~]# env LANG=C TZ=Asia/Shanghai /usr/bin/mrtg /etc/mrtg/test-zfs-host.cfg
24-12-2012 01:46:41, Rateup WARNING: /usr/bin/rateup could not read the primary log file for zfs_space_test-zfs-host
24-12-2012 01:46:41, Rateup WARNING: /usr/bin/rateup The backup log file for zfs_space_test-zfs-host was invalid as well
24-12-2012 01:46:41, Rateup WARNING: /usr/bin/rateup Can't remove zfs_space_test-zfs-host.old updating log file
24-12-2012 01:46:41, Rateup WARNING: /usr/bin/rateup Can't rename zfs_space_test-zfs-host.log to zfs_space_test-zfs-host.old updating log file

[root@test-centos ~]# env LANG=C TZ=Asia/Shanghai /usr/bin/mrtg /etc/mrtg/test-zfs-host.cfg
24-12-2012 01:46:46, Rateup WARNING: /usr/bin/rateup Can't remove zfs_space_test-zfs-host.old updating log file

[root@test-centos ~]# env LANG=C TZ=Asia/Shanghai /usr/bin/mrtg /etc/mrtg/test-zfs-host.cfg

[root@test-centos ~]# indexmaker --output=/var/www/mrtg/index.html /etc/mrtg/test-zfs-host.cfg

Now add cronjob:

0-59/5 * * * * env LANG=C TZ=Asia/Shanghai /usr/bin/mrtg /etc/mrtg/test-zfs-host.cfg

Visit http://<your linux box's ip address>/mrtg/ to get the GUI result(you'll wait 5 minutes for the initial result, be patient!)

  • Part 5 - troubleshooting

  • If you met error messages like the following:

[root@test-centos ~]# env LANG=C TZ=Asia/Shanghai /usr/bin/mrtg /etc/mrtg/test-zfs-host.cfg
Argument "v4only" isn't numeric in int at /usr/bin/../lib64/mrtg2/SNMP_Session.pm line 183.
backoff (v4only) must be a number >= 1.0 at /usr/bin/../lib64/mrtg2/SNMP_util.pm line 465

That's because you're using OIDs/MIBs alias rather than number. Change alias to number, i.e. change UCD-SNMP-MIB::ucdavis to .1.3.6.1.4.1.2021, and then re-check.

  • If you met error messages like the following:

[root@test-centos ~]# env LANG=C TZ=Asia/Shanghai /usr/bin/mrtg /etc/mrtg/test-zfs-host.cfg
cannot encode Object ID 34.4.1.2.28.122.102.115.45.115.108.99.101.49.48.115.110.48.49.45.110.105.99.45.98.97.110.100.119.105.100.116.104.3: first subid too big in Object ID 34.4.1.2.28.122.102.115.45.115.108.99.101.49.48.115.110.48.49.45.110.105.99.45.98.97.110.100.119.105.100.116.104.3 at /usr/bin/mrtg line 2035
Tuesday, 25 December 2012 at 0:34: ERROR: Target[zfs-test-zfs-host-io-average-latency-igb2][_IN_] ' $target->[3]{$mode} ' did not eval into defined data
Tuesday, 25 December 2012 at 0:34: ERROR: Target[zfs-test-zfs-host-io-average-latency-igb2][_OUT_] ' $target->[3]{$mode} ' did not eval into defined data

Then you should carefully check the scripts you write for the OIDs, and run several times of snmpwalk to ensure the values are correct(is your script's output variable, this may cause problems)

  • If you met error messages like the following:

[root@test-centos mrtg]# env LANG=C TZ=Asia/Shanghai /usr/bin/mrtg /etc/mrtg/test-zfs-host.cfg
SNMP Error:
Received SNMP response with error code
error status: noSuchName
index 2 (OID: 1.3.6.1.4.1.2021.44)
SNMPv1_Session (remote host: "localhost" [127.0.0.1].161)
community: "public"
request ID: 1786815468
PDU bufsize: 8000 bytes
timeout: 2s
retries: 5
backoff: 1)
at /usr/bin/../lib64/mrtg2/SNMP_util.pm line 490
SNMPGET Problem for .1.3.6.1.4.1.2021.44 .1.3.6.1.4.1.2021.44 sysUptime sysName on public@localhost::::::v4only
at /usr/bin/mrtg line 2035
SNMP Error:
Received SNMP response with error code
error status: noSuchName
index 2 (OID: 1.3.6.1.4.1.2021.45)
SNMPv1_Session (remote host: "localhost" [127.0.0.1].161)
community: "public"
request ID: 1786815469
PDU bufsize: 8000 bytes
timeout: 2s
retries: 5
backoff: 1)
at /usr/bin/../lib64/mrtg2/SNMP_util.pm line 490
SNMPGET Problem for .1.3.6.1.4.1.2021.45 .1.3.6.1.4.1.2021.45 sysUptime sysName on public@localhost::::::v4only
at /usr/bin/mrtg line 2035
Monday, 24 December 2012 at 14:41: ERROR: Target[zfs_space_test-zfs-host][_IN_] '( $target->[0]{$mode} ) * 10000 / ( $target->[1]{$mode} )' (warn): Use of uninitialized value in division (/) at (eval 16) line 1.
Monday, 24 December 2012 at 14:41: ERROR: Target[zfs_space_test-zfs-host][_OUT_] '( $target->[0]{$mode} ) * 10000 / ( $target->[1]{$mode} )' (warn): Use of uninitialized value in division (/) at (eval 17) line 1.

Then one possible culprit is that new net-snmp stop supporting "exec", use "extend" instead and re-try.

Note that you can also use "snmpd -f -Le" to check error messages related to snmpd.

PS:
Here's more links from where you can read more about net-snmp/mrtg:

  1. net-snmp FAQ http://www.net-snmp.org/FAQ.html
  2. mrtg configuration opstions http://oss.oetiker.ch/mrtg/doc/mrtg-reference.en.html
  3. SUN zfs storage SNMP - http://docs.oracle.com/cd/E22471_01/html/820-4167/configuration__services__snmp.html
  4. net-snmp extending agent functionality - http://linux.die.net/man/5/snmpd.conf (search for 'extending agent functionality' on this page)
  5. To get only numbers of snmpwalk, add -On. More info on http://linux.die.net/man/1/snmpcmd
  6. Now here's the image of the baby(click on it to see the larger one:

403 CoachingSessionExceeded - McAfee the culprit

Today I tried access one website within company's network which was very important to me. But the site loaded incompletely and so was not usable.

As always, company network is somehow restricted, for example, only output port 80 and 443 are enabled in my company's network. As many companies do, McAfee software is used to restrict internal employees surfing the internet. So I firstly thought McAfee was the culprit.

As I'm using chrome browser, so I press F12 to open the developer tools. On "Network" tab(you may need ctrl+F5 to forcely refresh the website), I found the following error status:

Then I opened that url source in a new tab, and the following page occured:

So everything sorted! I clicked the button "click here if you have read ....", and then went back to the site, refresh, and later everything was ok!

resolved - error: conflicting types for 'pci_pcie_cap' - Infiniband driver OFED installation

OFED is an abbr for "OpenFabrics Enterprise Distribution". When installing OFED-1.5.4.1 on a centos/RHEL 5.8 linux system, I met the following problem:

In file included from /var/tmp/OFED_topdir/BUILD/ofa_kernel-1.5.4.1/drivers/infiniband/core/notice.c:37:
/var/tmp/OFED_topdir/BUILD/ofa_kernel-1.5.4.1/kernel_addons/backport/2.6.18-EL5.7/include/linux/pci.h:164: error: conflicting types for 'pci_pcie_cap'
include/linux/pci.h:1015: error: previous definition of 'pci_pcie_cap' was here
make[4]: *** [/var/tmp/OFED_topdir/BUILD/ofa_kernel-1.5.4.1/drivers/infiniband/core/notice.o] Error 1
make[3]: *** [/var/tmp/OFED_topdir/BUILD/ofa_kernel-1.5.4.1/drivers/infiniband/core] Error 2
make[2]: *** [/var/tmp/OFED_topdir/BUILD/ofa_kernel-1.5.4.1/drivers/infiniband] Error 2
make[1]: *** [_module_/var/tmp/OFED_topdir/BUILD/ofa_kernel-1.5.4.1] Error 2
make[1]: Leaving directory `/usr/src/kernels/2.6.18-308.4.1.0.1.el5-x86_64'
make: *** [kernel] Error 2
error: Bad exit status from /var/tmp/rpm-tmp.70159 (%build)

After googling and experimenting, I can tell you the definite resolution for this problem -> install another version of OFED, OFED-1.5.3.2. Go to the following official site to download: http://www.openfabrics.org/downloads/OFED/ofed-1.5.3/

PS:

1. Here's the file of OFED installation howto: OFED installation README.txt

2.Some Infiniband commands that may be useful for you:

service openibd status/start/stop
lspci -k|grep -i infini
ibnetdiscover
ibhosts
ibswitches
ibv_devinfo #Verify the status of ports by using ibv_devinfo: all connected ports should report a "PORT_ACTIVE" state

java plugin of firefox installation on linux platform

Some iLom software(for example oracle iLom) use java jnlp applet(java console) to open remote console of the server. To use this functionality, you need java plugin on firefox installed. Here's the detailed steps:

  • Update firefox: yum update firefox, please note that you better remove firefox 64 bit version and use 32 version ones(reason followed). After this, you'll have firefox 10 on the system.
PS:
If you found some elements not showing in firefox after the upgrading(for example "Launch Remote Console" disappeared), you'll need install Stylish addon in firefox and then create the following new style using code below:
@namespace url(http://www.w3.org/1999/xhtml);
#mainpage {
visibility: visible !important;
}
  • Install JRE 6: Go to http://www.oracle.com/technetwork/java/javase/downloads/jre6u37-downloads-1859589.html and download jre-6u37-linux-i586-rpm.bin. After that, install the package on the system using rpm -Uvh.(make sure NOT to download 64bits version, as java console only support 32bits java plugin)
  • Make java plugin available to firefox: If you are planning to run firefox under root, do the following steps:

mkdir -p /root/.mozilla/plugins/

ln -s /usr/java/jre1.6.0_37/lib/i386/libnpjp2.so /root/.mozilla/plugins/

PS:

  • If you are using OEL6/RHEL6, then you can use IcedTea Web Start (yum install icedtea-web) instead of javaws. You may find that for some iLOM, only 32bits OEL6/RHEL6 with IcedTea Web start can open the console to mount ISO image.
  • if you've installed java plugin before, you'll first need remove it.
  • Now restart firefox, and go to about:plugins, you'll see java plugin enabled. Open java console and enjoy!(select /usr/bin/javaws if prompted)
  • If you still find the console not prompting up, then it may be caused by java security setting. You can refer to here for WA.
  • If you met error like "No appropriate protocol (protocol is disabled or cipher suites are inappropriate)", then one possibility is that firefox/java/libnpjp2.so are not of the same architecture, you should use i386(or x86_64) for all of them.
  • Here's the steps to install firefox browser under linux platform:

1. Download and install EPEL repo(wget epel-release package on the page below, then install it using "rpm -i <package name>.rpm"):

If you are running an EL7 version, please visit here to get the newest 'epel-release' package for EL7: The newest version of 'epel-release' for EL7

If you are running an EL6 version, please visit here to get the newest 'epel-release' package for EL6: The newest version of 'epel-release' for EL6

If you are running an EL5 version, please visit here to get the newest 'epel-release' package for EL5: The newest version of 'epel-release' for EL5

2. Install firefox:

yum install -y firefox-31.4.0-1.0.1.el5_11

3. Enable newer version of firefox by run firefox in VNC. If you find it's still the old version, then run "killall firefox" as root, and then try again.