This article is going to show the basic configuration of snmptrapd and it’s traphandle command.

Assumptions:
snmptrapd is running on a linux host named “test-centos”;
The host sending snmptrap messages in this example is named “test-zfs-host”

Now first we’re going to set snmptrapd up on the snmptrap server side:

###Server side
[root@test-centos snmp]# cat /etc/snmp/snmptrapd.conf
#traphandle default /bin/mail -s “snmpdtrapd messages” <put your mail address here>
traphandle default /root/lognotify
authCommunity log,execute,net public

[root@test-centos snmp]# service snmptrapd restart

[root@test-centos snmp]# cat /root/lognotify
#!/bin/bash
read host
read ip
vars=

while read oid val
do
if [ "$vars" = "" ]
then
vars=”$oid = $val”
else
vars=”$vars, $oid = $val”
fi
done
echo trap: $host $ip $vars >/var/tmp/snmptrap.out

And to test whether snmptrapd is working as expected:

###On client side
snmptrap -v2c -c public test-centos:162 “” SNMPv2-MIB::sysDescr SNMPv2-MIB::sysDescr.0 s “test-zfs-host test-zfs-host.ip this is a test snmptrap string”

And after running this command, you can have a check of /var/tmp/snmptrap.out on the snmptrapd server side(test-centos):

[root@test-centos ~]# cat /var/tmp/snmptrap.out

PS:
If you’re using sun zfs head, you can set snmptrap destinations in zfs BUI(configuration -> SNMP), here’s the snapshot(click to see the larger image):

This article is going to talk about zfs storage 7320 monitoring using net-snmp and mrtg. Although the monitored system is sun zfs storage 7320, you’ll find the main idea of this article can be applied to many different system monitoring, including but not limited to cpu usage/network/bandwidth/disk/temperature of cisco switches, other linux systems and even windows systems.

As net-snmp extending agent functionality is not supported on sun zfs storage 7320 which is solaris 11 express system, so I’m going to monitor sun zfs storage through using of one linux snmp client by writing monitoring scripts on that linux client rather than on zfs itself.

Now, here goes the steps:

Part 1 – set up snmp client and mrtg on a linux host

yum -y install gcc-* gd-* libpng-* zlib-* httpd
yum -y install net-snmp* net-snmp-libs lm_sensors lm_sensors-devel
yum -y install mrtg
cp /etc/snmp/snmpd.conf{,.bak}
echo “rocommunity public” > /etc/snmp/snmpd.conf
mkdir -p /etc/mrtg
chkconfig –level 2345 snmpd on
service snmpd start

Ensure snmpd is listening:

netstat -tunlp |grep snmp
tcp 0 0 127.0.0.1:199 0.0.0.0:* LISTEN 26427/snmpd
udp 0 0 0.0.0.0:161 0.0.0.0:* 26427/snmpd

And let’s have a test to make sure snmp client is working as expected:

[root@test-centos mrtg]# snmpwalk -v2c -c public localhost interface
IF-MIB::ifNumber.0 = INTEGER: 4
IF-MIB::ifIndex.1 = INTEGER: 1
IF-MIB::ifIndex.2 = INTEGER: 2
IF-MIB::ifIndex.3 = INTEGER: 3
IF-MIB::ifIndex.4 = INTEGER: 4
IF-MIB::ifDescr.1 = STRING: lo
IF-MIB::ifDescr.2 = STRING: eth0
IF-MIB::ifDescr.3 = STRING: eth1
……
……

Now it’s time to configure mrtg on linux:

[root@test-centos mrtg]# cat /etc/httpd/conf.d/mrtg.conf
Alias /mrtg /var/www/mrtg
<Location /mrtg>
Order deny,allow
Allow from all
</Location>

Now, do a httpd restart:

[root@test-centos ~]# apachectl restart

Part 2 – configure snmp on SUN zfs storage 7320 web UI

Log on SUN zfs storage 7320 web UI, navigate through “Configuration” -> “SNMP”, and configure as the following:

After this, you’ll need enable/restart SNMP service on zfs.
Now do a snmpwalk from snmp linux client to SUN zfs storage:

[root@test-centos ~]# snmpwalk -v2c -c public test-zfs-host interface
IF-MIB::ifNumber.0 = INTEGER: 6
IF-MIB::ifIndex.1 = INTEGER: 1
IF-MIB::ifIndex.4 = INTEGER: 4
IF-MIB::ifIndex.5 = INTEGER: 5
IF-MIB::ifIndex.6 = INTEGER: 6
IF-MIB::ifIndex.7 = INTEGER: 7
IF-MIB::ifIndex.8 = INTEGER: 8
IF-MIB::ifDescr.1 = STRING: lo0
……
……

• Part 3 – extending snmp mibs/oids on snmp client(the linux host)

As stated at the beginning of this article, net-snmp extending agent functionality is not supported on sun zfs storage 7320. So I’m going to monitor sun zfs storage through using of one linux snmp client by writing monitoring scripts on that linux client rather than on zfs itself.

To avoid ssh asking for password when connecting from the linux host, you need add the linux host’s pubkey to sun zfs storage:
On snmp Linux client:

[root@test-centos ~]# ssh-keygen -t rsa #if you already have pubkey, skip this step
[root@test-centos ~]# cat /root/.ssh/id_rsa.pub |awk ‘{print $2}’
AAAAB3NzaC1yc2EAAAABIwAAAQEAxYd97A/V5RwdkfzbkmYBqF189pTLOlbYt0dZzO395dfU0Sp/Ykrk+sOJO0bJZEtytuTcCz/bVutWB7vLzeQPxIToRUQnZX7ZoMsjyaFk3LhtAgFhYIycOw2FQL8Qvb5yMBASB2/KthsqaiNqOP/2Vy5e0aCFFIV5DlKQTp/3eceSMq8kTx+e801lZow++yT70rp3p+5WtriN/NKYI0B3cpSQY/36D/TcOF9v5IaqQokp/mLRoc1MLOhN0sy0ipCdT+0bbkZ4Lh8bEeQO48UGKEOnYrYto33tay4mZk8HPWFK4w/TQGxBLthiuPQ4oZzG3gVpQUS4GRwI9zZoGtgELQ==

On Sun zfs storage:
Click “Configuration”, then add the snmp client’s public key(ensure RSA is selected):

Do a ssh connection from snmp linux client to the sun zfs storage host, it should now not asking for password.(ensure to do this, as there’s possibility that sun zfs’s key is not on the linux client as you may never have connected from that linux client to the sun zfs storage system)

Now we’re to the most important part. Assume we want to monitoring space usage on the SUN zfs storage system, to do this, you’ll need do the following on the snmp linux client:

[root@test-centos mrtg]# cat /etc/snmp/snmpd.conf
rocommunity public
extend .1.3.6.1.4.1.2021.31 zfs-test-zfs-host-total /bin/bash /var/tmp/mrtg/zfs-test-zfs-host-total.sh
extend .1.3.6.1.4.1.2021.32 zfs-test-zfs-host-used /bin/bash /var/tmp/mrtg/zfs-test-zfs-host-used.sh

[root@test-centos ~]# cat /var/tmp/mrtg/zfs-test-zfs-host-used.sh
#!/bin/bash
_used=`ssh test-zfs-host “status ls”|grep Used|awk ‘{print gensub(‘/T/’,”",”g”,$2)}’`
echo $_used;

[root@test-centos ~]# cat /var/tmp/mrtg/zfs-test-zfs-host-total.sh
_used=`ssh test-zfs-host “status ls”|grep Used|awk ‘{print gensub(‘/T/’,”",”g”,$2)}’` #I trimed ‘T’, you may need modify this to meet your environment
_avail=`ssh test-zfs-host “status ls”|grep Avail|awk ‘{print gensub(‘/T/’,”",”g”,$2)}’` #I trimed ‘T’, you may need modify this to meet your environment
_all=`echo $_used + $_avail|bc`
echo $_all;

[root@test-centos ~]# chmod +x /var/tmp/mrtg/zfs-test-zfs-host-used.sh
[root@test-centos ~]# chmod +x /var/tmp/mrtg/zfs-test-zfs-host-total.sh

Now, let’s do a snmp restart on snmp linux client and then test the newly added OIDs:

[root@test-centos ~]# service snmpd restart

[root@test-centos ~]# snmpwalk -v2c -c public localhost .1.3.6.1.4.1.2021.32
UCD-SNMP-MIB::ucdavis.32.1.0 = INTEGER: 1
UCD-SNMP-MIB::ucdavis.32.2.1.2.19.122.102.115.45.115.108.99.101.49.48.115.110.48.49.45.117.115.101.100 = STRING: “/bin/bash”
UCD-SNMP-MIB::ucdavis.32.2.1.3.19.122.102.115.45.115.108.99.101.49.48.115.110.48.49.45.117.115.101.100 = STRING: “/var/tmp/mrtg/zfs-test-zfs-host-used.sh”
UCD-SNMP-MIB::ucdavis.32.2.1.4.19.122.102.115.45.115.108.99.101.49.48.115.110.48.49.45.117.115.101.100 = “”
UCD-SNMP-MIB::ucdavis.32.2.1.5.19.122.102.115.45.115.108.99.101.49.48.115.110.48.49.45.117.115.101.100 = INTEGER: 5
UCD-SNMP-MIB::ucdavis.32.2.1.6.19.122.102.115.45.115.108.99.101.49.48.115.110.48.49.45.117.115.101.100 = INTEGER: 1
UCD-SNMP-MIB::ucdavis.32.2.1.7.19.122.102.115.45.115.108.99.101.49.48.115.110.48.49.45.117.115.101.100 = INTEGER: 1
UCD-SNMP-MIB::ucdavis.32.2.1.20.19.122.102.115.45.115.108.99.101.49.48.115.110.48.49.45.117.115.101.100 = INTEGER: 4
UCD-SNMP-MIB::ucdavis.32.2.1.21.19.122.102.115.45.115.108.99.101.49.48.115.110.48.49.45.117.115.101.100 = INTEGER: 1
UCD-SNMP-MIB::ucdavis.32.3.1.1.19.122.102.115.45.115.108.99.101.49.48.115.110.48.49.45.117.115.101.100 = STRING: “9.31″
UCD-SNMP-MIB::ucdavis.32.3.1.2.19.122.102.115.45.115.108.99.101.49.48.115.110.48.49.45.117.115.101.100 = STRING: “9.31″
UCD-SNMP-MIB::ucdavis.32.3.1.3.19.122.102.115.45.115.108.99.101.49.48.115.110.48.49.45.117.115.101.100 = INTEGER: 1
UCD-SNMP-MIB::ucdavis.32.3.1.4.19.122.102.115.45.115.108.99.101.49.48.115.110.48.49.45.117.115.101.100 = INTEGER: 0
UCD-SNMP-MIB::ucdavis.32.4.1.2.19.122.102.115.45.115.108.99.101.49.48.115.110.48.49.45.117.115.101.100.1 = STRING: “9.31″
[root@test-centos ~]# snmpwalk -v2c -c public localhost .1.3.6.1.4.1.2021.31
UCD-SNMP-MIB::ucdavis.31.1.0 = INTEGER: 1
UCD-SNMP-MIB::ucdavis.31.2.1.2.20.122.102.115.45.115.108.99.101.49.48.115.110.48.49.45.116.111.116.97.108 = STRING: “/bin/bash”
UCD-SNMP-MIB::ucdavis.31.2.1.3.20.122.102.115.45.115.108.99.101.49.48.115.110.48.49.45.116.111.116.97.108 = STRING: “/var/tmp/mrtg/zfs-test-zfs-host-total.sh”
UCD-SNMP-MIB::ucdavis.31.2.1.4.20.122.102.115.45.115.108.99.101.49.48.115.110.48.49.45.116.111.116.97.108 = “”
UCD-SNMP-MIB::ucdavis.31.2.1.5.20.122.102.115.45.115.108.99.101.49.48.115.110.48.49.45.116.111.116.97.108 = INTEGER: 5
UCD-SNMP-MIB::ucdavis.31.2.1.6.20.122.102.115.45.115.108.99.101.49.48.115.110.48.49.45.116.111.116.97.108 = INTEGER: 1
UCD-SNMP-MIB::ucdavis.31.2.1.7.20.122.102.115.45.115.108.99.101.49.48.115.110.48.49.45.116.111.116.97.108 = INTEGER: 1
UCD-SNMP-MIB::ucdavis.31.2.1.20.20.122.102.115.45.115.108.99.101.49.48.115.110.48.49.45.116.111.116.97.108 = INTEGER: 4
UCD-SNMP-MIB::ucdavis.31.2.1.21.20.122.102.115.45.115.108.99.101.49.48.115.110.48.49.45.116.111.116.97.108 = INTEGER: 1
UCD-SNMP-MIB::ucdavis.31.3.1.1.20.122.102.115.45.115.108.99.101.49.48.115.110.48.49.45.116.111.116.97.108 = STRING: “16.32″
UCD-SNMP-MIB::ucdavis.31.3.1.2.20.122.102.115.45.115.108.99.101.49.48.115.110.48.49.45.116.111.116.97.108 = STRING: “16.32″
UCD-SNMP-MIB::ucdavis.31.3.1.3.20.122.102.115.45.115.108.99.101.49.48.115.110.48.49.45.116.111.116.97.108 = INTEGER: 1
UCD-SNMP-MIB::ucdavis.31.3.1.4.20.122.102.115.45.115.108.99.101.49.48.115.110.48.49.45.116.111.116.97.108 = INTEGER: 0
UCD-SNMP-MIB::ucdavis.31.4.1.2.20.122.102.115.45.115.108.99.101.49.48.115.110.48.49.45.116.111.116.97.108.1 = STRING: “16.32″

From the output, we can see that OIDs UCD-SNMP-MIB::ucdavis.32.3.1.1.19.122.102.115.45.115.108.99.101.49.48.115.110.48.49.45.117.115.101.100 (used space) and UCD-SNMP-MIB::ucdavis.31.3.1.1.20.122.102.115.45.115.108.99.101.49.48.115.110.48.49.45.116.111.116.97.108 (total space) are the two OIDs we want.

• Part 4 – do the mrtg drawing

As we got the OIDs we want, it’s now easier for us to do the mrtg drawing. On the snmp linux host, do the following steps:

[root@test-centos ~]# cat /etc/mrtg/test-zfs-host.cfg
#LoadMIBs: /usr/share/snmp/mibs/UCD-SNMP-MIB.txt,/usr/share/snmp/mibs/TCP-MIB.txt
workdir: /var/www/mrtg/
Title[zfs_space_test-zfs-host]: Percentage used space on zfs
PageTop[zfs_space_test-zfs-host]: <h1>Percentage used space on zfs</h1>
Target[zfs_space_test-zfs-host]: .1.3.6.1.4.1.2021.32.3.1.1.19.122.102.115.45.115.108.99.101.49.48.115.110.48.49.45.117.115.101.100&.1.3.6.1.4.1.2021.31.3.1.1.20.122.102.115.45.115.108.99.101.49.48.115.110.48.49.45.116.111.116.97.108:public@localhost
Options[zfs_space_test-zfs-host]: growright,gauge,transparent,nopercent
Unscaled[zfs_space_test-zfs-host]: ymwd
MaxBytes[zfs_space_test-zfs-host]: 100
YLegend[zfs_space_test-zfs-host]: UsedSpace %
ShortLegend[zfs_space_test-zfs-host]: T
LegendI[zfs_space_test-zfs-host]: Used
LegendO[zfs_space_test-zfs-host]: Total
Legend1[zfs_space_test-zfs-host]: Percentage used space on zfs
Legend2[zfs_space_test-zfs-host]: Percentage all space on zfs

PS:
You need replace “UCD-SNMP-MIB::ucdavis” with .1.3.6.1.4.1.2021 or you’ll get error messages like the following:

[root@test-centos ~]# env LANG=C TZ=Asia/Shanghai /usr/bin/mrtg /etc/mrtg/test-zfs-host.cfg
Argument “v4only” isn’t numeric in int at /usr/bin/../lib64/mrtg2/SNMP_Session.pm line 183.
backoff (v4only) must be a number >= 1.0 at /usr/bin/../lib64/mrtg2/SNMP_util.pm line 465

Let’s continue:

[root@test-centos ~]# env LANG=C TZ=Asia/Shanghai /usr/bin/mrtg /etc/mrtg/test-zfs-host.cfg
24-12-2012 01:46:41, Rateup WARNING: /usr/bin/rateup could not read the primary log file for zfs_space_test-zfs-host
24-12-2012 01:46:41, Rateup WARNING: /usr/bin/rateup The backup log file for zfs_space_test-zfs-host was invalid as well
24-12-2012 01:46:41, Rateup WARNING: /usr/bin/rateup Can’t remove zfs_space_test-zfs-host.old updating log file
24-12-2012 01:46:41, Rateup WARNING: /usr/bin/rateup Can’t rename zfs_space_test-zfs-host.log to zfs_space_test-zfs-host.old updating log file

[root@test-centos ~]# env LANG=C TZ=Asia/Shanghai /usr/bin/mrtg /etc/mrtg/test-zfs-host.cfg
24-12-2012 01:46:46, Rateup WARNING: /usr/bin/rateup Can’t remove zfs_space_test-zfs-host.old updating log file

[root@test-centos ~]# env LANG=C TZ=Asia/Shanghai /usr/bin/mrtg /etc/mrtg/test-zfs-host.cfg

[root@test-centos ~]# indexmaker –output=/var/www/mrtg/index.html /etc/mrtg/test-zfs-host.cfg

Now add cronjob:

0-59/5 * * * * env LANG=C TZ=Asia/Shanghai /usr/bin/mrtg /etc/mrtg/test-zfs-host.cfg

Visit http://<your linux box’s ip address>/mrtg/ to get the GUI result(you’ll wait 5 minutes for the initial result, be patient!)

• Part 5 – troubleshooting

• If you met error messages like the following:

[root@test-centos ~]# env LANG=C TZ=Asia/Shanghai /usr/bin/mrtg /etc/mrtg/test-zfs-host.cfg
Argument “v4only” isn’t numeric in int at /usr/bin/../lib64/mrtg2/SNMP_Session.pm line 183.
backoff (v4only) must be a number >= 1.0 at /usr/bin/../lib64/mrtg2/SNMP_util.pm line 465

That’s because you’re using OIDs/MIBs alias rather than number. Change alias to number, i.e. change UCD-SNMP-MIB::ucdavis to .1.3.6.1.4.1.2021, and then re-check.

• If you met error messages like the following:

• If you met error messages like the following:

[root@test-centos mrtg]# env LANG=C TZ=Asia/Shanghai /usr/bin/mrtg /etc/mrtg/test-zfs-host.cfg
SNMP Error:
Received SNMP response with error code
error status: noSuchName
index 2 (OID: 1.3.6.1.4.1.2021.44)
SNMPv1_Session (remote host: “localhost” [127.0.0.1].161)
community: “public”
request ID: 1786815468
PDU bufsize: 8000 bytes
timeout: 2s
retries: 5
backoff: 1)
at /usr/bin/../lib64/mrtg2/SNMP_util.pm line 490
SNMPGET Problem for .1.3.6.1.4.1.2021.44 .1.3.6.1.4.1.2021.44 sysUptime sysName on public@localhost::::::v4only
at /usr/bin/mrtg line 2035
SNMP Error:
Received SNMP response with error code
error status: noSuchName
index 2 (OID: 1.3.6.1.4.1.2021.45)
SNMPv1_Session (remote host: “localhost” [127.0.0.1].161)
community: “public”
request ID: 1786815469
PDU bufsize: 8000 bytes
timeout: 2s
retries: 5
backoff: 1)
at /usr/bin/../lib64/mrtg2/SNMP_util.pm line 490
SNMPGET Problem for .1.3.6.1.4.1.2021.45 .1.3.6.1.4.1.2021.45 sysUptime sysName on public@localhost::::::v4only
at /usr/bin/mrtg line 2035
Monday, 24 December 2012 at 14:41: ERROR: Target[zfs_space_test-zfs-host][_IN_] ‘( $target->[0]{$mode} ) * 10000 / ( $target->[1]{$mode} )’ (warn): Use of uninitialized value in division (/) at (eval 16) line 1.
Monday, 24 December 2012 at 14:41: ERROR: Target[zfs_space_test-zfs-host][_OUT_] ‘( $target->[0]{$mode} ) * 10000 / ( $target->[1]{$mode} )’ (warn): Use of uninitialized value in division (/) at (eval 17) line 1.

Then one possible culprit is that new net-snmp stop supporting “exec”, use “extend” instead and re-try.

Note that you can also use “snmpd -f -Le” to check error messages related to snmpd.

PS:
Here’s more links from where you can read more about net-snmp/mrtg:

1. net-snmp FAQ http://www.net-snmp.org/FAQ.html
2. mrtg configuration opstions http://oss.oetiker.ch/mrtg/doc/mrtg-reference.en.html
3. SUN zfs storage SNMP – http://docs.oracle.com/cd/E22471_01/html/820-4167/configuration__services__snmp.html
4. net-snmp extending agent functionality – http://linux.die.net/man/5/snmpd.conf (search for ‘extending agent functionality’ on this page)
5. Now here’s the image of the baby(click on it to see the larger one:

This article is going to talk about the concept of hash/md5 vs crc32 hash.

• Concept

Hash is some sort of a fingerprint of some information (data). It is designed to produce a code of specific length (somewhere between 8 and 512 bits) that satisfies the following:
- Every bit in the message contributes to the hash. This means that changing any bit in the message should change the hash.
- Relatively small changes in the message should always result in changes in the hash. We want to be sure that it would take an extremely unlikely combination of errors to produce an identical hash.
- The histogram of output hash values for input messages should tend to be flat. For a given input message, we want the probability of a given hash being produced to be nearly equal across the entire range of possible hash values (for example 0h to FFh for 8 bit hashes).

CRC stands for Cyclic redundancy code. It is a code that provides efficient error detection. It is used by many programs (for example Zip compression programs, communication protocols…). There are many variants of this code.
CRC is also often used for reference to general hash code.

CRC-32 is an acronym for the 32 bit Cyclical Redundancy Check algorithm. CRC-32 generally refers to a specific 32 bit CRC formula sanctioned by the CCITT, an international standards body primarily concerned with telecommunications. CRC-32 is used in communications protocols such as HDLC and ZMODEM to verify the integrity of blocks of data being transferred through various media.

MD5 stands for Message Digest 5 algorithm. Algorithm developed by Professor Ronald L. Rivest of MIT and used in many applications.

• Reliability

CRC-32 code is 32 bit number (code) generated by CRC-32 algorithm based on data input (in our case file content). This code is some sort of a “fingerprint”. However it differs somewhat from the human fingerprint. It is often said that no two people have identical fingerprints. This can’t be the case for our CRC fingerprint. Since there are more than 4,294,967,296 different files in the world, it is a foregone conclusion that some of them must have identical codes. However, the CRC-32 does satisfy all the above mentioned requirements for hash code. They were the goals that the CCITT had in mind when selecting the CRC-32 algorithm. In practice, the chances of inadvertently damaging or modifying a file without modifying the CRC is vanishingly small, so for all practical purposes testing CRC code to detect changes can be considered to be infallible.

However if file would be intentionally damaged (for example by a virus) the CRC could be restored to its previous state. This could be done by using using brute force method to add some bytes to end of the file. So this is one thing it cannot protect you from (although this is almost never done).

MD5 was developed by Professor Ronald L. Rivest of MIT. What it does, to quote the executive summary of rfc1321, is: The MD5 algorithm takes as input a message of arbitrary length and produces as output a 128-bit “fingerprint” or “message digest” of the input. It is conjectured that it is computationally infeasible to produce two messages having the same message digest, or to produce any message having a given prespecified target message digest. The MD5 algorithm is intended for digital signature applications, where a large file must be “compressed” in a secure manner before being encrypted with a private (secret) key under a public-key cryptosystem such as RSA.

In essence, MD5 is a way to verify data integrity, and is much more reliable than checksum and many other commonly used methods.

• Implementation

Hash is implemented using concurrent reading and hash calculation. This is done by writing data from media into intermediate buffer in one thread and reading from buffer and calculating hash values in other thread. Hash values are written to disk as they are calculated to minimize memory usage.

OFED is an abbr for “OpenFabrics Enterprise Distribution”. When installing OFED-1.5.4.1 on a centos/RHEL 5.8 linux system, I met the following problem:

In file included from /var/tmp/OFED_topdir/BUILD/ofa_kernel-1.5.4.1/drivers/infiniband/core/notice.c:37:
/var/tmp/OFED_topdir/BUILD/ofa_kernel-1.5.4.1/kernel_addons/backport/2.6.18-EL5.7/include/linux/pci.h:164: error: conflicting types for ‘pci_pcie_cap’
include/linux/pci.h:1015: error: previous definition of ‘pci_pcie_cap’ was here
make[4]: *** [/var/tmp/OFED_topdir/BUILD/ofa_kernel-1.5.4.1/drivers/infiniband/core/notice.o] Error 1
make[3]: *** [/var/tmp/OFED_topdir/BUILD/ofa_kernel-1.5.4.1/drivers/infiniband/core] Error 2
make[2]: *** [/var/tmp/OFED_topdir/BUILD/ofa_kernel-1.5.4.1/drivers/infiniband] Error 2
make[1]: *** [_module_/var/tmp/OFED_topdir/BUILD/ofa_kernel-1.5.4.1] Error 2
make[1]: Leaving directory `/usr/src/kernels/2.6.18-308.4.1.0.1.el5-x86_64′
make: *** [kernel] Error 2
error: Bad exit status from /var/tmp/rpm-tmp.70159 (%build)

After googling and experimenting, I can tell you the definite resolution for this problem -> install another version of OFED, OFED-1.5.3.2. Go to the following official site to download: http://www.openfabrics.org/downloads/OFED/ofed-1.5.3/

PS:

1. Here’s the file of OFED installation howto: OFED installation README.txt

2.Some Infiniband commands that may be useful for you:

service openibd status/start/stop
lspci |grep -i infini
ibnetdiscover
ibhosts
ibswitches
ibv_devinfo #Verify the status of ports by using ibv_devinfo: all connected ports should report a “PORT_ACTIVE” state

There’s a weird incident occurred on a linux box. The linux box turned not responsible to ping or ssh, although from ifconfig and /proc/net/bonding/bond0 file, the system said it’s running ok. After some google work, I found that the issue may related to the NIC driver. I tried bring down/bring up NICs one by one, but got error:

Bringing up loopback interface bond0: bnx2i: dev eth0 does not support iscsi

bnx2i: iSCSI not supported, dev=eth0

bonding: no command found in slaves file for bond bond0. Use +ifname or -ifname

At last, I tried restart the whole network i.e. /etc/init.d/network restart. And that did the trick, the networking was then running ok and can ping/ssh to it without problem.

To know nfs version in linux/solaris:

• On the nfs server side, you can run a nfsstat -s to check. The used version of nfs will have data summary other than 0% ones, as the following:

[email protected]# nfsstat -s
Server rpc stats:
calls badcalls badauth badclnt xdrcall
28 0 0 0 0

Server nfs v3:
null getattr setattr lookup access readlink
3 11% 4 14% 0 0% 1 3% 4 14% 0 0%
read write create mkdir symlink mknod
0 0% 0 0% 0 0% 0 0% 0 0% 0 0%
remove rmdir rename link readdir readdirplus
0 0% 0 0% 0 0% 0 0% 0 0% 2 7%
fsstat fsinfo pathconf commit
9 33% 4 14% 0 0% 0 0%

• On the nfs server, we can also have a checking on what versions(2/3/4) and transport protocols(tcp/udp) the nfs supported with the command “rpcinfo -p localhost|grep nfs”:

root@doxer# rpcinfo -p localhost|grep nfs
100003 2 udp 2049 nfs
100003 3 udp 2049 nfs
100003 4 udp 2049 nfs
100003 2 tcp 2049 nfs
100003 3 tcp 2049 nfs
100003 4 tcp 2049 nfs

• On the nfs client hosts, you can run a nfsstat -c to check the version the client is using. As always, the used version of nfs will have data summary other than 0% ones, as the following:

[email protected]# nfsstat -c

Client rpc:
Connection oriented:
calls badcalls badxids timeouts newcreds badverfs
1219760 322812 0 0 0 0
timers cantconn nomem interrupts
0 322808 0 0
Connectionless:
calls badcalls retrans badxids timeouts newcreds
0 0 0 0 0 0
badverfs timers nomem cantsend
0 0 0 0

Client nfs:
calls badcalls clgets cltoomany
753081 28 753081 0
Version 2: (0 calls)
null getattr setattr root lookup readlink
0 0% 0 0% 0 0% 0 0% 0 0% 0 0%
read wrcache write create remove rename
0 0% 0 0% 0 0% 0 0% 0 0% 0 0%
link symlink mkdir rmdir readdir statfs
0 0% 0 0% 0 0% 0 0% 0 0% 0 0%
Version 3: (748700 calls)
null getattr setattr lookup access readlink
0 0% 140588 18% 61939 8% 184611 24% 150266 20% 8 0%
read write create mkdir symlink mknod
35415 4% 58540 7% 11703 1% 562 0% 248 0% 0 0%
remove rmdir rename link readdir readdirplus
3264 0% 0 0% 9 0% 0 0% 1165 0% 1219 0%
fsstat fsinfo pathconf commit
33435 4% 7160 0% 3309 0% 55259 7%

Client nfs_acl:
Version 2: (0 calls)
null getacl setacl getattr access
0 0% 0 0% 0 0% 0 0% 0 0%
Version 3: (4382 calls)
null getacl setacl
0 0% 4382 100% 0 0%

• Also, you can run nfsstat -m on nfs client hosts to print information about each of the mounted NFS file systems(the output info has nfs version indicated also):

[email protected] # nfsstat -m
/apps/uriman/tmp from doxer:/export/was/trncsc_cell_urimantmp
Flags: vers=3,proto=tcp,sec=none,hard,intr,link,symlink,acl,rsize=32768,wsize=32768,retrans=5,timeo=600
Attr cache: acregmin=3,acregmax=60,acdirmin=30,acdirmax=60