Archive

Posts Tagged ‘zfs’

perl script for getting sun zfs head project and share usage info

April 26th, 2013 Comments off

Sometimes you would like to know on sun zfs head, which project occupies most of the space, and which shares of that occupies most space of that project.

Here's a perl script to fulfill this(it's a little cumbersome, but it works anyway)

#!/usr/bin/perl
use strict;
use warnings;
use Net::SSH::Perl;
use List::Util qw/sum/;
my $host = 'test-zfs-host';
my $user = 'root';
my $password = 'password';

my $ssh = Net::SSH::Perl->new($host);
$ssh->login($user,$password);
my ($stdout,$stderr,$exit) = $ssh->cmd("shares show");
my @std_arr=split(/:/,$stdout);
my @projects_arr = split(/\n/, $std_arr[2]);

foreach(@projects_arr){
$_ =~ s/^\s+|\s+$//g;
}
shift @projects_arr;
pop @projects_arr;
pop @projects_arr;

my @space_projects;
foreach(@projects_arr){
my ($stdout2,$stderr2,$exit2) = $ssh->cmd("shares select $_ get");
my @stdout_arr=split(/\n/,$stdout2);
my $space_temp=join("\n",grep(/space_total/,@stdout_arr));
my @space_total_project = "project ".$_.$space_temp;
push(@space_projects,@space_total_project);
}
my @space_projects3;
foreach(@projects_arr){
my ($stdout3,$stderr3,$exit3) = $ssh->cmd("shares select $_ ls");
my @stdout_arr3=split(/\n/,$stdout3);
my @space_temp3=grep(/\/export\//,@stdout_arr3);
my @space_temp4=grep(!/mountpoint/,@space_temp3);
push(@space_projects3,@space_temp4);
}
open(my $temp1,'>','/var/tmp/temp1') or die("cannot open file temp1");
open(my $temp2,'>','/var/tmp/temp2') or die("cannot open file temp2");
open(my $temp3,'>','/var/tmp/temp3') or die("cannot open file temp3");
open(my $temp4,'>','/var/tmp/temp4') or die("cannot open file temp4");
my @var_T;
my @var_G;
my @var_M;
my @var_K;

foreach(@space_projects){
if($_ =~ /.+space_total\s+(.*)K/){
push(@var_K,$_);
}
elsif($_ =~ /.+space_total\s+(.*)M/){
push(@var_M,$_);
}
elsif($_ =~ /.+space_total\s+(.*)G/){
push(@var_G,$_);
}
elsif($_ =~ /.+space_total\s+(.*)T/){
push(@var_T,$_);
}
}

select $temp1;
foreach(@var_T){
print $_."\n";
}
close $temp1;

select $temp2;
foreach(@var_G){
print $_."\n";
}
close $temp2;
select $temp3;
foreach(@var_M){
print $_."\n";
}
close $temp3;
select $temp4;
foreach(@var_K){
print $_."\n";
}
close $temp4;

system("echo \"======zfs project usage info(Descending)======\"");
system("sort -r -n -k 5 /var/tmp/temp1;sort -r -n -k 5 /var/tmp/temp2;sort -r -n -k 5 /var/tmp/temp3;sort -r -n -k 5 /var/tmp/temp4");
open(my $temp5,'>','/var/tmp/temp5') or die("cannot open file temp5");
open(my $temp6,'>','/var/tmp/temp6') or die("cannot open file temp6");
open(my $temp7,'>','/var/tmp/temp7') or die("cannot open file temp7");
open(my $temp8,'>','/var/tmp/temp8') or die("cannot open file temp8");
my @var_T_2;
my @var_G_2;
my @var_M_2;
my @var_K_2;

foreach(@space_projects3){
if($_ =~ /\s+.*K\s+.*/){
push(@var_K_2,$_);
}
elsif($_ =~ /\s+.*M\s+.*/){
push(@var_M_2,$_);
}
elsif($_ =~ /\s+.*G\s+.*/){
push(@var_G_2,$_);
}
elsif($_ =~ /\s+.*T\s+.*/){
push(@var_T_2,$_);
}
}

select $temp5;
foreach(@var_T_2){
print $_."\n";
}
close $temp5;

select $temp6;
foreach(@var_G_2){
print $_."\n";
}
close $temp6;
select $temp7;
foreach(@var_M_2){
print $_."\n";
}
close $temp7;
select $temp8;
foreach(@var_K_2){
print $_."\n";
}
close $temp8;

system("echo \"\n\n\n======zfs share usage info(Descending)======\"");
system("sort -r -n -k 2 /var/tmp/temp5;sort -r -n -k 2 /var/tmp/temp6;sort -r -n -k 2 /var/tmp/temp7;sort -r -n -k 2 /var/tmp/temp8");

The output would be like:

======zfs project usage info(Descending)======
project DC2_DMZ space_total = 7.68T
project dc2_c9testga space_total = 1.10T
project fa_trialadcf space_total = 277G
project fa_rehydration space_total = 266G
project common space_total = 10.0G
project NODE_8 space_total = 93K
project default space_total = 31K

 

======zfs share usage info(Descending)======
Service_Mid-2 1.44T /export/DC2_DMZ/Service_Mid-2
Service_Web 1.22T /export/DC2_DMZ/Service_Web
dc2_shared_idm 743G /export/DC2_DMZ/dc2_shared_idm
Infra_Web 400G /export/DC2_DMZ/Infra_Web
nuviaq_local02 988M /export/DC2_DMZ/nuviaq_local02
sftp_staging 127M /export/DC2_DMZ/sftp_staging
sftp_local03 14.8M /export/DC2_DMZ/sftp_local03
sftp_manager_local01 85K /export/DC2_DMZ/sftp_manager_local01

resolved – differences between zfs ARC L2ARC ZIL

January 31st, 2013 Comments off
  • ARC

zfs ARC(adaptive replacement cache) is a very fast cache located in the server’s memory.

For example, our ZFS server with 12GB of RAM has 11GB dedicated to ARC, which means our ZFS server will be able to cache 11GB of the most accessed data. Any read requests for data in the cache can be served directly from the ARC memory cache instead of hitting the much slower hard drives. This creates a noticeable performance boost for data that is accessed frequently.

  • L2ARC

As a general rule, you want to install as much RAM into the server as you can to make the ARC as big as possible. At some point, adding more memory is just cost prohibitive. That is where the L2ARC becomes important. The L2ARC is the second level adaptive replacement cache. The L2ARC is often called “cache drives” in the ZFS systems.

L2ARC is a new layer between Disk and the cache (ARC) in main memory for ZFS. It uses dedicated storage devices to hold cached data. The main role of this cache is to boost the performance of random read workloads. The intended L2ARC devices include 10K/15K RPM disks like short-stroked disks, solid state disks (SSD), and other media with substantially faster read latency than disk.

  • ZIL

ZIL(ZFS Intent Log) exists for performance improvement on synchronous writes. Synchronous write is very slow than asynchronous write, but it's more stable. Essentially, the intent log of a file system is nothing more than an insurance against power failures, a to-do list if you will, that keeps track of the stuff that needs to be updated on disk, even if the power fails (or something else happens that prevents the system from updating its disks).

To get better performance, use separated disks(SSD) for ZIL, such as zpool add pool log c2d0.

Now I'm giving you an true example about zfs ZIL/L2ARC/ARC on SUN ZFS 7320 head:

test-zfs# zpool iostat -v exalogic
capacity operations bandwidth
pool alloc free read write read write
------------------------- ----- ----- ----- ----- ----- -----
exalogic 6.78T 17.7T 53 1.56K 991K 25.1M
mirror 772G 1.96T 6 133 111K 2.07M
c0t5000CCA01A5FDCACd0 - - 3 36 57.6K 2.07M #these are the physical disks
c0t5000CCA01A6F5CF4d0 - - 2 35 57.7K 2.07M
mirror 772G 1.96T 5 133 110K 2.07M
c0t5000CCA01A6F5D00d0 - - 2 36 56.2K 2.07M
c0t5000CCA01A6F64F4d0 - - 2 35 57.3K 2.07M
mirror 772G 1.96T 5 133 110K 2.07M
c0t5000CCA01A76A7B8d0 - - 2 36 56.3K 2.07M
c0t5000CCA01A746CCCd0 - - 2 36 56.8K 2.07M
mirror 772G 1.96T 5 133 110K 2.07M
c0t5000CCA01A749A88d0 - - 2 35 56.7K 2.07M
c0t5000CCA01A759E90d0 - - 2 35 56.1K 2.07M
mirror 772G 1.96T 5 133 110K 2.07M
c0t5000CCA01A767FDCd0 - - 2 35 56.1K 2.07M
c0t5000CCA01A782A40d0 - - 2 35 57.1K 2.07M
mirror 772G 1.96T 5 133 110K 2.07M
c0t5000CCA01A782D10d0 - - 2 35 57.2K 2.07M
c0t5000CCA01A7465F8d0 - - 2 35 56.3K 2.07M
mirror 772G 1.96T 5 133 110K 2.07M
c0t5000CCA01A7597FCd0 - - 2 35 57.6K 2.07M
c0t5000CCA01A7828F4d0 - - 2 35 56.2K 2.07M
mirror 772G 1.96T 5 133 110K 2.07M
c0t5000CCA01A7829ACd0 - - 2 35 57.1K 2.07M
c0t5000CCA01A78278Cd0 - - 2 35 57.4K 2.07M
mirror 772G 1.96T 6 133 111K 2.07M
c0t5000CCA01A736000d0 - - 3 35 57.3K 2.07M
c0t5000CCA01A738000d0 - - 2 35 57.3K 2.07M
c0t5000A72030061B82d0 224M 67.8G 0 98 1 1.62M #ZIL(SSD write cache, ZFS Intent Log)
c0t5000A72030061C70d0 224M 67.8G 0 98 1 1.62M
c0t5000A72030062135d0 223M 67.8G 0 98 1 1.62M
c0t5000A72030062146d0 224M 67.8G 0 98 1 1.62M
cache - - - - - -
c2t2d0 334G 143G 15 6 217K 652K #L2ARC(SSD cache drives)
c2t3d0 332G 145G 15 6 215K 649K
c2t4d0 333G 144G 11 6 169K 651K
c2t5d0 333G 144G 13 6 192K 650K
c2t2d0 - - 0 0 0 0
c2t3d0 - - 0 0 0 0
c2t4d0 - - 0 0 0 0
c2t5d0 - - 0 0 0 0

And as for ARC:

test-zfs:> status memory show
Memory:
Cache 63.4G bytes #ARC
Unused 17.3G bytes
Mgmt 561M bytes
Other 491M bytes
Kernel 14.3G bytes

perl script for monitoring sun zfs memory usage

January 16th, 2013 Comments off

On zfs's aksh, I can check memory usage with the following:

test-zfs:> status memory show
Memory:
Cache 719M bytes
Unused 15.0G bytes
Mgmt 210M bytes
Other 332M bytes
Kernel 7.79G bytes

So now I want to collect this memory usae information automatically for SNMP's use. Here's the steps:

cpan> o conf prerequisites_policy follow
cpan> o conf commit

Since the host is using proxy to get on the internet, so in /etc/wgetrc:

http_proxy = http://www-proxy.us.example.com:80/
ftp_proxy = http://www-proxy.us.example.com:80/
use_proxy = on

Now install the Net::SSH::Perl perl module:

PERL_MM_USE_DEFAULT=1 perl -MCPAN -e 'install Net::SSH::Perl'

And to confirm that Net::SSH::Perl was installed, run the following command:

perl -e 'use Net::SSH::Perl' #no output is good, as it means the package was installed successfully

Now here goes the perl script to get the memory usage of sun zfs head:

[root@test-centos ~]# cat /var/tmp/mrtg/zfs-test-zfs-memory.pl
#!/usr/bin/perl
use strict;
use warnings;
use Net::SSH::Perl;
my $host = 'test-zfs';
my $user = 'root';
my $password = 'password';

my $ssh = Net::SSH::Perl->new($host);
$ssh->login($user,$password);
my ($stdout,$stderr,$exit) = $ssh->cmd("status memory show");
$ssh->cmd("exit");
if($stderr){
print "ErrorCode:$exit\n";
print "ErrorMsg:$stderr";
} else {
my @std_arr = split(/\n/, $stdout);
shift @std_arr;
foreach(@std_arr) {
if ($_ =~ /.+\b\s+(.+)M\sbytes/){
$_=$1/1024;
}
elsif($_ =~ /.+\b\s+(.+)G\sbytes/){
$_=$1;
}
else{}
}
foreach(@std_arr) {
print $_."\n";
}
}
exit $exit;

PS:
If you get the following error messages during installation of a perl module:

[root@test-centos ~]# perl -MCPAN -e 'install SOAP::Lite'
CPAN: Storable loaded ok
CPAN: LWP::UserAgent loaded ok
Fetching with LWP:
ftp://ftp.perl.org/pub/CPAN/authors/01mailrc.txt.gz
LWP failed with code[500] message[LWP::Protocol::MyFTP: connect: Connection timed out]
Fetching with Net::FTP:
ftp://ftp.perl.org/pub/CPAN/authors/01mailrc.txt.gz

Trying with "/usr/bin/links -source" to get
ftp://ftp.perl.org/pub/CPAN/authors/01mailrc.txt.gz
ELinks: Connection timed out

Then you may have a check of whether you're using proxy to get on the internet(run cpan > o conf init to re-configure cpan; later you should set /etc/wgetrc: http_proxy, ftp_proxy, use_proxy).

 

SUN zfs storage 7320 monitoring using net-snmp and mrtg

December 25th, 2012 Comments off

This article is going to talk about zfs storage 7320 monitoring using net-snmp and mrtg. Although the monitored system is sun zfs storage 7320, you'll find the main idea of this article can be applied to many different system monitoring, including but not limited to cpu usage/network/bandwidth/disk/temperature of cisco switches, other linux systems and even windows systems.

As net-snmp extending agent functionality is not supported on sun zfs storage 7320 which is solaris 11 express system, so I'm going to monitor sun zfs storage through using of one linux snmp client by writing monitoring scripts on that linux client rather than on zfs itself.

Now, here goes the steps:

Part 1 - set up snmp client and mrtg on a linux host

yum -y install gcc-* gd-* libpng-* zlib-* httpd
yum -y install net-snmp* net-snmp-libs lm_sensors lm_sensors-devel
yum -y install mrtg
cp /etc/snmp/snmpd.conf{,.bak}
echo "rocommunity public" > /etc/snmp/snmpd.conf
mkdir -p /etc/mrtg
chkconfig --level 2345 snmpd on
service snmpd start

Ensure snmpd is listening:

netstat -tunlp |grep snmp
tcp 0 0 127.0.0.1:199 0.0.0.0:* LISTEN 26427/snmpd
udp 0 0 0.0.0.0:161 0.0.0.0:* 26427/snmpd

And let's have a test to make sure snmp client is working as expected:

[root@test-centos mrtg]# snmpwalk -v2c -c public localhost interface
IF-MIB::ifNumber.0 = INTEGER: 4
IF-MIB::ifIndex.1 = INTEGER: 1
IF-MIB::ifIndex.2 = INTEGER: 2
IF-MIB::ifIndex.3 = INTEGER: 3
IF-MIB::ifIndex.4 = INTEGER: 4
IF-MIB::ifDescr.1 = STRING: lo
IF-MIB::ifDescr.2 = STRING: eth0
IF-MIB::ifDescr.3 = STRING: eth1
......
......

Now it's time to configure mrtg on linux:

[root@test-centos mrtg]# cat /etc/httpd/conf.d/mrtg.conf
Alias /mrtg /var/www/mrtg
<Location /mrtg>
Order deny,allow
Allow from all
</Location>

Now, do a httpd restart:

[root@test-centos ~]# apachectl restart

Part 2 - configure snmp on SUN zfs storage 7320 web UI

Log on SUN zfs storage 7320 web UI, navigate through "Configuration" -> "SNMP", and configure as the following:

After this, you'll need enable/restart SNMP service on zfs.
Now do a snmpwalk from snmp linux client to SUN zfs storage:

[root@test-centos ~]# snmpwalk -v2c -c public test-zfs-host interface
IF-MIB::ifNumber.0 = INTEGER: 6
IF-MIB::ifIndex.1 = INTEGER: 1
IF-MIB::ifIndex.4 = INTEGER: 4
IF-MIB::ifIndex.5 = INTEGER: 5
IF-MIB::ifIndex.6 = INTEGER: 6
IF-MIB::ifIndex.7 = INTEGER: 7
IF-MIB::ifIndex.8 = INTEGER: 8
IF-MIB::ifDescr.1 = STRING: lo0
......
......

  • Part 3 - extending snmp mibs/oids on snmp client(the linux host)

As stated at the beginning of this article, net-snmp extending agent functionality is not supported on sun zfs storage 7320. So I'm going to monitor sun zfs storage through using of one linux snmp client by writing monitoring scripts on that linux client rather than on zfs itself.

To avoid ssh asking for password when connecting from the linux host, you need add the linux host's pubkey to sun zfs storage:
On snmp Linux client:

[root@test-centos ~]# ssh-keygen -t rsa #if you already have pubkey, skip this step
[root@test-centos ~]# cat /root/.ssh/id_rsa.pub |awk '{print $2}'
AAAAB3NzaC1yc2EAAAABIwAAAQEAxYd97A/V5RwdkfzbkmYBqF189pTLOlbYt0dZzO395dfU0Sp/Ykrk+sOJO0bJZEtytuTcCz/bVutWB7vLzeQPxIToRUQnZX7ZoMsjyaFk3LhtAgFhYIycOw2FQL8Qvb5yMBASB2/KthsqaiNqOP/2Vy5e0aCFFIV5DlKQTp/3eceSMq8kTx+e801lZow++yT70rp3p+5WtriN/NKYI0B3cpSQY/36D/TcOF9v5IaqQokp/mLRoc1MLOhN0sy0ipCdT+0bbkZ4Lh8bEeQO48UGKEOnYrYto33tay4mZk8HPWFK4w/TQGxBLthiuPQ4oZzG3gVpQUS4GRwI9zZoGtgELQ==

On Sun zfs storage:
Click "Configuration", then add the snmp client's public key(ensure RSA is selected):

Do a ssh connection from snmp linux client to the sun zfs storage host, it should now not asking for password.(ensure to do this, as there's possibility that sun zfs's key is not on the linux client as you may never have connected from that linux client to the sun zfs storage system)

Now we're to the most important part. Assume we want to monitoring space usage on the SUN zfs storage system, to do this, you'll need do the following on the snmp linux client:

[root@test-centos mrtg]# cat /etc/snmp/snmpd.conf
rocommunity public
extend .1.3.6.1.4.1.2021.31 zfs-test-zfs-host-total /bin/bash /var/tmp/mrtg/zfs-test-zfs-host-total.sh
extend .1.3.6.1.4.1.2021.32 zfs-test-zfs-host-used /bin/bash /var/tmp/mrtg/zfs-test-zfs-host-used.sh

[root@test-centos ~]# cat /var/tmp/mrtg/zfs-test-zfs-host-used.sh
#!/bin/bash
_used=`ssh test-zfs-host "status ls"|grep Used|awk '{print gensub('/T/',"","g",$2)}'`
echo $_used;

[root@test-centos ~]# cat /var/tmp/mrtg/zfs-test-zfs-host-total.sh
_used=`ssh test-zfs-host "status ls"|grep Used|awk '{print gensub('/T/',"","g",$2)}'` #I trimed 'T', you may need modify this to meet your environment
_avail=`ssh test-zfs-host "status ls"|grep Avail|awk '{print gensub('/T/',"","g",$2)}'` #I trimed 'T', you may need modify this to meet your environment
_all=`echo $_used + $_avail|bc`
echo $_all;

[root@test-centos ~]# chmod +x /var/tmp/mrtg/zfs-test-zfs-host-used.sh
[root@test-centos ~]# chmod +x /var/tmp/mrtg/zfs-test-zfs-host-total.sh

Now, let's do a snmp restart on snmp linux client and then test the newly added OIDs:

[root@test-centos ~]# service snmpd restart

[root@test-centos ~]# snmpwalk -v2c -c public localhost .1.3.6.1.4.1.2021.32
UCD-SNMP-MIB::ucdavis.32.1.0 = INTEGER: 1
UCD-SNMP-MIB::ucdavis.32.2.1.2.19.122.102.115.45.115.108.99.101.49.48.115.110.48.49.45.117.115.101.100 = STRING: "/bin/bash"
UCD-SNMP-MIB::ucdavis.32.2.1.3.19.122.102.115.45.115.108.99.101.49.48.115.110.48.49.45.117.115.101.100 = STRING: "/var/tmp/mrtg/zfs-test-zfs-host-used.sh"
UCD-SNMP-MIB::ucdavis.32.2.1.4.19.122.102.115.45.115.108.99.101.49.48.115.110.48.49.45.117.115.101.100 = ""
UCD-SNMP-MIB::ucdavis.32.2.1.5.19.122.102.115.45.115.108.99.101.49.48.115.110.48.49.45.117.115.101.100 = INTEGER: 5
UCD-SNMP-MIB::ucdavis.32.2.1.6.19.122.102.115.45.115.108.99.101.49.48.115.110.48.49.45.117.115.101.100 = INTEGER: 1
UCD-SNMP-MIB::ucdavis.32.2.1.7.19.122.102.115.45.115.108.99.101.49.48.115.110.48.49.45.117.115.101.100 = INTEGER: 1
UCD-SNMP-MIB::ucdavis.32.2.1.20.19.122.102.115.45.115.108.99.101.49.48.115.110.48.49.45.117.115.101.100 = INTEGER: 4
UCD-SNMP-MIB::ucdavis.32.2.1.21.19.122.102.115.45.115.108.99.101.49.48.115.110.48.49.45.117.115.101.100 = INTEGER: 1
UCD-SNMP-MIB::ucdavis.32.3.1.1.19.122.102.115.45.115.108.99.101.49.48.115.110.48.49.45.117.115.101.100 = STRING: "9.31"
UCD-SNMP-MIB::ucdavis.32.3.1.2.19.122.102.115.45.115.108.99.101.49.48.115.110.48.49.45.117.115.101.100 = STRING: "9.31"
UCD-SNMP-MIB::ucdavis.32.3.1.3.19.122.102.115.45.115.108.99.101.49.48.115.110.48.49.45.117.115.101.100 = INTEGER: 1
UCD-SNMP-MIB::ucdavis.32.3.1.4.19.122.102.115.45.115.108.99.101.49.48.115.110.48.49.45.117.115.101.100 = INTEGER: 0
UCD-SNMP-MIB::ucdavis.32.4.1.2.19.122.102.115.45.115.108.99.101.49.48.115.110.48.49.45.117.115.101.100.1 = STRING: "9.31"
[root@test-centos ~]# snmpwalk -v2c -c public localhost .1.3.6.1.4.1.2021.31
UCD-SNMP-MIB::ucdavis.31.1.0 = INTEGER: 1
UCD-SNMP-MIB::ucdavis.31.2.1.2.20.122.102.115.45.115.108.99.101.49.48.115.110.48.49.45.116.111.116.97.108 = STRING: "/bin/bash"
UCD-SNMP-MIB::ucdavis.31.2.1.3.20.122.102.115.45.115.108.99.101.49.48.115.110.48.49.45.116.111.116.97.108 = STRING: "/var/tmp/mrtg/zfs-test-zfs-host-total.sh"
UCD-SNMP-MIB::ucdavis.31.2.1.4.20.122.102.115.45.115.108.99.101.49.48.115.110.48.49.45.116.111.116.97.108 = ""
UCD-SNMP-MIB::ucdavis.31.2.1.5.20.122.102.115.45.115.108.99.101.49.48.115.110.48.49.45.116.111.116.97.108 = INTEGER: 5
UCD-SNMP-MIB::ucdavis.31.2.1.6.20.122.102.115.45.115.108.99.101.49.48.115.110.48.49.45.116.111.116.97.108 = INTEGER: 1
UCD-SNMP-MIB::ucdavis.31.2.1.7.20.122.102.115.45.115.108.99.101.49.48.115.110.48.49.45.116.111.116.97.108 = INTEGER: 1
UCD-SNMP-MIB::ucdavis.31.2.1.20.20.122.102.115.45.115.108.99.101.49.48.115.110.48.49.45.116.111.116.97.108 = INTEGER: 4
UCD-SNMP-MIB::ucdavis.31.2.1.21.20.122.102.115.45.115.108.99.101.49.48.115.110.48.49.45.116.111.116.97.108 = INTEGER: 1
UCD-SNMP-MIB::ucdavis.31.3.1.1.20.122.102.115.45.115.108.99.101.49.48.115.110.48.49.45.116.111.116.97.108 = STRING: "16.32"
UCD-SNMP-MIB::ucdavis.31.3.1.2.20.122.102.115.45.115.108.99.101.49.48.115.110.48.49.45.116.111.116.97.108 = STRING: "16.32"
UCD-SNMP-MIB::ucdavis.31.3.1.3.20.122.102.115.45.115.108.99.101.49.48.115.110.48.49.45.116.111.116.97.108 = INTEGER: 1
UCD-SNMP-MIB::ucdavis.31.3.1.4.20.122.102.115.45.115.108.99.101.49.48.115.110.48.49.45.116.111.116.97.108 = INTEGER: 0
UCD-SNMP-MIB::ucdavis.31.4.1.2.20.122.102.115.45.115.108.99.101.49.48.115.110.48.49.45.116.111.116.97.108.1 = STRING: "16.32"

From the output, we can see that OIDs UCD-SNMP-MIB::ucdavis.32.3.1.1.19.122.102.115.45.115.108.99.101.49.48.115.110.48.49.45.117.115.101.100 (used space) and UCD-SNMP-MIB::ucdavis.31.3.1.1.20.122.102.115.45.115.108.99.101.49.48.115.110.48.49.45.116.111.116.97.108 (total space) are the two OIDs we want.

  • Part 4 - do the mrtg drawing

As we got the OIDs we want, it's now easier for us to do the mrtg drawing. On the snmp linux host, do the following steps:

[root@test-centos ~]# cat /etc/mrtg/test-zfs-host.cfg
#LoadMIBs: /usr/share/snmp/mibs/UCD-SNMP-MIB.txt,/usr/share/snmp/mibs/TCP-MIB.txt
workdir: /var/www/mrtg/
Title[zfs_space_test-zfs-host]: Percentage used space on zfs
PageTop[zfs_space_test-zfs-host]: <h1>Percentage used space on zfs</h1>
Target[zfs_space_test-zfs-host]: .1.3.6.1.4.1.2021.32.3.1.1.19.122.102.115.45.115.108.99.101.49.48.115.110.48.49.45.117.115.101.100&.1.3.6.1.4.1.2021.31.3.1.1.20.122.102.115.45.115.108.99.101.49.48.115.110.48.49.45.116.111.116.97.108:public@localhost
Options[zfs_space_test-zfs-host]: growright,gauge,transparent,nopercent
Unscaled[zfs_space_test-zfs-host]: ymwd
MaxBytes[zfs_space_test-zfs-host]: 100
YLegend[zfs_space_test-zfs-host]: UsedSpace %
ShortLegend[zfs_space_test-zfs-host]: T
LegendI[zfs_space_test-zfs-host]: Used
LegendO[zfs_space_test-zfs-host]: Total
Legend1[zfs_space_test-zfs-host]: Percentage used space on zfs
Legend2[zfs_space_test-zfs-host]: Percentage all space on zfs

PS:
You need replace "UCD-SNMP-MIB::ucdavis" with .1.3.6.1.4.1.2021 or you'll get error messages like the following:

[root@test-centos ~]# env LANG=C TZ=Asia/Shanghai /usr/bin/mrtg /etc/mrtg/test-zfs-host.cfg
Argument "v4only" isn't numeric in int at /usr/bin/../lib64/mrtg2/SNMP_Session.pm line 183.
backoff (v4only) must be a number >= 1.0 at /usr/bin/../lib64/mrtg2/SNMP_util.pm line 465

Let's continue:

[root@test-centos ~]# env LANG=C TZ=Asia/Shanghai /usr/bin/mrtg /etc/mrtg/test-zfs-host.cfg
24-12-2012 01:46:41, Rateup WARNING: /usr/bin/rateup could not read the primary log file for zfs_space_test-zfs-host
24-12-2012 01:46:41, Rateup WARNING: /usr/bin/rateup The backup log file for zfs_space_test-zfs-host was invalid as well
24-12-2012 01:46:41, Rateup WARNING: /usr/bin/rateup Can't remove zfs_space_test-zfs-host.old updating log file
24-12-2012 01:46:41, Rateup WARNING: /usr/bin/rateup Can't rename zfs_space_test-zfs-host.log to zfs_space_test-zfs-host.old updating log file

[root@test-centos ~]# env LANG=C TZ=Asia/Shanghai /usr/bin/mrtg /etc/mrtg/test-zfs-host.cfg
24-12-2012 01:46:46, Rateup WARNING: /usr/bin/rateup Can't remove zfs_space_test-zfs-host.old updating log file

[root@test-centos ~]# env LANG=C TZ=Asia/Shanghai /usr/bin/mrtg /etc/mrtg/test-zfs-host.cfg

[root@test-centos ~]# indexmaker --output=/var/www/mrtg/index.html /etc/mrtg/test-zfs-host.cfg

Now add cronjob:

0-59/5 * * * * env LANG=C TZ=Asia/Shanghai /usr/bin/mrtg /etc/mrtg/test-zfs-host.cfg

Visit http://<your linux box's ip address>/mrtg/ to get the GUI result(you'll wait 5 minutes for the initial result, be patient!)

  • Part 5 - troubleshooting

  • If you met error messages like the following:

[root@test-centos ~]# env LANG=C TZ=Asia/Shanghai /usr/bin/mrtg /etc/mrtg/test-zfs-host.cfg
Argument "v4only" isn't numeric in int at /usr/bin/../lib64/mrtg2/SNMP_Session.pm line 183.
backoff (v4only) must be a number >= 1.0 at /usr/bin/../lib64/mrtg2/SNMP_util.pm line 465

That's because you're using OIDs/MIBs alias rather than number. Change alias to number, i.e. change UCD-SNMP-MIB::ucdavis to .1.3.6.1.4.1.2021, and then re-check.

  • If you met error messages like the following:

[root@test-centos ~]# env LANG=C TZ=Asia/Shanghai /usr/bin/mrtg /etc/mrtg/test-zfs-host.cfg
cannot encode Object ID 34.4.1.2.28.122.102.115.45.115.108.99.101.49.48.115.110.48.49.45.110.105.99.45.98.97.110.100.119.105.100.116.104.3: first subid too big in Object ID 34.4.1.2.28.122.102.115.45.115.108.99.101.49.48.115.110.48.49.45.110.105.99.45.98.97.110.100.119.105.100.116.104.3 at /usr/bin/mrtg line 2035
Tuesday, 25 December 2012 at 0:34: ERROR: Target[zfs-test-zfs-host-io-average-latency-igb2][_IN_] ' $target->[3]{$mode} ' did not eval into defined data
Tuesday, 25 December 2012 at 0:34: ERROR: Target[zfs-test-zfs-host-io-average-latency-igb2][_OUT_] ' $target->[3]{$mode} ' did not eval into defined data

Then you should carefully check the scripts you write for the OIDs, and run several times of snmpwalk to ensure the values are correct(is your script's output variable, this may cause problems)

  • If you met error messages like the following:

[root@test-centos mrtg]# env LANG=C TZ=Asia/Shanghai /usr/bin/mrtg /etc/mrtg/test-zfs-host.cfg
SNMP Error:
Received SNMP response with error code
error status: noSuchName
index 2 (OID: 1.3.6.1.4.1.2021.44)
SNMPv1_Session (remote host: "localhost" [127.0.0.1].161)
community: "public"
request ID: 1786815468
PDU bufsize: 8000 bytes
timeout: 2s
retries: 5
backoff: 1)
at /usr/bin/../lib64/mrtg2/SNMP_util.pm line 490
SNMPGET Problem for .1.3.6.1.4.1.2021.44 .1.3.6.1.4.1.2021.44 sysUptime sysName on public@localhost::::::v4only
at /usr/bin/mrtg line 2035
SNMP Error:
Received SNMP response with error code
error status: noSuchName
index 2 (OID: 1.3.6.1.4.1.2021.45)
SNMPv1_Session (remote host: "localhost" [127.0.0.1].161)
community: "public"
request ID: 1786815469
PDU bufsize: 8000 bytes
timeout: 2s
retries: 5
backoff: 1)
at /usr/bin/../lib64/mrtg2/SNMP_util.pm line 490
SNMPGET Problem for .1.3.6.1.4.1.2021.45 .1.3.6.1.4.1.2021.45 sysUptime sysName on public@localhost::::::v4only
at /usr/bin/mrtg line 2035
Monday, 24 December 2012 at 14:41: ERROR: Target[zfs_space_test-zfs-host][_IN_] '( $target->[0]{$mode} ) * 10000 / ( $target->[1]{$mode} )' (warn): Use of uninitialized value in division (/) at (eval 16) line 1.
Monday, 24 December 2012 at 14:41: ERROR: Target[zfs_space_test-zfs-host][_OUT_] '( $target->[0]{$mode} ) * 10000 / ( $target->[1]{$mode} )' (warn): Use of uninitialized value in division (/) at (eval 17) line 1.

Then one possible culprit is that new net-snmp stop supporting "exec", use "extend" instead and re-try.

Note that you can also use "snmpd -f -Le" to check error messages related to snmpd.

PS:
Here's more links from where you can read more about net-snmp/mrtg:

  1. net-snmp FAQ http://www.net-snmp.org/FAQ.html
  2. mrtg configuration opstions http://oss.oetiker.ch/mrtg/doc/mrtg-reference.en.html
  3. SUN zfs storage SNMP - http://docs.oracle.com/cd/E22471_01/html/820-4167/configuration__services__snmp.html
  4. net-snmp extending agent functionality - http://linux.die.net/man/5/snmpd.conf (search for 'extending agent functionality' on this page)
  5. To get only numbers of snmpwalk, add -On. More info on http://linux.die.net/man/1/snmpcmd
  6. Now here's the image of the baby(click on it to see the larger one: