http://blogs.dootdoot.com/mike

After fumbling around a bit with bash in Part 3, I got to a point where I could graph a single storage pool using the data that was being updated in our /tmp/iostat.cache file. While this is great for small ZFS setups, what happens if you have more than one pool?

Since I was creating this custom solution from scratch anyway, I figured I might as well make this a bit more flexible. I scraped my previous script since it was very focused on a singular pool, and set out to get stats for all pools. I used the zpool capacity done by “NiTRo” at hypervisor.fr as an example of how to collect stats for multiple pools and expanded my solution from the other day.

I started by creating a basic script to create a new iostat.cache file called iostats_make_cache.sh:

#!/bin/sh
#
###############################################################
#
# Add the line below to crontab to create the iostat.cache
#
# * * * * * cd /opt/utils/iostats && /opt/utils/iostats/iostats_make_cache.sh iostat.cache
#
###############################################################
#
# This script will produce the file iostat.cache that will be
# used by the various /opt/utils/isotat/* scripts for returning
# SNMP data via net-snmp.
#
###############################################################

FILENAME=$1                                             # Use file name from parameter
                                                        #
zpool iostat 5 2 | sed '/-----/d' | sed '1,2d' > $1     # dump zpool stats of all pools (and remove headings)
                                                        # from a 5 second interval to filename $1
                                                        #
N=`wc -l < $1`                                          # get number of lines
((N=$N/2));                                             # divide by 2
                                                        #
sed '1,'$N'd' -i $1                                     # delete first half of file

This very basic script takes a single parameter for the output filename. The script runs the zpool iostat command for a 5 second interval and strips the output down removing the formatting, line separators and the useless “I/O stats since power on” which are not helpful in determining real-time information.

The resulting iostat.cache file looks like:

rpool       11.5G   533G      0     15      0  66.6K
storage      258K   278G      0      0      0      0

Very basic, but we have a an easy to access file that has up-to-date I/O stats.

Next, I created a very simple script called iostats_get_item.sh to retrieve an individual item for all storage pools (such as Read IOPS) from the iostat.cache file:

#!/bin/bash
#
###############################################################
#
# Usage:   ./iostats_get_item.sh <#> <io_cache.file>
#
# The parameter # has the following results
#
#       1 - Storage Pool Name
#       2 - Allocated Space
#       3 - Free Space
#       4 - Read IOPS
#       5 - Write IOPS
#       6 - Read Bandwidth
#       7 - Write Bandwidth
#
# Ex:  ./iostats_get_item.sh 1 /tmp/iostat.cache
#
###############################################################

FILENAME=$2
let x=1
cat $2 |
while read line; do
        for i in $line; do
                if [ $x -eq $1 ]; then          # if the first line item
                        if [ $1 -eq 1 ]; then
                                echo $i         # print storage pool name
                        else
                                echo $i | sed -e "s/K/*1024/g;s/M/*1024*1024/;s/G/*1024*1024*1024/;s/T/*1024*1024*1024*1024/" | bc | sed 's/[.].*//'
                        fi
                fi
                ((x++))
        done
        ((x=1))
done

Using this script, I can now easily return the information for each individual stat via SNMP for all zpools, ex:

~#: ./iostats_get_item.sh 1 iostat.cache
rpool
storage

~#: ./iostats_get_item.sh 2 iostat.cache
12348030976
264192

At this point you can use the NET-SNMP-EXTEND-MIB to extend the data by calling the script and passing the appropriate parameters. Unfortunately, NET-SNMP-EXTEND was a little flaky with passing parameters. Depending on how you use the extend/pass features, I had mixed results.

To guarantee things work, and keep things more readable, we created an individual script files to run the actual commands:

~#: cat getName.sh
/opt/utils/iostats/iostats_get_item.sh 1 /opt/utils/iostats/iostat.cache

~# cat getAllocSpace.sh
/opt/utils/iostats/iostats_get_item.sh 2 /opt/utils/iostats/iostat.cache

~# cat getFreeSpace.sh
/opt/utils/iostats/iostats_get_item.sh 3 /opt/utils/iostats/iostat.cache

~# cat getReadIOP.sh
/opt/utils/iostats/iostats_get_item.sh 4 /opt/utils/iostats/iostat.cache

~# cat getWriteIOP.sh
/opt/utils/iostats/iostats_get_item.sh 5 /opt/utils/iostats/iostat.cache

~# cat getReadBand.sh
/opt/utils/iostats/iostats_get_item.sh 6 /opt/utils/iostats/iostat.cache

~# cat getWriteBand.sh
/opt/utils/iostats/iostats_get_item.sh 7 /opt/utils/iostats/iostat.cache

It seemed a little self-defeating after putting the work into a single script that can get the individual items, but at least this way we know it will always works as intended… but at least I got to use the script.

At this point, simply add the following lines to your SNMP configuration file:

########################################################################################
# Capacity
########################################################################################

extend .1.3.6.1.4.1.2021.87 zpool_name /usr/gnu/bin/sh /opt/utils/capacity/zpools_name.sh
extend .1.3.6.1.4.1.2021.87 zpool_capacity /usr/gnu/bin/sh /opt/utils/capacity/zpools_capacity.sh

extend .1.3.6.1.4.1.2021.88 zio_name /usr/gnu/bin/sh /opt/utils/iostats/getName.sh
extend .1.3.6.1.4.1.2021.88 zio_allo /usr/gnu/bin/sh /opt/utils/iostats/getAllocSpace.sh
extend .1.3.6.1.4.1.2021.88 zio_free /usr/gnu/bin/sh /opt/utils/iostats/getFreeSpace.sh

########################################################################################
# IO STATS
########################################################################################

############################# IOPS ############################################

extend .1.3.6.1.4.1.2021.89 zio_name /usr/gnu/bin/sh /opt/utils/iostats/getName.sh
extend .1.3.6.1.4.1.2021.89 zio_readIOPs /usr/gnu/bin/sh /opt/utils/iostats/getReadIOP.sh
extend .1.3.6.1.4.1.2021.89 zio_writeIOPs /usr/gnu/bin/sh /opt/utils/iostats/getWriteIOP.sh

############################ Bandwidth ########################################

extend .1.3.6.1.4.1.2021.90 zio_name /usr/gnu/bin/sh /opt/utils/iostats/getName.sh
extend .1.3.6.1.4.1.2021.90 zio_readBand /usr/gnu/bin/sh /opt/utils/iostats/getReadBand.sh
extend .1.3.6.1.4.1.2021.90 zio_writeBand /usr/gnu/bin/sh /opt/utils/iostats/getWriteBand.sh

Now we’re ready to make some graphs!

Comments

3 Responses to “ZFS Monitoring and Cacti – Part 4”

  1. Darren on October 17th, 2012 11:32 pm

    Looks good,
    are you going to make the cacti templates available

    Cheers

  2. Mike on October 24th, 2012 10:38 am

    Thanks Darren. I certainly can. I will see what I can do to put these together and make them available soon.

  3. David on July 19th, 2013 6:04 pm

    Has this been published as a Cacti template?

    Thanks!

Leave a Reply