http://blogs.dootdoot.com/mike

After seeing a basic example of creating a custom OID for the zpool capacity, I really wanted to extend some additional statistics that are available using the zpool iostat command.

In it’s simplest form, you can use the following command to see two sets of statistics on a zpool:

# zpool iostat zpool_name 5 2
               capacity     operations    bandwidth
pool        alloc   free   read  write   read  write
----------  -----  -----  -----  -----  -----  -----
zpool_name  30.7G   247G      9     11  1.14M  1.32M
zpool_name  30.7G   247G      0      0      0      0

The command is very basic, run iostats on a storage pool named “zpool_name”. Run the statistics with a 5 second interval, show results for two iterations.

As a quick run down, (ignoring the first three lines that are simply headings) we have two lines of actual information. The first line is an “average” since the system was powered on, and the second is the statistics from the next 5 second interval since the command was run.

Since we are polling for SNMP information on approximately a 5m interval, we are really only interested in the very last line. We can use sed or a combination of grep and sed to isolate down to a single line, something like:

zpool iostate zpool_name 5 2 | grep zpool_name | sed -n 2p

Grep will return all lines with zpool_name in the line and then sed will only show the second line of those results.

Since this will take a minimum of 5 seconds to run to get the correct stats, the best option I can think of is to store our results in a cache file that SNMP can read from, then have the script run as a cron job every few minutes. Your data will be within a close enough result during every poll. It’s not perfect, but it’s close enough.

At this point we can combine a little linux magic to script that we can add to the crontab that will do most of the work for us. The script below will produce a /tmp/iostat.cache that is used for returning the SNMP data via NET-SNMP.

/opt/utils/iostats-create.sh:

#!/bin/bash
#
###############################################################
#
# Add the line below to crontab to create the iostat.cache
#
# * * * * * cd /tmp && /opt/utils/iostats-create.sh pool_name > io.tmp && mv io.tmp iostat.cache
#
###############################################################

let x=1
zpool iostat $1 5 2 | grep $1 | sed -n 2p |
while read line; do
        for i in $line; do
                if [ $x -eq 1 ]; then                   # if the first line item
                        echo $i                         #       print storage pool name
                else                                    # else print the number in bytes
                        echo $i | sed -e "s/K/*1024/g;s/M/*1024*1024/;s/G/*1024*1024*1024/;s/T/*1024*1024*1024*1024/" | bc | sed "s/[.].*//"
                fi
                ((x++))
        done
done

Since the data we’re given is already converted to human readable values, such as 1.2K, we’re fairly limited on our accuracy, but it’s better than nothing, and we can do our best to make it work.

Cacti needs all of the information in bytes to do it’s scaling (especially for some of the data like IOPS that will range from 0 to ~10K in our case).

Most of the scripts reads pretty well other than the sed command that converts any number from KB/MB/GB/etc to bytes by substituting each letter with the appropriate math (ex: substitute K with *1024) and then piping that into bc to get the result. I threw an extra sed on at the end to remove anything after the decimal place to keep it clean for Cacti.

The script contains the information required to add to the crontab. Once the system runs for a bit, you should eventually see something like this in your /tmp/iostat.cache file:

# cat /tmp/iostat.cache
zpool_name
32963873996
265214230528
0
0
0
0

Since this is a test system, there’s really nothing going on for me right now, so ignore the zero’s.

For fun, I also started a C++ program that did the same thing as the script. While I was able to kick it out faster than the time it took to learn how to do loops and if’s in bash as well as some nasty sed, the program is way uglier. However, it was fun to do:

/**********************************************
 * iostat.cpp
 * --------------------------------------------
 *
 * Calls the zpool iostat command using:
 *
 * zpool iostat $1 5 2 | grep $1 | sed -n 2p
 *
 * --------------------------------------------
 *
 * Converts the output of zpool iostat from
 * K/M/G/T to Bytes for use by net-snmp
 *
 *********************************************/

#include <iostream>
#include <iomanip>
#include <fstream>
#include <sstream>
#include <string>
using namespace std;

void    print_iostats(string filename, string name, double iostats[]);
string  exec(char* cmd);

int main(int argc, const char* argv[])
{
        string pool_name, line, result, sub, cmd, filename;
        double iostats[8];
        int i=0;

        if ( argc < 1 )
        {
                cout << endl;
                cout << "ERROR: " << argv[0] << " requires input parmater." << endl;
                cout << endl;
                cout << "usage: " << argv[0] << " tank" << endl;
                cout << "       where \'tank\' is the name of a zfs pool." << endl;
                cout << endl;
        }
        else
        {
                if( argv[2] )
                {
                        filename = argv[2];
                }
                else
                {
                        filename = "/tmp/io.cache";
                        cout << "No output filename provided." << endl;
                        cout << "Using: " << filename << endl;
                }

                pool_name = argv[1];

                cmd = "zpool iostat " + pool_name + " 5 2 | grep " + pool_name + " | sed -n 2p";
                char * cmdd = (char*)cmd.c_str();
                result = exec( cmdd );

                result.erase(remove(result.begin(), result.end(), '\n'), result.end());

                // set the pool name based on the given paramater
                istringstream iss(result);

                while (iss>>sub)
                {
                        if(i>0) // skip the storage pool name
                        {
                                // get items 2 - 7 of data and store them into IOSTAT[0-6]
                                // free space, used space, read ops, write ops, read bandwidth, write bandwidth
                                iostats[i-1] = atof( sub.c_str() );
                                        // convert to bytes (rough approximation based on available information)
                                if(sub[sub.length()-1] == 'K') {iostats[i-1] = iostats[i-1]*1024; }
                                if(sub[sub.length()-1] == 'M') {iostats[i-1] = iostats[i-1]*1024*1024; }
                                if(sub[sub.length()-1] == 'G') {iostats[i-1] = iostats[i-1]*1024*1024*1024; }
                                if(sub[sub.length()-1] == 'T') {iostats[i-1] = iostats[i-1]*1024*1024*1024*1024; }
                        }
                        i++;
                }
                print_iostats(filename, pool_name,iostats);
        }

        return 0;
}

void print_iostats(string filename, string name, double iostats[])
{
        ofstream iocache;
        iocache.open ( filename.c_str() );
        iocache << name << endl;
        iocache << fixed << setprecision (0);
        for(int i=0; i        {
                iocache << iostats[i] << endl;
        }

        iocache.close();
}

string exec(char* cmd)
{
        FILE* pipe = popen(cmd, "r");
        if (!pipe) return "ERROR";

        char buffer[128];
        string result = "";
        while(!feof(pipe)) {
                if(fgets(buffer, 128, pipe) != NULL)
                        result += buffer;
        }
        pclose(pipe);
        return result;
}

Like I said, it was a quick and ugly hack-job of a solution. I take no pride in it, other than it makes that simple bash script seem pretty nice.

Next, to take the data stored in iostat.cache and do something useful with it.

Comments

One Response to “ZFS Monitoring and Cacti – Part 3”

  1. ZFS Monitoring and Cacti – Part 4 : dsu | Mike on July 3rd, 2012 11:55 pm

    [...] fumbling around a bit with bash in Part 3, I got to a point where I could graph a single storage pool using the data that was being updated [...]

Leave a Reply