http://blogs.dootdoot.com/mike

After the mini-blunder with OpenIndiana picking up the Intel Quad-Port card out of order the other day, I figured everything should smooth sailing form here on out, as I’ve already setup link aggregation once before.

The Plan: Remove the current aggr0 on bnx0/bnx1 and create a new aggr0 on igb0/igb1, potentially adding in igb2/igb3 later, or possibly setting up igb2/igb3 as dedicated iSCSI ports instead (TBD).

Tearing down the link aggregation was easy:

root@storage:~# dladm delete-aggr aggr0

I temporarily brought up each device igb0, igb1, igb2, igb3 to test each port. Everything was good.

I then went and created a new aggr0 with the igb0/igb1 ports, and set a new static IP:

root@storage:~# dladm create-aggr -l igb0 -l igb1 aggr0
root@storage:~# ipadm create-addr -T static -a 192.168.1.100/24 aggr0/v4

Recycled the system, and everything came up as expected, but I received a nasty error message at the console on boot:

igb0: DL_BIND_REQ failed: DL_SYSERR (errno 16)
igb0: DL_UNBIND_REQ failed: DL_OUTSTATE
igb1: DL_BIND_REQ failed: DL_SYSERR (errno 16)
igb1: DL_UNBIND_REQ failed: DL_OUTSTATE
Failed to plumb IPv4 itnerface(s): igb0 igb1

Well, after a little digging, I found that when I tested the network connection I had created two files (/etc/hostname.igb0 and /etc/hostname.igb1) that each contained an IP address. The error above simply was letting me know that I was attempting to set an IP twice, as aggr0 was also setting this IP.

Deleted the two hostname.igb* files, rebooted, and again, everything came up as expected…

Well… maybe.

I got back to my desk to do a little testing, but I was seeing some very strange behavior. Of my two test systems, one of my test Linux boxes could connect via NFS and one couldn’t. In fact, the one that couldn’t had no access at all, even ping failed.

Rebooted, and both systems worked fine. Except now a third system was unable to connect. At this point it was obvious something was incorrectly configured.

After a little searching, I found this article that made me think there might be a driver problem somewhere. With a little more digging and a quick netstat -p, I found that  several of my connections were trying to use ports that weren’t connected.

Apparently I had left igb2 and igb3 “configured” but they did not have an actual Ethernet cable. For some reason, OpenIndiana was still trying to optimize when aggr0 was busy and connect on these ports even though they didn’t have a physical connection.

Ugh. Removed the final two /etc/hostname.igb* files, rebooted, and everything came back up as expected.

Really for real this time.

Comments

One Response to “OpenIndiana: Strange Network Problem”

  1. Marco Trujillo on May 7th, 2013 12:27 pm

    Hi,

    You must delete ip protocol over physical interfaces before configure them into link aggregation.

    eje.

    ipadm delete-ip igb0
    ipadm delete-ip igb1

Leave a Reply