0 Replies Latest reply on Nov 17, 2014 5:25 PM by Kai_Lanz

    E1G44ET2 server adapter: port order changes on reboot

    Kai_Lanz

      I'm posting here in the hope that someone has seen symptoms like this and can

      offer some insight as to what's going on.

       

      Executive summary: We have a newly-installed E1G44ET2BLK 4-port server adapter;

      following a server reboot, we found that the ethernet jacks on that card had

      been reordered! How did that happen, and how do we prevent it?

       

      Here's how ifconfig saw this card after it was installed, with an ethernet

      cable plugged into the bottom jack:

       

      igb0: flags=1000802<BROADCAST,MULTICAST,IPv4> mtu 1500 index 13

              inet 0.0.0.0 netmask 0

      igb1: flags=1000802<BROADCAST,MULTICAST,IPv4> mtu 1500 index 14

              inet 0.0.0.0 netmask 0

      igb2: flags=1000802<BROADCAST,MULTICAST,IPv4> mtu 1500 index 12

              inet 0.0.0.0 netmask 0

      igb3: flags=1001000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4,FIXEDMTU> mtu 1500 index 5

              inet 171.64.175.54 netmask fffff800 broadcast 171.64.175.255

       

      We checked every port on the card one at a time by plugging in an ethernet

      cable and running ifconfig to see which network interface showed valid link

      status (the RUNNING bit). We labeled the jacks b0 through b3 on the back panel; no

      surprise, they were ordered top-to-bottom, b0 to b3.

       

      After a server reboot, ifconfig showed the following:

       

      igb0: flags=1000802<BROADCAST,MULTICAST,IPv4> mtu 1500 index 13

              inet 0.0.0.0 netmask 0

      igb1: flags=1000802<BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 14

              inet 0.0.0.0 netmask 0

      igb2: flags=1000802<BROADCAST,MULTICAST,IPv4> mtu 1500 index 12

              inet 0.0.0.0 netmask 0

      igb3: flags=1001000843<UP,BROADCAST,MULTICAST,IPv4,FIXEDMTU> mtu 1500 index 5

              inet 171.64.175.54 netmask fffff800 broadcast 171.64.175.255

       

      Notice that igb3 is no longer RUNNING; igb1 shows as RUNNING, instead. Yet the

      cable had not been moved! It's still plugged in to the bottom jack. What's

      changed is the mapping from network device name to physical port: the bottom

      jack, which used to be igb3, is now igb1. And igb1, which used to be the 2nd

      jack down from the top, is now the bottom jack. Our carefully-applied labels

      are now wrong!

       

      To restore network connectivity, I had to move the cable from the bottom jack

      to the 2nd jack down. This restored RUNNING status to igb3.

       

      We can't live with network jacks that move around on our server every time

      we reboot. How is this remapping happening? Is this a feature of the E1G44ET2

      card? An issue with the card's firmware? A device driver or OS problem?

       

      The OS is NexentaStor v3.1.3 Enterprise (SunOS Release 5.11, OpenSolaris

      Build 134). Server node is based on the Xyratex OneStorHS 1235T chassis.

      Driver version is apparently 1.1.18:

       

      # modinfo | grep igb

      179 fffffffff8084000  1e178 284   1  igb (Intel 1Gb Ethernet)

       

      # strings /kernel/drv/amd64/igb | grep igb

      igb 1.1.18

       

      -- Kai Lanz            Stanford University        School of Earth Sciences