9 Replies Latest reply on Jul 16, 2013 7:14 PM by Berserk

    How to do NIC teaming (LACP) and VLANs on Windows Server 2008 R2 Hyper-V?

    Berserk

      First a bit about my environment:

       

      • Dell PowerEdge 2970
      • Intel Quad-port I340-T4 10/100/1000 Ethernet HBA
      • Intel driver version: 12.7.28.0 (installed drivers and PROSet using PROWinx64.exe, v. 18.3)
      • Microsoft Windows Server 2008 R2 Datacenter w/SP1 (Core, with Hyper-V role)
      • Cisco (Linksys) SRW2016 16-port full GbE (10/100/1000) switch

       

      I used PROSetCL.exe and the SRW2016 web management interface to configure a 3-port LAG (in LACP mode). The NIC team comes up fine and I'm able to use it for "normal" network stuff. The teamed interface is used by the virtual servers / guests for network connectivity. I'm not doing anything with VMDq yet, although I'm interested in exploring it after I get VLANs working with teaming. Here is the output for PROSetcl.exe Team_GetTeamInfo 1:

       

      1) TEAM : VMNetwork

       

         Team Information:

       

             Team GUID       - '{305A7788-9AFC-4DD4-9B47-2CF0CF70412B}'

             Team MAC        - '001B21AEB9A5'

             Team Mode       - 'IEEE 802.3ad Dynamic Link Aggregation'

             Team Name       - 'VMNetwork'

       

      And the output for PROSetcl.exe Team_EnumerateAdaptersInTeam 1:

       

      1) TEAM : VMNetwork

       

         Number of adapters currently present: 3

       

             4) TEAM : VMNetwork - Intel(R) Ethernet Server Adapter I340-T4

             3) TEAM : VMNetwork - Intel(R) Ethernet Server Adapter I340-T4 #4

             2) TEAM : VMNetwork - Intel(R) Ethernet Server Adapter I340-T4 #3

       

      On the SRW2016, the LAG corresponding to these 3 network interfaces is configured for VLAN trunking, untagged in VLAN 1 and tagged in all other VLANs. Somewhere in my explorations, I stumbled on a suggestion to also configure individual ports for VLANs, but when I tried this on the SRW2016, it wouldn't let me add the ports to a LAG. So, I can configure VLANs on a LAG, but the individual ports (apparently in trunk mode) are excluded from all VLANs.

       

      I haven't set any VLAN configuration (using PROSetCL.exe) on the team (or individual NICs) in Hyper-V, and the related documentation I've read suggests that not only is it unnecessary, doing so is undesirable. My understanding is that if I set any VLAN configuration at the Hyper-V host level, then I must configure all VLANs I wish to be used by Hyper-V guests.

       

      See:

      http://www.intel.com/support/network/sb/cs-030993.htm

      http://social.technet.microsoft.com/wiki/contents/articles/151.hyper-v-virtual-networking-survival-guide.aspx#Understanding_Hyper-V_VLANs

      http://www.aidanfinn.com/?p=10164

       

      For Hyper-V host management, I'm using a separate / dedicated NIC (not the teamed NIC interface) not involving VLANs. So, in Hyper-V's virtual switch connected to the teamed network interface, the option Allow management operating system to share this network adapter is unchecked, and there are no VLAN settings at that level.

       

      One Hyper-V guest (running Windows Server 2008 R2 Standard) is multi-homed (two hybrid virtual network adapters). One of the adapters is configured for VLAN identification (in Hyper-V's settings, not inside the guest OS). Inside the guest OS, an appropriate IP address is assigned to the adapter, but the guest cannot communicate with other hosts on the VLAN it is configured for.

       

      I'm just now beginning to think I should be using VMLB instead of LACP...? Am I correct in thinking that with VMLB, teaming is one-sided, meaning the physical switch knows nothing of the team configuration, and physical switch ports are configured individually? Or do I need a physical switch that understands / supports VMLB?

       

      I would appreciate any suggestions on troubleshooting or getting this to work.

       

      Thank you!

        • 1. Re: How to do NIC teaming (LACP) and VLANs on Windows Server 2008 R2 Hyper-V?
          Berserk

          Changing the team to VMLB was less painful than expected. On the Core Server with the Hyper-V role:

           

          PROSetCL.exe Team_ModifyTeamInfo 1 mode VMLB

           

          In SRW2016 web management, I removed the LAG. The ports are now individually managed on the physical switch. This made no difference with VLAN functionality, even after configuring the physical switch ports for VLAN trunking and tagging the VLANs I want to be able to access from my virtual servers / guests.

           

          I confirmed my VLAN configuration up to the SRW2016 by configuring a laptop for one of my tagged VLANs and connecting it to a free port on the SRW2016. I was able to ping several systems on that VLAN as expected, but wasn't able to ping one of the virtual servers / guests that has a network connection configured for that VLAN. Not sure what I'm missing...

          • 2. Re: How to do NIC teaming (LACP) and VLANs on Windows Server 2008 R2 Hyper-V?
            mark_h_@intel

            The setup you are doing where you have an LAG team that you use for your virtual switch should work. You are correct that a VMLB team does not need switch support, but you shouldn't have to use that team type.

             

            I am not familiar with having multiple virtual NICs for the VM. You might want to simplify the configuration for troubleshooting and only use one NIC to check your communication on the VLAN. You might want to try a static IP from the appropriate VLAN on the remaining virtual NIC. If everything works, then you could try adding the second virtual NIC.

             

            Mark H

             

             


            • 3. Re: How to do NIC teaming (LACP) and VLANs on Windows Server 2008 R2 Hyper-V?
              mark_h_@intel

              Your last post happened while I was typing mine, so I didn't see it until after I replied. Are you still running with two virtual NICs in the VM? Are you configuring the VLAN in the VM virtual NIC only? Or are you configuring the VLAN on the virtual network and/or on the team? Just configuring the VLAN on the virtual NIC of the VM should be enough, but if you have more than one virtual NIC, maybe the VM doesn't know which one to use.

              • 4. Re: How to do NIC teaming (LACP) and VLANs on Windows Server 2008 R2 Hyper-V?
                Berserk

                mark_h_@intel wrote:

                 

                Are you still running with two virtual NICs in the VM?

                Yes. One connects to the default / untagged VLAN. The other is configured (in the Hyper-V VM settings) to "Enable virtual LAN identification", with the appropriate VLAN id in the text box provided.

                 

                mark_h_@intel wrote:

                 

                Are you configuring the VLAN in the VM virtual NIC only? Or are you configuring the VLAN on the virtual network and/or on the team?

                In the VM virtual NIC only (in the Hyper-V network adapter settings for the VM in question). I'm not configuring the VLAN on the virtual network or on the team.

                 

                mark_h_@intel wrote:

                 

                Just configuring the VLAN on the virtual NIC of the VM should be enough, but if you have more than one virtual NIC, maybe the VM doesn't know which one to use.

                I also thought that should be sufficient, but I'm obviously missing something. IP configuration is set statically on both network interfaces inside the VM, and there are no connectivity problems with the adapter connected to the default / untagged VLAN. The adapter for the tagged VLAN is the one with connectivity problems (can't ping other hosts on that VLAN, and vice versa).

                • 5. Re: How to do NIC teaming (LACP) and VLANs on Windows Server 2008 R2 Hyper-V?
                  mark_h_@intel

                  I have heard from some of the other people at Intel who believe that the VM can't tell which interface belongs to which virtual network. So there might be a problem with the dual virtual NIC configuration. I would try removing the virtual NIC for the untagged VLAN to see if the tagged VLAN starts working.

                   

                  Mark H

                  • 6. Re: How to do NIC teaming (LACP) and VLANs on Windows Server 2008 R2 Hyper-V?
                    Berserk

                    mark_h_@intel wrote:

                     

                    I would try removing the virtual NIC for the untagged VLAN to see if the tagged VLAN starts working.

                     

                    Mark H

                    I gave it a shot: shutdown the VM, removed the virtual NIC for the untagged VLAN, and booted the VM. No difference. I'm not exactly surprised, though. In my exploration, I stumbled on several references to configuring VMs with multiple network adapters / VLANs. Here is one.

                    • 7. Re: How to do NIC teaming (LACP) and VLANs on Windows Server 2008 R2 Hyper-V?
                      Berserk

                      Possible clues...?

                       

                      Notice the output for PROSetCL.exe Adapter_EnumerateSettings 1 (a non-teamed network interface):

                       

                      1) Intel(R) Ethernet Server Adapter I340-T4 #2

                       

                         Settings:

                       

                             LLIPorts                 -

                             NetworkAddress           -

                             *ReceiveBuffers          - 256

                             *RssBaseProcNumber       - 0

                             *TransmitBuffers         - 512

                             EnableLLI                - Disabled

                             *FlowControl             - Rx & Tx Enabled

                             *InterruptModeration     - Enabled

                             *IPChecksumOffloadIPv4   - Rx & Tx Enabled

                             *JumboPacket             - Disabled

                             *LsoV2IPv4               - Enabled

                             *LsoV2IPv6               - Enabled

                             *MaxRssProcessors        - 8

                             *NumaNodeId              - System Default

                             *PriorityVLANTag         - Priority & VLAN Enabled

                             *RSS                     - Enabled

                             *SpeedDuplex             - Auto Negotiation

                             *TCPChecksumOffloadIPv4  - Rx & Tx Enabled

                             *TCPChecksumOffloadIPv6  - Rx & Tx Enabled

                             *UDPChecksumOffloadIPv4  - Rx & Tx Enabled

                             *UDPChecksumOffloadIPv6  - Rx & Tx Enabled

                             *VMQ                     - Disabled

                             *WakeOnMagicPacket       - Enabled

                             *WakeOnPattern           - Enabled

                             EnablePME                - Disabled

                             ITR                      - Adaptive

                             LogLinkStateEvent        - Enabled

                             MasterSlave              - Auto Detect

                             NumRssQueues             - 1 Queue

                             ReduceSpeedOnPowerDown   - Enabled

                             WaitAutoNegComplete      - Auto Detect

                             WakeOnLink               - Disabled

                       

                      Here is the output for PROSetCL.exe Adapter_EnumerateSettings 2 (an interface in the VMLB team):

                       

                      2) TEAM : VMNetwork - Intel(R) Ethernet Server Adapter I340-T4 #3

                       

                         Settings:

                       

                             LLIPorts                 -

                             NetworkAddress           - 001B21AEB9A5

                             *ReceiveBuffers          - 256

                             *RssBaseProcNumber       - 0

                             *TransmitBuffers         - 512

                             EnableLLI                - Disabled

                             *FlowControl             - Rx & Tx Enabled

                             *InterruptModeration     - Enabled

                             *IPChecksumOffloadIPv4   - Rx & Tx Enabled

                             *JumboPacket             - Disabled

                             LSO                      - Enabled

                             *LsoV2IPv6               - Enabled

                             *MaxRssProcessors        - 8

                             *NumaNodeId              - System Default

                             *RSS                     - Enabled

                             *SpeedDuplex             - Auto Negotiation

                             *TCPChecksumOffloadIPv4  - Rx & Tx Enabled

                             *TCPChecksumOffloadIPv6  - Rx & Tx Enabled

                             *UDPChecksumOffloadIPv4  - Rx & Tx Enabled

                             *UDPChecksumOffloadIPv6  - Rx & Tx Enabled

                             *VMQ                     - Disabled

                             ITR                      - Adaptive

                             LogLinkStateEvent        - Enabled

                             MasterSlave              - Auto Detect

                             NumRssQueues             - 1 Queue

                             WaitAutoNegComplete      - Auto Detect

                       

                      Note the absence of Priority / VLAN settings in the enumerated settings for the teamed interface.

                       

                      Finally, here is the output for PROSetCL.exe Team_EnumerateSettings 1:

                       

                      1) TEAM : VMNetwork

                       

                         Settings:

                       

                             BalanceInterval          - 10

                             CheckTime                - 1000

                             MaxNumProbeRetry         - 10

                             STForwardDelay           - 0

                             FailbackEnabled          - Enabled

                             ProbeEnabled             - Enabled

                             ProbePacketType          - Broadcast Probes

                       

                      Again, no settings for Priority / VLAN.

                       

                      As a test, I removed one of the teamed network interfaces from the VMLB team and created a new Hyper-V virtual network (external) connected to the interface removed from the team. On the VM with 2 virtual network adapters, I configured the tagged-VLAN adapter to connect to the newly created virtual network (instead of the virtual network associated with the VMLB team). After booting the VM, I successfully tested basic connectivity on the default (untagged) VLAN and the tagged VLAN.

                       

                      Are VLANs unsupported on I340-T4 teamed interfaces? I haven't seen anything explicitly stating so, and have seen documentation, threads and posts suggesting the opposite. Well, maybe not for the I340-T4 in particular, but teamed Intel server nics in general.

                      • 8. Re: How to do NIC teaming (LACP) and VLANs on Windows Server 2008 R2 Hyper-V?
                        mark_h_@intel

                        Yes, VLANs are supported on a team interface. You can explicitly configure VLANs on the teamed interface using PROSetCL, but that is not supposed to be necessary. However, you might want to give it a try.

                         

                        When you configure VLANs on a physical port, a hardware filter is created so that VLANs that do not match get blocked. I'm not sure how that differs on the teamed interface. Anyway, the point is, with filters setup using PROSetCL, you will have to make sure that you assign all the VLAN numbers on the trunk and an untagged VLAN. Make sure you configure all VLANs that are used by VMs connected to the HyperV network that uses the team. Any tagged VLAN traffic that does not have a match will get blocked and dropped by the team. And if you do not configure the untagged VLAN, any untagged traffic would also get dropped.

                         

                        Mark H

                        • 9. Re: How to do NIC teaming (LACP) and VLANs on Windows Server 2008 R2 Hyper-V?
                          Berserk

                          Hi Mark,

                           

                          Thanks for the follow-up. I had already tried adding VLANs to the team interface, and I was aware of the requirement to configure all VLANs (used by VMs) if any are defined at the interface level. Here is what happens when I try using PROSetCL.exe to add a 'default' (untagged) VLAN:

                           

                          > PROSetCL.exe Team_CreateVlan 1 untagged default

                           

                          1) TEAM : VMNetwork

                           

                             Attempting to create VLAN 'default' ...

                           

                                 Failed to create VLAN

                           

                          and if I try to add a VLAN for our IP phone VLAN:

                           

                          > PROSetCL.exe Team_CreateVlan 1 5 IP_PHONES

                           

                          1) TEAM : VMNetwork

                           

                             Attempting to create VLAN 'IP_PHONES' ...

                           

                                 Failed to create VLAN

                           

                          I checked Windows' event logs but didn't find anything additionally relevant. Recall from my previous post how priority / VLAN settings don't appear in Adapter_EnumerateSettings output for teamed network interfaces, and never appear for the team itself, in the output for Team_EnumerateSettings. Perhaps incorrectly, this suggested to me that (at least for my combination of hardware, drivers, etc.) prioritization and VLANs are unsupported on teams...

                           

                          For now, I've reserved a non-teamed interface for our non-default (tagged) VLANs. This works, but isn't an "ideal" solution, and doesn't satisfy my curiosity.