9 Replies Latest reply on Jun 1, 2012 7:38 PM by sparks

    Intel NIC cards bleeding tagged vLANs into non-tagged vLANs (IPv6)

    sgonsalv

      Hi Guys,

       

      First time i'm logging a case on this forum.

       

      We have an interesting issue that i'd like to flag with Intel, which can be summarised as follows.  Note, this issue is only seen on Windows machines, not Apple or Linux.

      1. We have Windows machines (XP or Vista or 7) connected to Cisco switches which have a standard config attached to it.  Here the switchport has both an access vlan (data) and a voice VLAN as shown below:

       

      switchport access vlan 84
      switchport mode access
      switchport nonegotiate
      switchport voice vlan 729
      switchport port-security maximum 3
      switchport port-security maximum 2 vlan access
      switchport port-security maximum 1 vlan voice
      switchport port-security
      switchport port-security aging time 1
      switchport port-security violation restrict
      switchport port-security aging type inactivity
      ipv6 traffic-filter RA-FILTER in
      no mdix auto
      storm-control broadcast level 1.00
      spanning-tree portfast
      spanning-tree bpduguard enable

       

      For the most part, we have thousands of Cisco IP phones deployed across the network, such that the Phone is inline with the PC - no issue here.

       

      The issue we see is when a PC is directly attached to the switchport.  Here, when IPv6 is enabled on the NIC card, the driver rather than discarding the tagged Voice vLAN traffic, presents the IPv6 router advertisments from the Voice vLAN to the data vLAN.  As such the RA packets bleed into the data vLAN.

      This causes the TCP/IP stack in microsoft to install 2 IPv6 addresses, one for the data vLAN and one for the voice vLAN!  IPv6 breaks as a result.

       

      The other related issue is that we've tried lowering the voice vLANs router priority and prefix lifetime such that the NIC card preferences the data vLAN over the voice vLAN.  This doesn't work either.

       

      I've tried downloading the most current driver for my Intel NIC card to no avail.  A ticket to troubleshoot this issue was logged with Microsoft, however they've pointed out that this is a driver issue that needs to be looked at by Intel.

       

      Have you seen such a problem before? or have a driver that resolves both issues?

       

      My NIC card is: Intel 82567LM Gigabit Network Connection, running 10.1.9.0 (4/7/2010).

       

      Let me know if you have any questions.

      thanks

      Sheldon

        • 1. Re: Intel NIC cards bleeding tagged vLANs into non-tagged vLANs (IPv6)
          freddyboomboom

          Is the second IPv6 address the link-local IPv6 address?

           

          Can you copy and paste the output of "ipconfig /all" into this thread? It would be nice to have it from a system connected to the switch, and a system connected to a phone.

           

          Are you actually using IPv6 in your environment?

           

          What is the model of Cisco phone you are using?

           

          Have you taken Wireshark traces from both scenarios (connected to switch and connected to phone) and analyzed them?

          • 2. Re: Intel NIC cards bleeding tagged vLANs into non-tagged vLANs (IPv6)
            freddyboomboom

            Another question, since I'm reading up on this...

             

            How do you have your voice vlan trunk setup, isl or dot1q?

             

            Or is it not trunked and the data vlan is trunked?

            • 3. Re: Intel NIC cards bleeding tagged vLANs into non-tagged vLANs (IPv6)
              sgonsalv

              Hi Freddy,

               

              Comments in line below:

               

              >>> Is the second IPv6 address the link-local IPv6 address?

               

              The second IPv6 address is not the link-local addess, it is based on the IPv6 prefix for the Voice VLAN (e.g. 2001:388:608c:28da::/64) as seen below in my ipconfig /all output.

               

              >>> Can you copy and paste the output of "ipconfig /all" into this thread? It would be nice to have it from a system connected to the switch, and a system connected to a phone.

               

              This is the output when connected to the switchport directly:

               

              Ethernet adapter Local Area Connection 4:

               

                      Connection-specific DNS Suffix  . : its.monash.edu.au
                      Description . . . . . . . . . . . : Intel(R) 82566DM Gigabit Network Connection
                      Physical Address. . . . . . . . . : 00-19-BB-D7-7E-34
                      Dhcp Enabled. . . . . . . . . . . : Yes
                      Autoconfiguration Enabled . . . . : Yes
                      IP Address. . . . . . . . . . . . : 130.194.85.158
                      Subnet Mask . . . . . . . . . . . : 255.255.254.0
                      IP Address. . . . . . . . . . . . : 2001:388:608c:28da:7dd9:bb14:60fa:a1aa
                      IP Address. . . . . . . . . . . . : 2001:388:608c:28da:219:bbff:fed7:7e34
                      IP Address. . . . . . . . . . . . : 2001:388:608c:4882:7dd9:bb14:60fa:a1aa
                      IP Address. . . . . . . . . . . . : 2001:388:608c:4882:44e4:d1ea:3e51:27b5
                      IP Address. . . . . . . . . . . . : 2001:388:608c:4882:219:bbff:fed7:7e34
                      IP Address. . . . . . . . . . . . : fe80::219:bbff:fed7:7e34%20
                      Default Gateway . . . . . . . . . : 130.194.85.254
                                                          fe80::5:73ff:fea0:6%20
                      DHCP Server . . . . . . . . . . . : 130.194.15.1
                      DNS Servers . . . . . . . . . . . : 130.194.1.99
                                                          130.194.7.99
                                                          fec0:0:0:ffff::1%1
                                                          fec0:0:0:ffff::2%1
                                                          fec0:0:0:ffff::3%1
                      Lease Obtained. . . . . . . . . . : Thursday, September 16, 2010 11:03:39 AM
                      Lease Expires . . . . . . . . . . : Thursday, September 16, 2010 11:03:39 PM

               


              The output when connected directly to the phone is the same as above, except, without the '2001:388:608c:28da::/64' addresses.  FYI, the following prefix 2001:388:608c:4882::/64 is for the data VLAN.

               

              >>> Are you actually using IPv6 in your environment?

               

              Yes we use it within our intranet and also when connecting to certain external sites (e.g. Google)

               


              >>> What is the model of Cisco phone you are using?

               

              Cisco IP Phone 7945

               

               

               

              >>> Have you taken Wireshark traces from both scenarios (connected to switch and connected to phone) and analyzed them?

               

              Yes have done this.  When looking at a trace when the machine is directly connected to the port, we observe RA messages from both the VOice and Data VLANs, however this isn't the case when the machine is connected via the phone.  Only the Data VLAN RA messages are seen as the phone filters the voice RA messages from the directly attached PC.  Further, all the RA messages are sourced from the router with the same Link local adress: ( as an example fe80::5:73ff:fea0:6).  All SVIs on the router use the same Link local address - the destination here is to all devices (ff02::1).  SO the only way to distinguish the traffic is based on the fact that the data VLAN is not tagged, but the Voice VLAN traffic is.  The NIC card should therefore ignore the Voice VLAN packets, and only present the data VLAN to the machine, as it happens under Ubuntu or MAC for instance.

               

              The other issue is that the NIC card driver does not honor the router priority set...which we've set to low for the Voice VLAN, and also set the prefix lifetime to be lower than the data VLAN.

               


              >>> Another question, since I'm reading up on this...how do you have your voice vlan trunk setup, isl or dot1q?
              Or is it not trunked and the data vlan is trunked?

               


              The switchport is setup as follows.  As such, the Data VLAN is not tagged, but the voice VLAN is tagged (dot1q)

               

              switchport access vlan 84  <<<<<<<<<<<<<<<<<<<<<  Data VLAN
              switchport mode access
              switchport nonegotiate
              switchport voice vlan 750  <<<<<<<<<<<<<<<<<<<<<  VOice VLAN
              switchport port-security maximum 3
              switchport port-security maximum 2 vlan access
              switchport port-security maximum 1 vlan voice
              switchport port-security
              switchport port-security aging time 1
              switchport port-security violation restrict
              switchport port-security aging type inactivity
              ipv6 traffic-filter RA-FILTER in
              no mdix auto
              storm-control broadcast level 1.00
              spanning-tree portfast
              spanning-tree bpduguard enable

               


              Let me know if you have any other questions.

               

              thanks

              Sheldon

              • 4. Re: Intel NIC cards bleeding tagged vLANs into non-tagged vLANs (IPv6)
                sparks

                Sheldon,

                Did you ever resolve this? We are having the exact same problem. Windows 7 machines are self-configuring IPv6 addresses on both the user and VoIP VLANs and causing connectivity problems.

                -Michael

                • 5. Re: Intel NIC cards bleeding tagged vLANs into non-tagged vLANs (IPv6)
                  sgonsalv

                  Hi Sparks,

                   

                  We couldn't get anywhere with Microsoft so we decided to tackle this from the switch end.  At the moment we have suppresed the RAs on the VoIP subnets using:

                  ipv6 nd ra suppress

                   

                  ..since the VoIP phones at this stage don't support IPv6.  Separately I successfully tested the use of smartport macros on the switch to only present the Voice VLAN whenever a phone is connected to the switchport, and just the standard data port when there isn't a phone connected.

                   

                  Hope this helps.

                  Cheers

                  Sheldon

                  • 6. Re: Intel NIC cards bleeding tagged vLANs into non-tagged vLANs (IPv6)
                    sparks

                    Well I have a workaround involving the drivers. Make sure you have the current Intel PROset driver installed (this has VLAN functions the standard driver does not). Then from Device Manager, configure the physical adapter to create a virtual adpater on the untagged VLAN. One catch: you have to configure a tagged VLAN virtual adapter before you can create an untagged VLAN virtual adapter, so just create one on any VLAN (it won't be used), create the untagged virtual adapter, and then disable the tagged one. If you delete it you lose them both. Not really something I want to do across hundres of machines...

                     

                    This works with Intel adapters, but the same approach doesn't work with Broadcom adapters and their BASP utility. For Broadcom I've found you have to select to disable VLAN in the adpater properties and also set a registry value to drop tagged packets. You have to do both.

                     

                    This is definitely a Microsoft problem with the way they handle VLAN tagged traffic. I don't see this on my Mac which has the same Broadcom chip as some of my Dells that do have the problem. However, it happens with ethernet chipsets from multiple vendors on my Windows boxes.

                    • 7. Re: Intel NIC cards bleeding tagged vLANs into non-tagged vLANs (IPv6)
                      sparks

                      Here's an update. Some guys in our central networking group talked with both Microsoft and Broadcom and figured out that this is intended behavior per Microsoft's NDIS specs. If the driver isn't configured for a VLAN, then incoming VLAN tagged traffic is to be converted to untagged. Makes no sense to me since Windows will receive the de-tagged traffic and have no way of properly replying to it on the correct VLAN. Seems to me if the driver isn't configured for a tagged VLAN then tagged traffic should just be ignored (as sane OSs like OSX and Linux already do).

                       

                      Configuring NIC drivers as I described in my first reply was going to be too much work for all our machines and varying NICs. Since this is a purely Windows problem, I created a Windows Group Policy setting for the Windows firewall to drop unsolicited RAs. The specific filter is for ICMPv6 RA addressed to the all-nodes multicast address (ff02::1). Windows still gets the unicast RAs in reply to Router Solicitations so IPv6 self-addressing still works fine. Since SLAAC still works and I don't know of any other need Windows would have for those dropped RAs, I think this is a safe thing to do. It's working for us.

                       

                      I'm not sure if the RA suppression on the VLAN as you describe suppresses both multicast and unicast RAs, but if there's a way to suppress just the multicast RAs at the network level that would also probably be a fair solution, as long any IPv6 device on your network issues RSs. I think I prefer the Windows firewall rule though, since this is really a Windows-only problem and there's no need to break multicast RAs for the whole network if you can do it just for Windows.

                       

                      Best,

                      Michael

                       

                      Message was edited by: Michael Sparks (after re-reading Sheldon's earlier reply and rethinking my comments about RA suppression)

                      • 8. Re: Intel NIC cards bleeding tagged vLANs into non-tagged vLANs (IPv6)
                        mark_h_@intel

                        Hi Sheldon,

                        Thank you for updating the thread.

                         

                        Mark H

                        • 9. Re: Intel NIC cards bleeding tagged vLANs into non-tagged vLANs (IPv6)
                          sparks

                          Turns out relying on unicast RAs is insufficient. It works for SLAAC, but without the multicast RA's Windows loses its default route. So it's back to blocking RAs from the specific VoIP VLAN interfaces, configuring the NIC drivers for the untagged VLAN, or waiting for Microsoft to fix the default handling of VLAN tagged packets. Or Sheldon's smartport macro solution, which seems to be the next best solution to Microsoft fixing it.