14 Replies Latest reply on Dec 17, 2015 11:17 PM by Surendar.Chandra

    Intel 750 not showing delete support under ESXi?

    shopkins

      Hi everyone, I've hit an unexpected issue with some new Intel 750's (1.2TB) and ESXi 6 (latest update). I have installed the latest driver as recommended (to improve the performance of the drives) and while they are working I have noticed that ESXi isn't attempting to reclaim any free space (and forcing it doesn't work). A check of the VAAI values of each drive showed the following:

       

      VAAI Plugin Name:

      ATS Status: unsupported

      Clone Status: unsupported

      Zero Status: unsupported

      Delete Status: unsupported

       

      While I expected ATS/Clone/Zero to be unsupported, I didn't expect to see Delete as unsupported. Using a standard SSD through an AHCI controller will show Delete as supported (as TRIM seems to get converted to DISCARD/UNMAP).

       

      Has anyone else seen this? I'm concerned that as a result my performance is going to degrade over time as there will be a lack of proper unmapping of free space.

        • 1. Re: Intel 750 not showing delete support under ESXi?
          jonathan_intel

          Hello shopkins,

           

          We will need some additional information to check about the "Delete Status: Unsupported". Please let us know the following:

           

          - Computer brand and model
          - BIOS version
          - SSD model number
          - SSD firmware version
          - Driver version used

           

          - Are you using an Add in card, or a 2.5 inch form factor SSD?
          - What type of PCIe slot are you using? (PCIe generation and number of lanes)

          • 2. Re: Intel 750 not showing delete support under ESXi?
            shopkins

            Hi Jonathon,

                 Additional information below (I've put quite a bit). I should add that all four of the drives are working and are currently being used as datastores. The concern is that the UNMAP support doesn't appear to be recognised and that as a result there will be a performance degredation. Looking at the nvme driver on SourceForge it does appear to support UNMAP, hence my confusion around why shows Delete as unsupported.

             

            - Supermicro X9DRW-7TPF+

            - BIOS 3.2 (latest available)

            - Intel 750 1.2TB AIC (PCI Gen-3 x4, not the 2.5" form-factor version)

            - Unknown firmware version (would be what was shipped with the card, didnt see a firmware upgrade available a few days ago)

            - VMW-ESX-5.5.0-intel-nvme-1.0e.1.1-2791094.zip (exact driver from ESXi below)

            - PCIe slots are x8 but have been clocked down to x4 (they work in either mode)

             

            > Module info

            [root@esxi:~] vmkload_mod -s intel-nvme

            vmkload_mod module information

            input file: /usr/lib/vmware/vmkmod/intel-nvme

            Version: 1.0e.1.1-1OEM.550.0.0.1391871

            License: BSD

            Required name-spaces:

              com.vmware.vmkapi#v2_2_0_0

            Parameters:

             

            > Storage Path List (for one device, I have four)

            [root@esxi:~] esxcli storage core path list

            pscsi.vmhba4-pscsi.0:0-naa.65fdfe40353136333030305731xxxxx

               UID: pscsi.vmhba4-pscsi.0:0-naa.65fdfe40353136333030305731xxxxx

               Runtime Name: vmhba4:C0:T0:L0

               Device: naa.65fdfe40353136333030305731xxxxx

               Device Display Name: Local NVMe Disk (naa.65fdfe40353136333030305731xxxxx)

               Adapter: vmhba4

               Channel: 0

               Target: 0

               LUN: 0

               Plugin: NMP

               State: active

               Transport: parallel

               Adapter Identifier: pscsi.vmhba4

               Target Identifier: pscsi.0:0

               Adapter Transport Details: Unavailable or path is unclaimed

               Target Transport Details: Unavailable or path is unclaimed

               Maximum IO Size: 131072

             

            > Device VAAI Status

            [root@esxi:~] esxcli storage core device vaai status get -d naa.65fdfe40353136333030305731xxxxxx

            naa.65fdfe40353136333030305731xxxxxx

               VAAI Plugin Name:

               ATS Status: unsupported

               Clone Status: unsupported

               Zero Status: unsupported

               Delete Status: unsupported

             

            > Storage Adapter List

            [root@esxi:~] esxcli storage core adapter list

            HBA Name  Driver        Link State  UID             Capabilities  Description

            --------  ------------  ----------  --------------  ------------  ------------------------------------------------------------------------------

            vmhba0    megaraid_sas  link-n/a    unknown.vmhba0                (0000:0b:00.0) LSI / Symbios Logic MegaRAID SAS Fusion Controller

            vmhba2    rste          link-n/a    pscsi.vmhba2                  (0000:02:00.0) Intel Corporation Patsburg 4-Port SATA/SAS Storage Control Unit

            vmhba3    intel-nvme    link-n/a    pscsi.vmhba3                  (0000:81:00.0) Intel Corporation NVM-Express PCIe SSD

            vmhba4    intel-nvme    link-n/a    pscsi.vmhba4                  (0000:03:00.0) Intel Corporation NVM-Express PCIe SSD

            vmhba5    intel-nvme    link-n/a    pscsi.vmhba5                  (0000:04:00.0) Intel Corporation NVM-Express PCIe SSD

            vmhba6    intel-nvme    link-n/a    pscsi.vmhba6                  (0000:82:00.0) Intel Corporation NVM-Express PCIe SSD

            vmhba32   usb-storage   link-n/a    usb.vmhba32                   () USB

            • 3. Re: Intel 750 not showing delete support under ESXi?
              shopkins

              On the subject of firmware, can someone comment on what the approach is for upgrading the firmware on systems that aren't running Windows... According to Intel® High Performance Solid-State Drive — Firmware Versions for Intel® Solid State Drives the firmware update tool doesn't support the 750's, which makes things a little tricky.

              • 4. Re: Intel 750 not showing delete support under ESXi?
                shopkins

                Quick update, firmware has been updated on all 4 drives however ESXi still shows the Delete capability as unsupported.

                 

                Before

                [root@localhost ~]# isdct show -intelssd

                - IntelSSD CVCQ5163000Wxxxxxx -

                DeviceStatus: Healthy

                Firmware: 8EV10135

                FirmwareUpdateAvailable: Firmware=8EV10171 Bootloader=8B1B0131

                ModelNumber: INTEL SSDPEDMW012T4

                ProductFamily: Intel SSD 750 Series

                SerialNumber: CVCQ5163000Wxxxxxx

                Index: 0

                DevicePath: /dev/nvme0n1

                Bootloader: 8B1B012E

                 

                After

                [root@localhost ~]# isdct show -intelssd

                - IntelSSD CVCQ5163000Wxxxxxx -

                DeviceStatus: Healthy

                Firmware: 8EV10171

                FirmwareUpdateAvailable: The selected Intel SSD contains current firmware as of this tool release.

                ModelNumber: INTEL SSDPEDMW012T4

                ProductFamily: Intel SSD 750 Series

                SerialNumber: CVCQ5163000Wxxxxxx

                Index: 0

                DevicePath: /dev/nvme0n1

                Bootloader: 8B1B0131

                • 5. Re: Intel 750 not showing delete support under ESXi?
                  jonathan_intel

                  Hello shopkins,

                   

                  Thank you for all the information you have provided. After reviewing it, the main recommendation I have at this time is to use the Intel-NVMe* driver 1.0e.2.0, this can be downloaded from VMware website at: Download VMware Virtual SAN


                  Please let us know if this helps, or if the issue persists so we can consider further options.

                  • 6. Re: Intel 750 not showing delete support under ESXi?
                    shopkins

                    Jonathon,

                         Driver has been upgraded and the system has been rebooted. Sadly the VAAI support still shows that Delete isnt supported:

                     

                    [root@esxi:~] vmkload_mod -s intel-nvme

                    vmkload_mod module information

                    input file: /usr/lib/vmware/vmkmod/intel-nvme

                    Version: 1.0e.2.0-1OEM.550.0.0.1391871

                    License: BSD

                    Required name-spaces:

                      com.vmware.vmkapi#v2_2_0_0

                    Parameters:

                     

                    [root@esxi:~] esxcli storage core device vaai status get

                    t10.NVMe____INTEL_SSDPEDMW012T4_CVCQ5163000Wxxxxxx__00000001

                       VAAI Plugin Name:

                       ATS Status: unsupported

                       Clone Status: unsupported

                       Zero Status: unsupported

                       Delete Status: unsupported

                     

                    As a comparison, a Samsung EVO 840 250GB shows the following:

                     

                    [root@testesxi:~] esxcli storage core device vaai status get

                    t10.ATA_____Samsung_SSD_840_EVO_250GB________________S1DDNEADxxxxxxF______

                        VAAI Plugin Name:

                        ATS Status: unsupported

                        Clone Status: unsupported

                        Zero Status: supported

                        Delete Status: supported

                     

                    With the above SSD, both Zero and Delete are supported which helps the long-term performance of the drive.

                    • 7. Re: Intel 750 not showing delete support under ESXi?
                      shopkins

                      I've found the following page (VMware/vSphere/NVMe - Omnia) which shows something interesting in comparison to my results... Their output of esxcfg-scsidevs -l shows that the VAAI status is unknown, which in ESXi terms means its not got full VAAI compatibility but does have at least one feature. The difference in their results (aside from using a P3600) appears to be that their driver is 'nvme' rather than 'intel-nvme', so I am assuming they are using the OOTB ESXi version.

                       

                      Can someone from Intel confirm if the external intel-nvme driver should have the VAAI capability, or if the only way to get it is to revert back to the OOTB driver and suffer the performance drop.

                      • 8. Re: Intel 750 not showing delete support under ESXi?
                        jonathan_intel

                        Hello shopkins,

                         

                        We will engage additional resources to review this issue, please allow us some time to perform additional research and we will provide any new information as soon as it is available.

                        • 9. Re: Intel 750 not showing delete support under ESXi?
                          aleki_intel

                          Hello shopkins,

                           

                          Thank you for your time. Our engineering department tested with both Intel® SSD 750 Series and Intel® SSD DC P3600 Series and the outcome was the following:

                          For the Intel® SSD 750 Series, it is not on the VMWare* hadware compatibility list and therefore, it will have less endurance that the Intel® SSD DC P3600.

                          For the Intel® SSD DC P3600, delete status is supported under ESXi 6, but not on 5.5 so our recommendation is to update to ESXi 6.

                          Overprovioning may reduce the need for TRIM.

                          • 10. Re: Intel 750 not showing delete support under ESXi?
                            shopkins

                            Hi aleki, I appreciate that the 750 series are not on the VMware HCL, that said, they both use the same NVMe driver and both device models do advertise TRIM capability. From an ESXi perspective I am running ESXi 6 (not 5.5). The 5.5 is from the driver version from your website (as the 5.5 driver is used for 6.0 as well).

                             

                            Is TRIM capability on the 750-series being deliberately disabled by your driver? (when running under ESXi 6)

                            • 11. Re: Intel 750 not showing delete support under ESXi?
                              jonathan_intel

                              Hello shopkins,

                               

                              Please take into consideration that the Intel® SSD 750 series is a high performance consumer drive, recommended for use in client and workstations storage. The 750 is not certified for use with ESXi and it is not listed in the VMware Hardware Compatibility List.

                               

                              The Intel® SSD 750 Series have many things in common with the Intel® SSD Data Center Family drives, so the same driver may recognize it and make it operational in your system; however, it does not have the same hardware and capabilities that the Data Center drives have. Since this is not a recommended configuration, we cannot guarantee that all features will be available for it in ESXi, hence, TRIM is not supported for the Intel® SSD 750 Series in VMware ESXi.

                              • 12. Re: Intel 750 not showing delete support under ESXi?
                                shopkins

                                Jonathon,

                                I appreciate that the 750 isn't on the VMware HCL, however that doesn't really answer the question as to if it is VMware (specifically ESXi) that is disabling TRIM support for the 750-series or the updated driver provided by Intel. Many consumer SSD's provide TRIM capability under ESXi (I've tested a few myself) and running the 750 under Linux also shows TRIM capability, so I'd like to understand where support for TRIM is being disabled in this instance.

                                • 13. Re: Intel 750 not showing delete support under ESXi?
                                  jonathan_intel

                                  Hello shopkins,

                                   

                                  I understand that this can be confusing. The Intel NVMe driver for VMware ESXi* 5.5 does not provide TRIM Support with the Intel® SSD 750 Series, and We understand that the native NVMe in VMware ESXi* 6.0 provides TRIM support for other NVMe drives.

                                   

                                  Please keep in mind that the Intel® SSD 750 series were not designed to be used in Data Center environments, so we cannot guarantee full support and compatibility with ESXi*.

                                  • 14. Re: Intel 750 not showing delete support under ESXi?
                                    Surendar.Chandra

                                    Related question, did you have to do something special to get 750 to work with ESXi? Mine does not even show up in 5.5 U3? Thanks