14 Replies Latest reply on Jun 22, 2016 9:41 AM by Intel Corporation

    Firmware Downgrade

    andy5508

      Does anyone know if it is possible to downgrade the firmware on the P3700?  We are struggling with a VMware VSAN implementation, and support has told us that the drives need to be running firmware 8DV10131, and the current firmware 8DV10171 is causing problems.  Our vendor has attempted to get an answer directly from Intel with no luck.

       

      Thanks,

      Andy

        • 1. Re: Firmware Downgrade
          nestor_intel

          andy5508,

           

          We are currently working on that case, we would be providing updates in the next days. Please expect an answer anytime soon from here as well.

          • 2. Re: Firmware Downgrade
            Emulsifide

            nestor_intel,

             

            Good afternoon. Has there been any update on this? I'm in the same situation where VSAN has certified 8DV10131, but my P3700s shipped with 8DV10171. Is Intel in the process of VSAN re-certification with 8DV10171 or do you now have a solution for downgrading the firmware? Help ASAP is greatly appreciated!

            • 3. Re: Firmware Downgrade
              nestor_intel

              Emulsifide,

               

              Our team is still working with VMware* in order to validate the firmwares we have, at the moment the situation is escalated with them. We recommend to open a case with them as well so you can keep a track of VMware* updates as well.

              • 4. Re: Firmware Downgrade
                Emulsifide

                nestor_intel,

                 

                Thanks for the quick response. Aside from certifying the latest firmware, does Intel have a method of downgrading the firmware in the meantime so we can be in compliance with VMWare VSAN?

                • 5. Re: Firmware Downgrade
                  elerium

                  I have a mix of VSAN 6.1/6.2 hybrid and all flash clusters, all using P3700 or P3600 for cache. I can tell you that in VSAN 6.2, using the HCL firmware/driver combo you would see very poor performance, congestion and latency problems. I recently built a new 6.2 VSAN all flash cluster that happened to ship with 8DV10131 (HCL) firmware. Using 1.0e.1.1-1OEM.550.0.0.1391871 HCL driver, there are severe write performance related issues. Result is the same after upgrading to firmware 8DV10171. Between VMWare and Intel or whoever is responsible for updating the HCL, I don't think any real testing went into it before it got 6.2 qualified. Testing the HCL combo even for 5 minutes, one would immediately notice a major latency/congestion issue, on even light stress testing. I believe it also has something to do with the new checksum functionality added in 6.2, if disabled in the storage profile, all performance returns back to normal levels.

                   

                  Personally I don't think it's an issue of downgrading firmware but for Intel to release a new inte-nvme driver (and or firmware update) that resolves issues discovered for version for 6.2.  Also none of the issues exist on 6.1 (probably because checksum feature isn't on 6.1).

                   

                  Here are my findings from 6.2 AF VSAN using Intel P3700 400GB for write cache and 4x Intel S3510 800GB for capacity:

                  P3700 400GB, firmware 8DV10131, intel-nvme 1.0e.1.1-1OEM.550.0.0.1391871 driver - severe latency/congestion issues from disk writes, no issues if disabling checksum

                  P3700 400GB, firmware 8DV10131, intel-nvme 1.0e.2.0-1OEM.550.0.0.1391871 driver - severe latency/congestion issues from disk writes, no issues if disabling checksum

                  P3700 400GB, firmware 8DV10171, intel-nvme 1.0e.1.1-1OEM.550.0.0.1391871 driver - severe latency/congestion issues from disk writes, no issues if disabling checksum

                  P3700 400GB, firmware 8DV10171, intel-nvme 1.0e.2.0-1OEM.550.0.0.1391871 driver - severe latency/congestion issues from disk writes, no issues if disabling checksum

                  P3700 400GB, firmware 8DV10171, nvme 1.2.0.27-4vmw.550.0.0.1331820 driver - no latency/congestion problems, sequential writes limited to 250MB/s, no issues if disabling checksum

                  1 of 1 people found this helpful
                  • 6. Re: Firmware Downgrade
                    Intel Corporation
                    This message was posted on behalf of Intel Corporation

                    elerium,

                    Thanks for this information. We will make sure to verify with our team and send this findings to the proper department.
                    As soon as we have any news or updates we will be posting it here.

                    1 of 1 people found this helpful
                    • 7. Re: Firmware Downgrade
                      Intel Corporation
                      This message was posted on behalf of Intel Corporation

                      Elerium & Emulsifide,

                      We have found some other users experiencing a similar issue reported in this thread by elerium, we are checking from our side, but it has been the first time we have made aware of it. We will share any findings as soon as we have any updates.

                      NC

                      • 8. Re: Firmware Downgrade
                        Intel Corporation
                        This message was posted on behalf of Intel Corporation

                        elerium,

                         

                        We would like to inform you that one of the big advantages of our Datacenter drives is the high lasting data integrity.

                         

                        Enabling the software Checksum feature available in VSAN 6.2 will add additional latency which results in an overall performance reduction, therefore due to our drive's high reliability towards preventing data corruption, it may not be necessary to enable this feature in VSAN unless the application requires so.

                        NC

                        • 9. Re: Firmware Downgrade
                          Intel Corporation
                          This message was posted on behalf of Intel Corporation

                          elerium,

                          We haven't heard from you in a while. We would like to know if you still need our assistance.

                          • 10. Re: Firmware Downgrade
                            Emulsifide

                            I disagree with your justification for not enabling checksum in VSAN 6.2. In a hybrid VSAN, the SSDs are not the only disks containing data that needs to be protected. Stating that checksum is not necessary does not solve the problem.

                             

                            Assistance is still needed. Please communicate directly with VMWare to straighten out the HCL certification issues as well as the latency issues.

                            • 11. Re: Firmware Downgrade
                              Intel Corporation
                              This message was posted on behalf of Intel Corporation

                              Emulsifide,

                              This case has been escalated with VMware* already and we will also share any more inputs as soon as our people in charge have some more news.
                              We will keep you all posted.

                              NC

                              • 12. Re: Firmware Downgrade
                                VSAN

                                Guys

                                 

                                I have also the same problem , I feel in risk know to put  the  our VSAN project to move in production because of the firmware  and driver  shows waring in HCL !

                                 

                                I have Intel 3600 DC PCIE on  for Vsan 6.2  .

                                I hope Intel could give us the right answer

                                 

                                Ash

                                • 13. Re: Firmware Downgrade
                                  Intel Corporation
                                  This message was posted on behalf of Intel Corporation

                                  Emulsifide and VSAN,
                                   

                                  We have already worked with VMware* to have our FW171 added into the HCL and the expectation is for VMWare* to have it updated by next week, you can keep an eye on their website. 

                                   

                                  On the other hand, the FW added into the HCL has no relationship with the SW Checksum added by VMWare* (as an option) to their VSAN 6.2; therefore, the latest FW will not fix the latency issues associated with the SW Checksum when enabled, as this is not really related to our drives, since our drives where designed and tested for high integrity and therefore non-validated or intended to operate with VSAN's SW Checksum feature.

                                   

                                  At the end it is up to everyone whether to rely on our high integrity SSD's or enable a SW Checksum which will add latency and therefore sacrifice performance.

                                  Let us know if you need more information.

                                  NC
                                  • 14. Re: Firmware Downgrade
                                    Intel Corporation
                                    This message was posted on behalf of Intel Corporation

                                    Hello all,

                                    VMware* has confirmed in the website that the latest firmware is now validated in HCL, please check the following links:

                                    P3600 & P3700

                                    Please let us know if you require any further assistance.

                                    NC