10 Replies Latest reply on Jul 13, 2016 4:24 PM by ch_ciab

    Intel 520 SSD causes system to lock up on format

    ch_ciab

      Hello!

       

      I'm in charge of our company's fleet of managed assets.  A few days ago, a problem surfaced with 240GB 520 series SSDs, utilized in Lenovo W510 laptops.

       

      Summary:  Upon clearing all partitions from the drive, creating a 2GiB and a 222GiB (approximate size), and attempting to format the larger one, after about a minute the entire system locks up, requiring a hard power-off.

       

      Details

      This is a scripted process, however for troubleshooting purposes I have also done it by hand, with the same result.  The steps involved are:

      1. PXE boot into Windows PE.
      2. Remove all partitions (PowerShell Clear-Disk -RemoveData -RemoveOEM).
      3. Initialize the disk (Initialize-Disk -PartitionStyle MBR)
      4. Create a 2GiB partition and a second one using all remaining space (New-Partition ...)
      5. Format both partitions to NTFS (Format-Volume -FileSystem NTFS -AllocationUnitSize 4096)
        1. 2GiB partition formats fine in just a few seconds.
        2. Big partition appears to work at first (HDD access light flickers as usual), but then something happens.  HDD access light begins blinking regularly, and after about a minute the computer locks up hard.

       

      Initially, I hit this with one drive and naturally thought it had failed.  I tried with several other units, however, and swapped the drives between identical computers, also swapped them into different models, with the same result for four different SSDs.  My script has not changed, and I verified that it still works on other hardware (using Crucial SSD in Lenovo W530).

       

      I have run SSD Toolbox on three of these drives.  Quick check shows both life expectancy bars at 100%, SMART shows no errors or warnings, and a long diagnostic turns up no problems.  These drives have been in use for over two years, and most of them have about 4 TiB written according to SSD Toolbox.

       

      I just tested with a SSD that had been through a detailed diagnostic with SSD Toolbox; I created partitions using the Windows Disk Manager, and formatted them just fine.  I put it back into a machine and attempted to format using my script.  Machine locked up.

       

      At this point I am stumped.

       

      Where would we go from here?

        • 1. Re: Intel 520 SSD causes system to lock up on format
          jonathan_intel

          Hello ch_ciab,

           

          Here are some options to consider based on the information you have provided:

           

          - Try doing a Secure Erase (Low level format), this will remove all data on the SSD, then try initializing the drive again. You can find more information about this in the following advisory: When You Run a Low-Level Format on Intel® Solid State Drives for...

           

          - Make sure that the Storage Controller is set to AHCI mode, and that the SSD's are detected correctly.

           

          - Check with the Computer Manufacturer Support and make sure you are using updated BIOS and any required drivers for the PC's.

          If the issue persists at this point, you might need to review the script, since you mentioned that Windows Disk Management is able to use the drive normally.

          • 2. Re: Intel 520 SSD causes system to lock up on format
            ch_ciab

            Hi, Jonathan, and thanks for the prompt reply!

             

            I will try using a low-level format utility because that's certainly something I hadn't considered yet!

             

            Meanwhile, regarding the other things you suggested:  Yes, storage controller is set to AHCI; SSD would appear to be detected correctly, considering that it is available to WMI and DiskPart; system BIOS is, in fact, a bit out of date so I'll fix that -- however the fact remains that the scripted setup I'm using has been working for over a year.

             

            As far as reviewing the script, I have been doing that because I couldn't find any fault with the hardware -- what puzzles me is why it would suddenly stop working, when it had been fine for over six months.  (I started making changes while troubleshooting yesterday; prior to that I had not changed anything since August.)  The same script works fine on other hardware, and these are just standard storage commands we're using here, so it's questionable.

             

            Anyway, I'll report back once I've done a low-level format.  I'm not thrilled about having to do that on (up to) 36 machines, however if it works it's better than replacing the drives.

            • 3. Re: Intel 520 SSD causes system to lock up on format
              ch_ciab

              OK, I've tried the other things you suggested.

               

              Update BIOS:  No change

              Low-level format:  No change

              Updated BIOS + low-level formatted drive:  No change

               

              Further, I did some reading and was thinking the "ATA Secure Erase" might be a good idea, and quicker than "low-level" formatting.  I attempted to use Secure Erase from within SSD Toolbox, but amazingly it told me that operation isn't supported in Win8/2012.  My test machines are Win 8.1 and Server 2012 R2.

               

              Is it possible we have a bad batch of drives here?

              • 4. Re: Intel 520 SSD causes system to lock up on format
                jonathan_intel

                As you mentioned, running the ATA Secure Erase from Intel® SSD Toolbox is not possible with current Windows® releases, since they block ATA Security commands once the drive is detected by the OS. Secure Erase may still be possible using 3rd party tools, or Linux OS.

                 

                We are not aware of this happening with retail versions of the Intel® SSD 520 series, however, I found some Lenovo forums and advisories about issues with the Lenovo OEM version of this drive.

                 

                If the drives are OEM versions from Lenovo, please check with the Computer Manufacturer Support, and make sure you are using the most recent Firmware version for the SSD's.

                • 5. Re: Intel 520 SSD causes system to lock up on format
                  ch_ciab

                  Ah, I had not considered potential differences between OEM and retail versions of the same product.

                   

                  From what I can see, the drives do have the latest firmware -- at least, the ones I have checked do.  SSD Toolbox reported no updates available, and I can get the revision easily using this:

                   

                  (Get-PhysicalDisk | where MediaType -eq "SSD").FirmwareVersion

                   

                  LF1i

                   

                  Do I understand correctly, that the "L" there denotes a Lenovo firmware?

                   

                  Also, because you said you've found Lenovo discussions and advisories regarding this drive, I have been searching for them.  I found a couple of forum threads (such as this, and this) but those concern the drive locking up under heavy load -- which is certainly not the same problem I'm encountering.  (Indeed, the SSDs have all performed very well in that regard; their usage here has them basically setting up a new OS every week, and often running virtual machines on top of that.)  Would you be able to link me to any of the discussions you found?

                   

                  I ask because if this is a known issue with Lenovo, that would help me make a case for getting them to replace these drives.

                  • 6. Re: Intel 520 SSD causes system to lock up on format
                    jonathan_intel

                    The firmware version in fact indicates that these are Lenovo OEM drives, Intel® SSD Toolbox can't update the firmware on OEM drives, so it will show that no updates are available. It is worth to mention that the retail version of the Intel® SSD 520 series did not get any firmware updates.

                     

                    The forums you found are the ones I saw before. If your computers still have support from Lenovo, you might be able to check with them on this issue as well.

                     

                    The fact that you are able to initialize the drive normally from disk management suggests there may be a problem with the variables in the script, or a possible firmware issue.

                     

                    - Have you tested if you have issues to partition and format the SSD's using other methods?
                    - Have you had success with any drive of this type?

                    • 7. Re: Intel 520 SSD causes system to lock up on format
                      ch_ciab

                      jonathan_intel wrote:

                       

                      The fact that you are able to initialize the drive normally from disk management suggests there may be a problem with the variables in the script, or a possible firmware issue.

                      I agree.  That's a reasonable idea, and (as I believe I mentioned earlier) I did check the script.  I had not made any changes to it between the time when it worked properly (three months ago) and the time when it failed (five days ago).  You mention that there have been no firmware updates for this device.  So if...

                      1. Script has not changed.
                      2. Firmware has not changed.
                      3. BIOS has not changed.

                      ...a reasonable conclusion would be that something about the SSD itself has changed.  This could include obvious statistics such as hours powered on, bytes of data written, number of power cycles, etc.  These would naturally increase with usage over time.  Non-obvious things... well, I guess I haven't found them because they aren't obvious. 

                       

                      - Have you tested if you have issues to partition and format the SSD's using other methods?

                      - Have you had success with any drive of this type?

                      Yes, I've tried a couple of other methods:

                      • Scripted using PowerShell in Windows PE 5.0:  Device locked up
                      • By hand using PowerShell in Windows PE 5.0:  Device locked up
                      • By hand using DiskPart in Windows PE 5.0:  Device locked up
                      • By hand using PowerShell in Server 2012 R2:  Worked normally
                      • By hand using Disk Management in Server 2012 R2:  Worked normally

                       

                      As for "success with any drive of this type," I'm not sure I understand the question.  These very same drives worked a few months ago with the same procedures.  The same script also works with HDD-based machines, and also with a set using Crucial SSDs. 

                       

                      We use the same process to set up dozens of computers each week, and hundreds every month; all of the laptops in our fleet are refreshed quarterly, so I've done this a lot.  Because of this, I have an interest in finding out why this is happening, so that I can determine whether it is likely to happen to any of our other hardware.

                      • 8. Re: Intel 520 SSD causes system to lock up on format
                        paramountain

                        Excuse me for butting in, but I may have a solution. Somewhere on intel.com (sorry, I did not retain the link) is a reference to DBAN. The Intel article suggested that if an SSD does not respond to the usual treatment, running a single-pass zero-write (the top option in DBAN's list) will completely wipe the drive and return it to a usable state. DBAN's website specifically states that it is not designed for SSDs, but I tried it. It allowed a drive which had screwy errors to one which was usable again. I have done this on Crucial and Intel SSDs, but it's only a last resort.

                        • 9. Re: Intel 520 SSD causes system to lock up on format
                          ch_ciab

                          paramountain wrote:

                           

                          Excuse me for butting in, but I may have a solution. Somewhere on intel.com (sorry, I did not retain the link) is a reference to DBAN. The Intel article suggested that if an SSD does not respond to the usual treatment, running a single-pass zero-write (the top option in DBAN's list) will completely wipe the drive and return it to a usable state. DBAN's website specifically states that it is not designed for SSDs, but I tried it. It allowed a drive which had screwy errors to one which was usable again. I have done this on Crucial and Intel SSDs, but it's only a last resort.

                          That's fine, thanks for the suggestion!  Actually the link above (in Jonathan's first reply to me) has a page with links to DBAN, KillDisk, and another program.  I have already done the zero-write procedure using KillDisk and it made no difference.

                           

                          I've also used DBAN before for wiping drives of work and personal machines, but in this case I chose KillDisk because I didn't want to make a boot disk.

                          • 10. Re: Intel 520 SSD causes system to lock up on format
                            ch_ciab

                            After a long time, I have been able to resolve this issue.  A recent test using the latest version of Windows PE (the one that ships with Windows 10) shows that it is capable of formatting the SSD without freezing the system.  I do not know what Microsoft changed between Windows PE 5.0 and 10.0.10586, but it seems they have fixed the storage subsystem so that it can now do this correctly.

                             

                            Therefore, the solution to this issue seems to be:  Don't use WinPE 5.0; instead use Windows 8.1 or higher, or WinPE 10.x.