4 Replies Latest reply on Oct 18, 2017 7:57 AM by Intel Corporation

    Intel® SSD DC P3700 Series I/O 0 QID 0 timeout, reset controller

    p0nts

      We have a few servers with nvme drives that are currently experiencing controller resets during I/O.

      Screenshot 2017-06-29 07.53.19.png

       

      Affected drives:

       

      Node             Model                Version  Namepace Usage                      Format           FW Rev

      ---------------- -------------------- -------- -------- -------------------------- ---------------- --------

      /dev/nvme0n1     INTEL SSDPE2MD800G4  1.0      1        800.17  GB / 800.17  GB    512   B +  0 B   8DV10131

      /dev/nvme1n1     INTEL SSDPE2MD800G4  1.0      1        800.17  GB / 800.17  GB    512   B +  0 B   8DV10131

      /dev/nvme2n1     INTEL SSDPE2MD800G4  1.0      1        800.17  GB / 800.17  GB    512   B +  0 B   8DV10131

      /dev/nvme3n1     INTEL SSDPE2MD800G4  1.0      1        800.17  GB / 800.17  GB    512   B +  0 B   8DV10131

      /dev/nvme4n1     INTEL SSDPE2MD800G4  1.0      1        800.17  GB / 800.17  GB    512   B +  0 B   8DV10131

      /dev/nvme5n1     INTEL SSDPE2MD800G4  1.0      1        800.17  GB / 800.17  GB    512   B +  0 B   8DV10131

      /dev/nvme6n1     INTEL SSDPE2MD800G4  1.0      1        800.17  GB / 800.17  GB    512   B +  0 B   8DV10131

      /dev/nvme7n1     INTEL SSDPE2MD800G4  1.0      1        800.17  GB / 800.17  GB    512   B +  0 B   8DV10131

      /dev/nvme8n1     INTEL SSDPE2MD800G4  1.0      1        800.17  GB / 800.17  GB    512   B +  0 B   8DV10131

      /dev/nvme9n1     INTEL SSDPE2MD800G4  1.0      1        800.17  GB / 800.17  GB    512   B +  0 B   8DV10131

       

      They run on Debian Wheezy 7.11 with a 3.16 kernel.

       

      some logging:

       

      [Tue Sep 5 11:28:48 2017] nvme 0000:05:00.0: I/O 0 QID 0 timeout, reset controller

      [Tue Sep 5 11:28:50 2017] nvme 0000:05:00.0: Device shutdown incomplete; abort shutdown

      [Tue Sep 5 11:28:50 2017] nvme 0000:05:00.0: Cancelling I/O 0 QID 0

      [Tue Sep 5 11:28:50 2017] nvme 0000:05:00.0: Cancelling I/O 1 QID 0

      [Tue Sep 5 11:28:53 2017] nvme 0000:05:00.0: irq 239 for MSI/MSI-X

      [Tue Sep 5 11:28:53 2017] nvme 0000:05:00.0: irq 240 for MSI/MSI-X

      [Tue Sep 5 11:28:53 2017] nvme 0000:05:00.0: irq 241 for MSI/MSI-X

      [Tue Sep 5 11:28:53 2017] nvme 0000:05:00.0: irq 242 for MSI/MSI-X

      [Tue Sep 5 11:28:53 2017] nvme 0000:05:00.0: irq 243 for MSI/MSI-X

      [Tue Sep 5 11:28:53 2017] nvme 0000:05:00.0: irq 244 for MSI/MSI-X

      [Tue Sep 5 11:28:53 2017] nvme 0000:05:00.0: irq 245 for MSI/MSI-X

      [Tue Sep 5 11:28:53 2017] nvme 0000:05:00.0: irq 246 for MSI/MSI-X

      [Tue Sep 5 11:28:53 2017] nvme 0000:05:00.0: irq 247 for MSI/MSI-X

      [Tue Sep 5 11:28:53 2017] nvme 0000:05:00.0: irq 248 for MSI/MSI-X

      [Tue Sep 5 11:28:53 2017] nvme 0000:05:00.0: irq 249 for MSI/MSI-X

      [Tue Sep 5 11:28:53 2017] nvme 0000:05:00.0: irq 250 for MSI/MSI-X

      [Tue Sep 5 11:28:53 2017] nvme 0000:05:00.0: irq 251 for MSI/MSI-X

      [Tue Sep 5 11:28:53 2017] nvme 0000:05:00.0: irq 252 for MSI/MSI-X

      [Tue Sep 5 11:28:53 2017] nvme 0000:05:00.0: irq 253 for MSI/MSI-X

      [Tue Sep 5 11:28:53 2017] nvme 0000:05:00.0: irq 254 for MSI/MSI-X

      [Tue Sep 5 11:28:53 2017] nvme 0000:05:00.0: irq 255 for MSI/MSI-X

      [Tue Sep 5 11:28:53 2017] nvme 0000:05:00.0: irq 256 for MSI/MSI-X

      [Tue Sep 5 11:28:53 2017] nvme 0000:05:00.0: irq 257 for MSI/MSI-X

      [Tue Sep 5 11:28:53 2017] nvme 0000:05:00.0: irq 258 for MSI/MSI-X

      [Sat Sep 9 18:23:21 2017] nvme 0000:03:00.0: I/O 0 QID 0 timeout, reset controller

      [Sat Sep 9 18:23:23 2017] nvme 0000:03:00.0: Device shutdown incomplete; abort shutdown

      [Sat Sep 9 18:23:23 2017] nvme 0000:03:00.0: Cancelling I/O 0 QID 0

      [Sat Sep 9 18:23:25 2017] nvme 0000:03:00.0: irq 219 for MSI/MSI-X

      [Sat Sep 9 18:23:25 2017] nvme 0000:03:00.0: irq 220 for MSI/MSI-X

      [Sat Sep 9 18:23:25 2017] nvme 0000:03:00.0: irq 221 for MSI/MSI-X

      [Sat Sep 9 18:23:25 2017] nvme 0000:03:00.0: irq 222 for MSI/MSI-X

      [Sat Sep 9 18:23:25 2017] nvme 0000:03:00.0: irq 223 for MSI/MSI-X

      [Sat Sep 9 18:23:25 2017] nvme 0000:03:00.0: irq 224 for MSI/MSI-X

      [Sat Sep 9 18:23:25 2017] nvme 0000:03:00.0: irq 225 for MSI/MSI-X

      [Sat Sep 9 18:23:25 2017] nvme 0000:03:00.0: irq 226 for MSI/MSI-X

      [Sat Sep 9 18:23:25 2017] nvme 0000:03:00.0: irq 227 for MSI/MSI-X

      [Sat Sep 9 18:23:25 2017] nvme 0000:03:00.0: irq 228 for MSI/MSI-X

      [Sat Sep 9 18:23:25 2017] nvme 0000:03:00.0: irq 229 for MSI/MSI-X

      [Sat Sep 9 18:23:25 2017] nvme 0000:03:00.0: irq 230 for MSI/MSI-X

      [Sat Sep 9 18:23:25 2017] nvme 0000:03:00.0: irq 231 for MSI/MSI-X

      [Sat Sep 9 18:23:25 2017] nvme 0000:03:00.0: irq 232 for MSI/MSI-X

      [Sat Sep 9 18:23:25 2017] nvme 0000:03:00.0: irq 233 for MSI/MSI-X

      [Sat Sep 9 18:23:25 2017] nvme 0000:03:00.0: irq 234 for MSI/MSI-X

      [Sat Sep 9 18:23:25 2017] nvme 0000:03:00.0: irq 235 for MSI/MSI-X

      [Sat Sep 9 18:23:25 2017] nvme 0000:03:00.0: irq 236 for MSI/MSI-X

      [Sat Sep 9 18:23:25 2017] nvme 0000:03:00.0: irq 237 for MSI/MSI-X

      [Sat Sep 9 18:23:25 2017] nvme 0000:03:00.0: irq 238 for MSI/MSI-X

      [Sun Sep 10 15:07:25 2017] nvme 0000:83:00.0: I/O 0 QID 0 timeout, reset controller

      [Sun Sep 10 15:07:28 2017] nvme 0000:83:00.0: Device shutdown incomplete; abort shutdown

      [Sun Sep 10 15:07:28 2017] nvme 0000:83:00.0: Cancelling I/O 0 QID 0

      [Sun Sep 10 15:07:31 2017] nvme 0000:83:00.0: irq 319 for MSI/MSI-X

      [Sun Sep 10 15:07:31 2017] nvme 0000:83:00.0: irq 320 for MSI/MSI-X

      [Sun Sep 10 15:07:31 2017] nvme 0000:83:00.0: irq 321 for MSI/MSI-X

      [Sun Sep 10 15:07:31 2017] nvme 0000:83:00.0: irq 322 for MSI/MSI-X

      [Sun Sep 10 15:07:31 2017] nvme 0000:83:00.0: irq 323 for MSI/MSI-X

      [Sun Sep 10 15:07:31 2017] nvme 0000:83:00.0: irq 324 for MSI/MSI-X

      [Sun Sep 10 15:07:31 2017] nvme 0000:83:00.0: irq 325 for MSI/MSI-X

      [Sun Sep 10 15:07:31 2017] nvme 0000:83:00.0: irq 326 for MSI/MSI-X

      [Sun Sep 10 15:07:31 2017] nvme 0000:83:00.0: irq 327 for MSI/MSI-X

      [Sun Sep 10 15:07:31 2017] nvme 0000:83:00.0: irq 328 for MSI/MSI-X

      [Sun Sep 10 15:07:31 2017] nvme 0000:83:00.0: irq 329 for MSI/MSI-X

      [Sun Sep 10 15:07:31 2017] nvme 0000:83:00.0: irq 330 for MSI/MSI-X

      [Sun Sep 10 15:07:31 2017] nvme 0000:83:00.0: irq 331 for MSI/MSI-X

      [Sun Sep 10 15:07:31 2017] nvme 0000:83:00.0: irq 332 for MSI/MSI-X

      [Sun Sep 10 15:07:31 2017] nvme 0000:83:00.0: irq 333 for MSI/MSI-X

      [Sun Sep 10 15:07:31 2017] nvme 0000:83:00.0: irq 334 for MSI/MSI-X

      [Sun Sep 10 15:07:31 2017] nvme 0000:83:00.0: irq 335 for MSI/MSI-X

      [Sun Sep 10 15:07:31 2017] nvme 0000:83:00.0: irq 336 for MSI/MSI-X

      [Sun Sep 10 15:07:31 2017] nvme 0000:83:00.0: irq 337 for MSI/MSI-X

      [Sun Sep 10 15:07:31 2017] nvme 0000:83:00.0: irq 338 for MSI/MSI-X

      [Tue Sep 12 15:06:34 2017] nvme 0000:03:00.0: I/O 0 QID 0 timeout, reset controller

      [Tue Sep 12 15:06:36 2017] nvme 0000:03:00.0: Device shutdown incomplete; abort shutdown

      [Tue Sep 12 15:06:36 2017] nvme 0000:03:00.0: Cancelling I/O 0 QID 0

      [Tue Sep 12 15:06:39 2017] nvme 0000:03:00.0: irq 219 for MSI/MSI-X

      [Tue Sep 12 15:06:39 2017] nvme 0000:03:00.0: irq 220 for MSI/MSI-X

      [Tue Sep 12 15:06:39 2017] nvme 0000:03:00.0: irq 221 for MSI/MSI-X

      [Tue Sep 12 15:06:39 2017] nvme 0000:03:00.0: irq 222 for MSI/MSI-X

      [Tue Sep 12 15:06:39 2017] nvme 0000:03:00.0: irq 223 for MSI/MSI-X

      [Tue Sep 12 15:06:39 2017] nvme 0000:03:00.0: irq 224 for MSI/MSI-X

      [Tue Sep 12 15:06:39 2017] nvme 0000:03:00.0: irq 225 for MSI/MSI-X

      [Tue Sep 12 15:06:39 2017] nvme 0000:03:00.0: irq 226 for MSI/MSI-X

      [Tue Sep 12 15:06:39 2017] nvme 0000:03:00.0: irq 227 for MSI/MSI-X

      [Tue Sep 12 15:06:39 2017] nvme 0000:03:00.0: irq 228 for MSI/MSI-X

      [Tue Sep 12 15:06:39 2017] nvme 0000:03:00.0: irq 229 for MSI/MSI-X

      [Tue Sep 12 15:06:39 2017] nvme 0000:03:00.0: irq 230 for MSI/MSI-X

      [Tue Sep 12 15:06:39 2017] nvme 0000:03:00.0: irq 231 for MSI/MSI-X

      [Tue Sep 12 15:06:39 2017] nvme 0000:03:00.0: irq 232 for MSI/MSI-X

      [Tue Sep 12 15:06:39 2017] nvme 0000:03:00.0: irq 233 for MSI/MSI-X

      [Tue Sep 12 15:06:39 2017] nvme 0000:03:00.0: irq 234 for MSI/MSI-X

      [Tue Sep 12 15:06:39 2017] nvme 0000:03:00.0: irq 235 for MSI/MSI-X

      [Tue Sep 12 15:06:39 2017] nvme 0000:03:00.0: irq 236 for MSI/MSI-X

      [Tue Sep 12 15:06:39 2017] nvme 0000:03:00.0: irq 237 for MSI/MSI-X

      [Tue Sep 12 15:06:39 2017] nvme 0000:03:00.0: irq 238 for MSI/MSI-X

       

      Can somebody please advise?