1 Reply Latest reply on Jun 12, 2017 5:52 PM by Intel Corporation

    Joule 550x + USB 3 hub + USB 3 camera causes Linux kernel crash or panic


      I've submitted product support request # 02861224 with detailed information on the problem. But here is an updated description of the problem:


      The Linux 4.4.0 kernel crashes (total Linux kernel lockup, has displayed a long kernel panic a few times) within seconds when transferring image data from a FLIR USB3 machine vision camera ( https://www.ptgrey.com/blackfly-s-50-mp-mono-usb3-vision-sony-imx250  ) connected via a USB 3 hub to the Joule 550x Dev Kit Expansion Board USB 3 Type A host port. No relevant messages are logged to /var/log/syslog before the kernel crash.


      At a software triggered / software managed acquisition rate of ~14 frames/second (at the native 5 megapixel resolution) a crash occurs in seconds. At a slower acquisition rate of ~0.5 frame/second a crash will occur in approximately 30 seconds to 5 minutes of image acquisition program run time.


      The OS running on the Joule is Ubuntu Server 16.04.02 x64 (very stock install, no issues) and is installed to the Joule's internal eMMC. Ubuntu x64 was chosen partly because its a requirement of FLIR's Spinnaker SDK. The Joule is being powered by its USB type C OTG port, using a Type C PD compliant source and cable. A USB type C voltage and current meter confirms Type C PD negotiation to 9 volts. The Joule displays no other stability problems (either idle or under heavy CPU load) and passed a 20+ minute memTest86 endurance test. Joule BIOS version was updated to version 1F1.


      Most notable is that a crash can be dramatically delayed when the USB 3 camera is connected directly to the Joule 550x with no USB 3 hub connected. This configuration performed one overnight endurance test, acquiring and displaying over 500K images at a software-triggered rate of 14 frames/second at the camera's native 5 megapixel resolution until it resulted in a kernel panic, at approximately 10 hours endurance. I've run several 1 to 2 hour USB3-hub-free endurance tests and have found the Joule and USB 3 camera can communicate quite reliability as long as there is no hub between them.


      Three different USB 3 hubs have been tested, two of them powered, decently high-end USB 3 hubs (TI and Genesis Logic chipsets). The TI chipset hub is sold by FLIR and is their most recommended USB 3 hub for use with their cameras in industrial applications. https://www.ptgrey.com/4-port-usb-30-hub-with-power-supply-3  FLIR also confirmed that our Genesis Logic chipset hub is on their recommended list. I've also tested with a Realtek chipset unpowered hub. All three hubs generate the crash when connected between the USB 3 camera and the Joule.


      A "control" configuration consisting of the same camera, cables, any of the three USB 3 hubs, same application software, same FLIR SDK, same Ubuntu server OS, same Linux kernel but with a different x86 host (Supermicro X10DRi server motherboard instead of the Joule 550x) does not crash the Linux kernel. I have not found any configuration using the Supermicro x86 host (instead of the Joule) that produces a Linux kernel crash or panic. All three hubs mentioned above perform equally well with the USB 3 camera and Supermicro host.


      FLIR has also performed their own related-configuration tests (though not with a Joule host) and have not been able to replicate any Linux kernel crashes. And FLIR has offered to help test a Joule in their labs.


      I have a Joule 570x Dev Kit on order and will compare ASAP.


      What can I do to help diagnose this problem?