11 Replies Latest reply on Sep 28, 2017 7:16 AM by WalkingThrough

    Joule 550x + USB 3 hub + USB 3 camera causes Linux kernel crash or panic

    ahs1000

      I've submitted product support request # 02861224 with detailed information on the problem. But here is an updated description of the problem:

       

      The Linux 4.4.0 kernel crashes (total Linux kernel lockup, has displayed a long kernel panic a few times) within seconds when transferring image data from a FLIR USB3 machine vision camera ( https://www.ptgrey.com/blackfly-s-50-mp-mono-usb3-vision-sony-imx250  ) connected via a USB 3 hub to the Joule 550x Dev Kit Expansion Board USB 3 Type A host port. No relevant messages are logged to /var/log/syslog before the kernel crash.

       

      At a software triggered / software managed acquisition rate of ~14 frames/second (at the native 5 megapixel resolution) a crash occurs in seconds. At a slower acquisition rate of ~0.5 frame/second a crash will occur in approximately 30 seconds to 5 minutes of image acquisition program run time.

       

      The OS running on the Joule is Ubuntu Server 16.04.02 x64 (very stock install, no issues) and is installed to the Joule's internal eMMC. Ubuntu x64 was chosen partly because its a requirement of FLIR's Spinnaker SDK. The Joule is being powered by its USB type C OTG port, using a Type C PD compliant source and cable. A USB type C voltage and current meter confirms Type C PD negotiation to 9 volts. The Joule displays no other stability problems (either idle or under heavy CPU load) and passed a 20+ minute memTest86 endurance test. Joule BIOS version was updated to version 1F1.

       

      Most notable is that a crash can be dramatically delayed when the USB 3 camera is connected directly to the Joule 550x with no USB 3 hub connected. This configuration performed one overnight endurance test, acquiring and displaying over 500K images at a software-triggered rate of 14 frames/second at the camera's native 5 megapixel resolution until it resulted in a kernel panic, at approximately 10 hours endurance. I've run several 1 to 2 hour USB3-hub-free endurance tests and have found the Joule and USB 3 camera can communicate quite reliability as long as there is no hub between them.

       

      Three different USB 3 hubs have been tested, two of them powered, decently high-end USB 3 hubs (TI and Genesis Logic chipsets). The TI chipset hub is sold by FLIR and is their most recommended USB 3 hub for use with their cameras in industrial applications. https://www.ptgrey.com/4-port-usb-30-hub-with-power-supply-3  FLIR also confirmed that our Genesis Logic chipset hub is on their recommended list. I've also tested with a Realtek chipset unpowered hub. All three hubs generate the crash when connected between the USB 3 camera and the Joule.

       

      A "control" configuration consisting of the same camera, cables, any of the three USB 3 hubs, same application software, same FLIR SDK, same Ubuntu server OS, same Linux kernel but with a different x86 host (Supermicro X10DRi server motherboard instead of the Joule 550x) does not crash the Linux kernel. I have not found any configuration using the Supermicro x86 host (instead of the Joule) that produces a Linux kernel crash or panic. All three hubs mentioned above perform equally well with the USB 3 camera and Supermicro host.

       

      FLIR has also performed their own related-configuration tests (though not with a Joule host) and have not been able to replicate any Linux kernel crashes. And FLIR has offered to help test a Joule in their labs.

       

      I have a Joule 570x Dev Kit on order and will compare ASAP.

       

      What can I do to help diagnose this problem?

        • 1. Re: Joule 550x + USB 3 hub + USB 3 camera causes Linux kernel crash or panic
          Intel Corporation
          This message was posted on behalf of Intel Corporation

          Hi Ahs,
           
          Thanks for contacting us.
           
          I have looked at ticket #02861224 and noticed that Pablo is helping you with this issue. I’d suggest you to keep an eye on your email since Pablo will continue assisting you through that channel.
           
          Regards,
          -Yermi A.
           

          • 2. Re: Joule 550x + USB 3 hub + USB 3 camera causes Linux kernel crash or panic
            WalkingThrough

            Hello -

             

            I just saw this post today.  Wish I'd seen it in June.

             

            I've a similar situation brewing... with a 570x, also with the 1F1 BIOS, and a USB3 camera.  By "similar"... in this case it doesn't matter if there is a hub or not between the camera and Joule.  There is no kernel "panic".  It's worse: the Joule shuts itself off.

             

            By "off" I mean the +5, +3.3, and +1.8 supplies turn off, the display goes blank, and there is nothing logged to /var/log/syslog.  I've also connected a laptop to the serial port (/dev/ttyS2) and nothing is dumped in that direction.  I've run dmesg -w while logged in on that serial port and nothing with that, either.  Regaining control means unplug/replug of the power supply; the white button "usually" has no effect.

             

            I did see the post about disabling the BIOS power management option and have disabled it.  No change in behavior.  Things start up... run for a random amount of time, then >poof< dead.

             

            No camera?  No crash.

             

            uname -a reports:

             

                 Linux joulex2 4.4.0-1000-joule #0+joule21-Ubuntu SMP PREEMPT Thu Mar 16 14:46:45 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

             

            Help would be appreciated.  I'm all eyes & ears on this one.

             

            Mark

            • 3. Re: Joule 550x + USB 3 hub + USB 3 camera causes Linux kernel crash or panic
              Intel Corporation
              This message was posted on behalf of Intel Corporation

              Hi Mark,
               
              I would like to know a little bit more information regarding your issue. Could you please provide the following information?

              • How are you powering the Joule board? Are you using a 12VDC supply in the DC input jack?
              • Could you please give more details regarding how you are connecting your camera and which camera it is?
              • Is this issue only occurring when you use that specific camera?
               
              Additionally, I would like to let you know that the official Joule’s image we support is the Ref-OS-IoT, for that reason I’d suggest you to try installing that one to do your tests.
               
              Regards,
              -Yermi A.
              • 4. Re: Joule 550x + USB 3 hub + USB 3 camera causes Linux kernel crash or panic
                WalkingThrough

                Hello Yermi... and thank you for responding.

                 

                1. The Joule is powered by a 12V lithium-ion battery, a Milwaukee M12 3 Amp-hour device, 16 AWG wires.  Everything seems to run well with this setup.. I've even flashed the BIOS with it.  Yermi, I do have a 12V @ 3A wall supply that was used before the battery... and I've noticed that the output is slightly high at 12.24V.  Is this significant?

                2. I started to write complete information about the camera... and the hardware engineer stopped in, looked, and said, "Don't do that."  Apologies... via email?  It's a USB 3.0 device.

                3. I have a couple of these cameras... all the same model.

                 

                You wrote:

                >

                > [ Ref-OS-IoT ]

                >

                 

                There is a move underway to do just that.  However, development had started "on the platform"... and I haven't started investigating if IDS' driver will run in the Ref-OS-IoT environment.

                 

                The camera is plugged into the USB 3.0 Type A socket on the Joule Expansion Board (not the BIOS flash port, the wide socket).  Software "sees" the camera on power-up and the driver,  The vendor driver installs without complaint; I've found a reboot is necessary.  After that... plug it in, run vendor example code, see images, and that's good.  It is not a "v4l" interface camera and does not show up in /dev/video* .  The vendor ships their own driver.

                 

                Mark

                • 5. Re: Joule 550x + USB 3 hub + USB 3 camera causes Linux kernel crash or panic
                  Intel Corporation
                  This message was posted on behalf of Intel Corporation

                  Thanks for the information provided. I’m not sure if that difference in the output could affect, however, could you please try with the 12V @ 3A wall supply to know if it changes that behavior? Also, have you tried using different powered hubs with your camera?
                   
                  On the other hand, I’m not sure if you are aware of the Intel Joule Discontinuation notice, however, I would like to bring it to you in case it may have any effect on your project.
                   
                  Regards,
                  -Yermi A.
                   

                  • 6. Re: Joule 550x + USB 3 hub + USB 3 camera causes Linux kernel crash or panic
                    WalkingThrough

                    Hello Yermi -

                     

                    In reverse order... Yes, I've seen the discontinuation notice.  Getting caught on the fulcrum point between "finish and show proof-of-concept to potential customers" and "take a month to re-do the entire system" is not a pleasant experience.  Everything ran from an AC supply; no one here thought using a battery would create ..crashes.

                     

                    Running off a +12V @ 3A wall supply lowers the incidence of crashes and also increases the time between crashes.  But it's like waiting for the other shoe to drop.

                     

                    The camera started off being plugged directly into the USB socket.  That caused the joule to crash at random intervals... Thursday it ran 45 minutes.  Friday afternoon there were three crashes in (I'm told) less than five minutes.  No change in software.  When on AC power, an AC-powered hub has been used.  Hm.  I have not tried using a hub inside the unit powered from the battery.

                     

                    What is the current limit for the USB 3 socket on the joule?  The camera has been plugged into a desktop and runs fine there.  But maybe the desktop can source more current than the joule?

                     

                    Mark

                    • 7. Re: Joule 550x + USB 3 hub + USB 3 camera causes Linux kernel crash or panic
                      Intel Corporation
                      This message was posted on behalf of Intel Corporation

                      Hi Mark,
                       
                      The USB3.0 Type-A can source up to 900 mA, USB loads that exceed the 900 mA limit can have unpredictable behavior. Please take a look at the following links for more details about that:

                       
                      Regards,
                      -Yermi A.

                      • 8. Re: Joule 550x + USB 3 hub + USB 3 camera causes Linux kernel crash or panic
                        WalkingThrough

                        Hello Yermi -

                         

                        Looks like there are a couple of problems...

                         

                        Voltage from the USB 3.0 Type A (the wide one with the blue plastic insert) is delivering 3.6 volts.  As an EE, that tells me there's a heavy load... but the device runs fine when plugged into another computer.

                         

                        There is a whopper of a ground loop mixed up with the USB 3.0 shield.  Current in the shield wanders between -10 mA and +15 mA.  It's not oscillating at RF... just wanders between about those two limits.  Not sure how I'm going to break that loop as the ground in the cable is connected to the shield at the Joule end.  Suggestions appreciated.

                         

                        My ..guess?  Ground loop interference with the USB power management system?  The attached peripheral works fine elsewhere.

                         

                        Mark

                        • 9. Re: Joule 550x + USB 3 hub + USB 3 camera causes Linux kernel crash or panic
                          Intel Corporation
                          This message was posted on behalf of Intel Corporation

                          Hi Mark,
                           
                          I would like to know if the USB Type A gives 5V when it does not have any load plugged. If it gives 5V, then the camera you are plugging is a very heavy load for the Joule.
                           
                          Also, I would recommend updating to the latest bios
                          https://downloadcenter.intel.com/download/26948/BIOS-for-Intel-Joule-Compute-Module.
                           
                          Regards,
                          -Yermi A.
                           

                          • 10. Re: Joule 550x + USB 3 hub + USB 3 camera causes Linux kernel crash or panic
                            WalkingThrough

                            Hello Yermi -

                             

                            Looks like the camera damaged the power supply: no-load voltage at the Type A socket is 3.6 volts whether or not anything is plugged into it.  All that surface-mount stuff... can't fix that.

                             

                            Joule + Expansion board assembly has been replaced.

                             

                            I don't know what's going on with that enormous ground loop current... but it was suggested that inserting a DC-DC isolated power supply (12V in, 12V out) to feed the Joule might solve the ground loop.

                             

                            That... seems to have worked.

                             

                            More testing to do.

                             

                            >

                            > [ upgrade the BIOS ]

                            >

                             

                            It's an idea... but this thing is running Canonical's Ubuntu Linux 16.04 LTS from their web site.  At the moment (knock on wood) it's running.  But if I flash up the BIOS, and something else quits, I might be left flat out of luck.  I take regular backups of MMIC memory... but if a mis-flash bricks a Joule or cripples Ubuntu Linux then I have problems.

                             

                            Oh.  On a crash that power button (near the green LED) does not successfully reboot.  Removal of power is required.

                             

                            Mark

                            • 11. Re: Joule 550x + USB 3 hub + USB 3 camera causes Linux kernel crash or panic
                              WalkingThrough

                              It's been a couple months since this crash-a-rama started.  At the moment (and I still don't believe it) there is an isolated DC-DC converter between the battery and the Joule.  That seems to work.  This is the ..equivalent to putting a 12V 3A supply wall-wart in there.

                               

                              Directly from a battery: crash.  Place a 12V to 12V isolated DC-DC converter in there?  Works fine.

                               

                              But it raises havoc with the idea of monitoring the battery voltage.  As the grounds are now isolated, using an INA219 (available from Adafruit) runs into the lack of a return ground...

                               

                              As an experiment, the grounds were all connected.  That WORKED.  But what an expensive fix?

                               

                              What in the world inside the Expansion board and Joule so objects to being driven from a lithium ion battery?