5 Replies Latest reply on Nov 21, 2015 7:02 PM by xthunderheartx

    SYSRQ ... Really? And what is it with tail -f ???

    xthunderheartx

      Nov  9 10:49:43 galileo user.info kernel: [32994.858726] SysRq : HELP : loglevel(0-9) reBoot Crash terminate-all-tasks(E) memory-full-oom-kill(F) kill-all-tasks(I) thaw-filesystems(J) saK show-memory-usage(M) nice-all-RT-tasks(N) powerOff show-registers(P) show-all-tim

       

      I am seeing this in my syslog and I have no idea why.  Trust me, my hands are no where near the Majik SysRQ key, much less the Alt or H keys.  Heck I was blissfully ignorant of the existence of the SysRQ key until this came along and I've been rocking Linux since kernel 0.92!  In this case it apparently thought I did an Alt-SysRQ-h but I've seen it spit out compaints that a SysRQ function not being implemented or some such.

       

      What really concerns me about this, other than I hate syslog being polluted, is that immediately after this happens I miss a character from ttyS1 which I am using to read data from a PIC controlling three ASONG DHT11/22's.  Obviously something untoward is happening and I need to get to the bottom of it.

       

      And why does tail -f just randomly decide to stop output to the screen?  I shell into my Gen2 target, tail -f /var/log/messages and everything is peach for a few seconds, and then output just stops.  The data is still streaming into the file, tail just isn't listening (or maybe talking) anymore.

       

      Any clues?

       

      Dallas

        • 1. Re: SYSRQ ... Really? And what is it with tail -f ???
          PabloM_Intel

          Hi xthunderheartx,

           

          Do you have anything connected to your board right now? Any external circuitry that might be sending electrical noise? This can be one of the reasons of why you’re getting this SysRq message in your syslog. Now, if you don't want this message to show up you can disable this in the kernel. I would suggest you to check this documentation https://www.kernel.org/doc/Documentation/sysrq.txt.

          About tail –f, let us test this and we will get back to you.

           

          Regards,

          PabloM_Intel

          • 2. Re: SYSRQ ... Really? And what is it with tail -f ???
            xthunderheartx

            Yes PabloM_Intel I do have other things that radiate: A Numato GPIO extender shield and our own custom board with a Silicon Labs World Modem and a PIC we use to interface ASONG 1-wire-ish sensors.  So what you are suggesting is that somehow, EMI generated off-board is generating an interrupt right?  Hummmm ... as I understand it, the way the Majik SysRQ works is that somehow, somebody (I imagine the keyboard driver), recognizes the special key pattern (Alt-SysRQ-<command>) and generates a software interrupt.  The handler for this then takes care of doing the Majikx.  That essentially correct?  Since we are unmapping TTYS1 and pointing it to the Arduino header maybe somehow we are generating the majik sequence or noise is doing something weird ???  I don't know dood.  Seems a little thin but stranger things have happened.  Kinda makes sense though.  When I shutdown the getty on TTYS1, wouldn't that tell the SysRQ handler to stop listening?  If for some reason it was still looking at the raw data coming from the UART (which we read via normal linux infrastructure, i.e. termios), maybe it sees that sequence?

             

            From: https://www.kernel.org/doc/Documentation/sysrq.txt

             

            On the serial console (PC style standard serial ports only) - You send a BREAK, then within 5 seconds a command key. Sending BREAK twice is interpreted as a normal BREAK.

             

            We use TTYS1 to receive exactly 6 bytes from our ASONG interface.  We never transmit.  We use one of three GPIO pins (4,5, or 6) to select a corresponding DTH11 sensor.  Then we yank our PIC out of reset (using another GPIO), it reads the sensor, and sends the data via the TTYS1 uart @ 1200 baud.  The data contains <Relative Humidity>, 00, <Air Temp in Celsius>, 00, <CHECKSUM>, <CARRIAGE RETURN>.  The temp in my Lab hovers around 70 +/- 2 degrees Fahrenheit and the humidity has been between 56% and 48%.  Here is some typical data:

             

            Nov  9 23:42:27 galileo user.debug AsongTest[522]: Supply Port->Asong::read (std::vector<int>*) called

            Nov  9 23:42:27 galileo user.debug AsongTest[522]: Supply Port->Asong::dataAged ()called

            Nov  9 23:42:28 galileo user.debug AsongTest[522]: pushing 51

            Nov  9 23:42:28 galileo user.debug AsongTest[522]: pushing 0

            Nov  9 23:42:28 galileo user.debug AsongTest[522]: pushing 21

            Nov  9 23:42:28 galileo user.debug AsongTest[522]: pushing 0

            Nov  9 23:42:28 galileo user.debug AsongTest[522]: pushing 72

            Nov  9 23:42:28 galileo user.debug AsongTest[522]: pushing 13

            Nov  9 23:42:28 galileo user.debug AsongTest[522]: pdata size 6

            Nov  9 23:42:28 galileo user.debug AsongTest[522]: Return Port->Asong::read (std::vector<int>*) called

            Nov  9 23:42:28 galileo user.debug AsongTest[522]: Return Port->Asong::dataAged ()called

            Nov  9 23:42:28 galileo user.debug AsongTest[522]: pushing 42

            Nov  9 23:42:28 galileo user.debug AsongTest[522]: pushing 0

            Nov  9 23:42:28 galileo user.debug AsongTest[522]: pushing 20

            Nov  9 23:42:28 galileo user.debug AsongTest[522]: pushing 0

            Nov  9 23:42:28 galileo user.debug AsongTest[522]: pushing 62

            Nov  9 23:42:28 galileo user.debug AsongTest[522]: pushing 13

            Nov  9 23:42:28 galileo user.debug AsongTest[522]: pdata size 6

            Nov  9 23:42:28 galileo user.debug AsongTest[522]: Outside Port->Asong::read (std::vector<int>*) called

            Nov  9 23:42:28 galileo user.debug AsongTest[522]: Outside Port->Asong::dataAged ()called

            Nov  9 23:42:29 galileo user.debug AsongTest[522]: pushing 61

            Nov  9 23:42:29 galileo user.debug AsongTest[522]: pushing 0

            Nov  9 23:42:29 galileo user.debug AsongTest[522]: pushing 20

            Nov  9 23:42:29 galileo user.debug AsongTest[522]: pushing 0

            Nov  9 23:42:29 galileo user.debug AsongTest[522]: pushing 81

            Nov  9 23:42:29 galileo user.debug AsongTest[522]: pushing 13

            Nov  9 23:42:29 galileo user.debug AsongTest[522]: pdata size 6

             

            Now if somehow that data stream simulated a break ...?  I don't know ... does that even seem to fall within the realm of possibility?  The fact that we miss a character from the interface every time we see the SysRQ definitely suggest that someone else is consuming data from the UART. Or possibly the low-level serial code sees data it doesn't think is a character, like a break for instance, and somehow this cause a SysRQ trigger???

             

            Interesting problem anyway you look at it.  Please add any thing you can and correct me if I'm going down a Rabbit Hole with this logic.

             

            Thanx,

             

            Dallas

            • 3. Re: SYSRQ ... Really? And what is it with tail -f ???
              PabloM_Intel

              Hi xthunderheartx,

               

              Did you already try adding 0 value to /proc/sys/kernel/sysrq? Did you get different results? For now we don’t exactly know what is causing the issue but we can try to avoid it.

               

              Regards,

              PabloM_Intel

              • 4. Re: SYSRQ ... Really? And what is it with tail -f ???
                CMata_Intel

                Hi Dallas,

                 

                Do you have updates on this? Did you try with Pablo’s suggestion?

                Have you try to see the signals with an oscilloscope or logic analyzer to identify if the data is actually there (related to SysRQ ) ?

                 

                Regards,

                Charlie

                • 5. Re: SYSRQ ... Really? And what is it with tail -f ???
                  xthunderheartx

                  Yeah Charlie I figured it out.  Again our circuit uses a PIC to read 1 of 3 ASONG sensors.  One of the PIC's pins acts as a transmit back to TTYS1 and another acts as the 1-wire(ish) bus to the sensor.  Each of the sensors can be muxed (actually switched) onto the the 1-wire pin through a Maxim MAX318 switch.  When the sensors aren't being read, we hold the PIC in reset.  When we get ready to read a sensor, we mux it onto the PIC then release the PIC from reset.  The PIC reads the ASONG 1-wire(ish) protocol to get the sensor data, then sends it up @ 1200 baud through TTYS1.

                   

                  The problem is that when the PIC is in reset he floats the pin we use to signal back to the Galileo and we don't pull it up.  As a result, occasionally, before the PIC gets out out reset, reads the sensor and starts transmitting back to us, we start to read TTYS1.  At that point the transmit line is floating around ground.  This results in a false "break" being detected by the UART which is what apparently triggers the SYSRQ handler.

                   

                  A pull-up on the signal from the PIC to TTYS1 makes everybody happy.