12 Replies Latest reply on Dec 4, 2014 2:13 PM by deium

    I partially bricked i2c-6... and my Edison

    DimitriM

      Hi everybody,

       

      I was experimenting with i2c in connection with MPU6050 writing in Eclipse. It worked so far without problems. A simple server was sending data via Wifi to a client written in Delphi.

      Nothing special actually. I just wanted to say that it has been doing right. Then, I wanted to test more distance between the Edison (on the Arduino board) and the router.

      So, I connected the board to another socket and promptly started getting following kernel messages in the terminal window when I ran the proggie from Eclipse:

       

      [ 7384.240427] i2c-designware-pci 0000:00:09.1: ===== REGISTER DUMP (i2c) =====

      [ 7384.240529] i2c-designware-pci 0000:00:09.1: DW_IC_CON:               0x65

      [ 7384.240612] i2c-designware-pci 0000:00:09.1: DW_IC_TAR:               0x68

      [ 7384.240693] i2c-designware-pci 0000:00:09.1: DW_IC_SS_SCL_HCNT:       0x2f8

      [ 7384.240773] i2c-designware-pci 0000:00:09.1: DW_IC_SS_SCL_LCNT:       0x37b

      [ 7384.240853] i2c-designware-pci 0000:00:09.1: DW_IC_FS_SCL_HCNT:       0x87

      [ 7384.240933] i2c-designware-pci 0000:00:09.1: DW_IC_FS_SCL_LCNT:       0x10a

      [ 7384.241013] i2c-designware-pci 0000:00:09.1: DW_IC_INTR_STAT:         0x0

      [ 7384.241091] i2c-designware-pci 0000:00:09.1: DW_IC_INTR_MASK:         0x246

      [ 7384.241171] i2c-designware-pci 0000:00:09.1: DW_IC_RAW_INTR_STAT:     0x10

      [ 7384.241250] i2c-designware-pci 0000:00:09.1: DW_IC_RX_TL:             0x20

      [ 7384.241328] i2c-designware-pci 0000:00:09.1: DW_IC_TX_TL:             0x20

      [ 7384.241407] i2c-designware-pci 0000:00:09.1: DW_IC_ENABLE:            0x1

      [ 7384.241485] i2c-designware-pci 0000:00:09.1: DW_IC_STATUS:            0x2

      [ 7384.241563] i2c-designware-pci 0000:00:09.1: DW_IC_TXFLR:             0x1

      [ 7384.241642] i2c-designware-pci 0000:00:09.1: DW_IC_RXFLR:             0x0

      [ 7384.241720] i2c-designware-pci 0000:00:09.1: DW_IC_TX_ABRT_SOURCE:    0x0

      [ 7384.241798] i2c-designware-pci 0000:00:09.1: DW_IC_DATA_CMD:          0x0

      [ 7384.241875] i2c-designware-pci 0000:00:09.1: ===============================

      [ 7384.241987] CPU: 0 PID: 414 Comm: TCP_server Tainted: G        W  O 3.10.17-poky-edison+ #1

      [ 7384.241992] Hardware name: Intel Corporation Merrifield/BODEGA BAY, BIOS 466 2014.06.23:19.20.05

      [ 7384.241999] task: f5700590 ti: f64f0000 task.ti: f64f0000

      [ 7384.242061] Stack:

      [ 7384.242130] Call Trace:

      [ 7384.242580] Code: b3 ff ff 89 f8 09 d0 80 ce 04 83 ff 02 0f 44 c2 8b 15 54 ab b0 c1 89 82 00 b3 ff ff f7 c6 00 02 00 00 74 14 e8 67 f7 0a 00 56 9d <83> c4 04 5b 5e 5f 5d c3 90 8d 74 26 00 56 9d e8 51 f3 0a 00 83

      [ 7384.242604] CPU: 1 PID: 0 Comm: swapper/1 Tainted: G        W  O 3.10.17-poky-edison+ #1

      [ 7384.242609] Hardware name: Intel Corporation Merrifield/BODEGA BAY, BIOS 466 2014.06.23:19.20.05

      [ 7384.242618] task: f6c83d30 ti: f6e1c000 task.ti: f6e1c000

      [ 7384.242678] Stack:

      [ 7384.242746] Call Trace:

      [ 7384.242951] Code: 8b 42 08 a8 08 75 24 31 c9 8d 42 08 89 ca 0f 01 c8 0f ae f0 89 f6 89 e0 25 00 e0 ff ff 8b 40 08 a8 08 75 07 b1 01 89 f0 0f 01 c9 <85> 1d 58 fd b3 c1 75 0d 8d 55 f0 b8 05 00 00 00 e8 8d 34 d9 ff

      [ 7384.372957] i2c-6: recovery ignore

       

      In the same time, in the console of Eclipse, it looked like this:

       

      root@edison:~# echo $PWD'>'

      /home/root>

      root@edison:~#

      root@edison:~# chmod 755 /tmp/TCP_server;/tmp/TCP_server;exit             <<<<< the program TCP_server is being started remotely

      Failed to write byte to I2C slave: Connection timed out

       

      Broadcast message from systemd-journald@edison (Thu 2014-11-20 01:33:40 UTC):

      kernel[198]: [ 7384.241987] CPU: 0 PID: 414 Comm: TCP_server Tainted: G        W  O 3.10.17-poky-edison+ #1

       

      Broadcast message from systemd-journald@edison (Thu 2014-11-20 01:33:40 UTC):

      kernel[198]: [ 7384.241992] Hardware name: Intel Corporation Merrifield/BODEGA BAY, BIOS 466 2014.06.23:19.20.05

       

      Broadcast message from systemd-journald@edison (Thu 2014-11-20 01:33:40 UTC):

      kernel[198]: [ 7384.241999] task: f5700590 ti: f64f0000 task.ti: f64f0000

       

      I have to mention that I used  /dev/i2c-6 in my tests as explained in the docs. Interestingly, when I switched in the program to, for example, /dev/i2c-7 or 5 the program terminated without all those kernel rants with an expected Remote I/O error. Well, no MPU6050 with addr 0x68 connected there. It's logical.

       

      I re-flashed the Edison with the same “edison-rel1-maint-weekly_build_16_2014-10-14_14-56-19” as before the crash. After logging in, I got this (Wed Nov 19 23:30:45 2014 is wrong. It's Nov 29 today):

       

      edison login: root

      [   18.331638] systemd-fsck[231]: /dev/mmcblk0p10: Superblock last mount time (Wed Nov 19 23:30:45 2014,

      [   18.334598] systemd-fsck[231]: now = Thu Oct  9 09:22:41 2014) is in the future.

      [   18.341925] systemd-fsck[231]: /dev/mmcblk0p10: UNEXPECTED INCONSISTENCY; RUN fsck MANUALLY.

      [   18.344492] systemd-fsck[231]: (i.e., without -a or -p options)

      root@edison:~#

      root@edison:~# fsck

      fsck from util-linux 2.24.1

      dosfsck 2.11, 12 Mar 2005, FAT32, LFN

      /dev/mmcblk0p7: 5 files, 2592/2819 clusters

      e2fsck 1.42.9 (28-Dec-2013)

      /dev/mmcblk0p10 is mounted.

      e2fsck: Cannot continue, aborting.

       

      root@edison:~#

       

      Now, it looked like a real trouble. Rebooting the Edison gave me following messages right after login as root:

       

      after 1st reboot

       

      edison login: root

      [   15.530733] systemd-fsck[232]: /dev/mmcblk0p10: Superblock last mount time is in the future.

      [   15.533549] systemd-fsck[232]: (by less than a day, probably due to the hardware clock being incorrectly set)  FIXED.

      [   15.536053] systemd-fsck[232]: /dev/mmcblk0p10: Superblock last write time is in the future.

      [   15.538985] systemd-fsck[232]: (by less than a day, probably due to the hardware clock being incorrectly set).  FIXED.

      [   15.541724] systemd-fsck[232]: /dev/mmcblk0p10 contains a file system with errors, check forced.

      [   15.639925] systemd-fsck[232]: /dev/mmcblk0p10: 16/152608 files (0.0% non-contiguous), 26870/610299 blocks

      root@edison:~#

       

       

      after 2nd reboot

       

      edison login: root

      [   14.910441] systemd-fsck[231]: /dev/mmcblk0p10: Superblock last mount time is in the future.

      [   14.913046] systemd-fsck[231]: (by less than a day, probably due to the hardware clock being incorrectly set)  FIXED.

      [   14.915579] systemd-fsck[231]: /dev/mmcblk0p10: Superblock last write time is in the future.

      [   14.931268] systemd-fsck[231]: (by less than a day, probably due to the hardware clock being incorrectly set).  FIXED.

      [   14.932956] systemd-fsck[231]: /dev/mmcblk0p10: clean, 16/152608 files, 26870/610299 blocks

      root@edison:~#

       

       

      after 3rd reboot

       

      edison login: root

      [   34.448363] systemd-fsck[231]: /dev/mmcblk0p10: Superblock last write time is in the future.

      [   34.451189] systemd-fsck[231]: (by less than a day, probably due to the hardware clock being incorrectly set).  FIXED.

      [   34.453557] systemd-fsck[231]: /dev/mmcblk0p10: clean, 16/152608 files, 26870/610299 blocks

      root@edison:~#

       

      I need your help to understand how severe this problem is. And how to fix it, of course.

       

      Best regards,

      Dmitri

        • 1. Re: I partially bricked i2c-6... and my Edison
          DimitriM

          I looked at date and hwclock settings. Both were way off what is today:

           

          root@edison:~# hwclock --localtime

          Sat Jan  1 00:02:28 2000  0.000000 seconds

          root@edison:~# date

          Thu Oct  9 09:25:18 UTC 2014

           

          Then, I sat the current date and time

           

          root@edison:~# date --set "2014-11-29 10:46"

          Sat Nov 29 10:46:00 UTC 2014

          root@edison:~# date

          Sat Nov 29 10:46:03 UTC 2014

           

          ...and rebooted the Edison without disconnecting the power supply (# reboot):

           

          edison login: root

          [  15.510324] systemd-fsck[231]: /dev/mmcblk0p10: Superblock last write time (Sat Nov 29 10:46:29 2014,

          [  15.579363] systemd-fsck[231]: now = Thu Oct  9 09:22:37 2014) is in the future.

          [  15.581123] systemd-fsck[231]: /dev/mmcblk0p10: UNEXPECTED INCONSISTENCY; RUN fsck MANUALLY.

          [  15.582675] systemd-fsck[231]: (i.e., without -a or -p options)

          root@edison:~# date

          Thu Oct  9 09:22:51 UTC 2014

          root@edison:~# hwclock --localtime

          Sat Jan  1 00:01:01 2000  0.000000 seconds

          root@edison:~#

           

          As you can see, the date has been reset to the old value (Thu Oct  9 09:22:51 UTC 2014).

           

          And it keeps ranting that from the system's point of view, something happened "in the future". Then, it pretends to fix it which kinda fails:

          edison login: root

          [   37.422874] systemd-fsck[231]: /dev/mmcblk0p10: Superblock last mount time is in the future.

          [   37.425737] systemd-fsck[231]: (by less than a day, probably due to the hardware clock being incorrectly set)  FIXED.

          [   37.428131] systemd-fsck[231]: /dev/mmcblk0p10: Superblock last write time is in the future.

          [   37.431007] systemd-fsck[231]: (by less than a day, probably due to the hardware clock being incorrectly set).  FIXED.

          [   37.433668] systemd-fsck[231]: /dev/mmcblk0p10: clean, 16/152608 files, 26870/610299 blocks

          root@edison:~#

           

          Sorry for so many listing. I tried to describe the problem as precise as possible ;-)

          • 2. Re: I partially bricked i2c-6... and my Edison
            deium

            Some of the output reminds my of the factory build_56

            you can find your version by running cat /etc/version

            Many things have been fixed in the images you should upgrade to the lastest firmware which will report build_16

            • 3. Re: I partially bricked i2c-6... and my Edison
              DimitriM

              Ty for your reply.

              As I stated before, my build is edison-rel1-maint-weekly_build_16_2014-10-14_14-56-19.


              root@edison:~# cat /etc/version

              edison-rel1-maint-weekly_build_16_2014-10-14_14-56-19


              And yes, build 56 was extremely buggy. Made me feel a lot of pain.

              • 4. Re: I partially bricked i2c-6... and my Edison
                deium

                With regards to the hwclock, it will hold if you have battery power for the RTC.  If not, the system will get its time from the timesync daemon (which grabs the time from google).  You can use hwclock --systohc to push network time to the RTC.  I know that the issue of the clock being off on a reboot before timesync has been mentioned before and is being worked on and possibly available in the next firmware release.

                 

                edison login: root

                [   37.422874] systemd-fsck[231]: /dev/mmcblk0p10: Superblock last mount time is in the future.

                [   37.425737] systemd-fsck[231]: (by less than a day, probably due to the hardware clock being incorrectly set)  FIXED.

                [   37.428131] systemd-fsck[231]: /dev/mmcblk0p10: Superblock last write time is in the future.

                [   37.431007] systemd-fsck[231]: (by less than a day, probably due to the hardware clock being incorrectly set).  FIXED.

                [   37.433668] systemd-fsck[231]: /dev/mmcblk0p10: clean, 16/152608 files, 26870/610299 blocks

                root@edison:~#

                • 5. Re: I partially bricked i2c-6... and my Edison
                  deium

                  > I have to mention that I used  /dev/i2c-6 in my tests as explained in the docs. Interestingly, when I switched in the program to, for example, /dev/i2c-7 or 5 the

                  > program terminated without all those kernel rants with an expected Remote I/O error. Well, no MPU6050 with addr 0x68 connected there. It's logical.

                   

                  The Intel Arduino board has only i2c-6, the Intel Edison mini breakout board exposes i2c-1 and i2c-6 (i2c-7/i2c-5  non existent or unexposed)

                  Note Edison i2c can only be configured as the master

                  • 6. Re: I partially bricked i2c-6... and my Edison
                    KurtE

                    This reminds me a lot of the issues I have with my Edison on the small (non-Arduino) breakout board and my experiments with I2C.  Details in the thread: I2C with Edison Breakout board

                     

                    Are you running this on the small mini board or the Arduino version?  Are you using some external Pull up resistors? ...   What voltage is your I2C device?  Are you running it through some voltage level converters?

                     

                    Background on my stuff:

                     

                    When I was running a simple test program to try to communicate with a old I2c compass (CMPS03). I had a couple of PuTTY windows running to the board(the hardware Serial port and one over wifi).  I then started the program, which which show any valid data in my debug terminal, and started to give the same broadcasts like you mention and lots of stuff you can see with dmesg.

                     

                    Some of the things I saw, included: both SCL and SDA had a low value (no pull-up), even though the device I was connected to bidirection level converter which has built in PU resistors on both sides of the level converter... I disconnected everything except still had SCL/SDA connected to Logic Analyzer, I manually enabled the PU resistors that are part of those IO pins and the errors went away.  

                     

                    I then redid the connections: I removed edison module, I touched each of the solder joints for the connectors (2 2x14 male headers) to make sure no cold joints, I attached Edison, I then hooked up jumpers, back to the breadboard with Level converter and device, reran the program and it worked. .  I ran it several different times with both an MRAA and an Arduino version.  Found a bug in MRAA, plus in my program, fixed the program and reran and then I started to test the MRAA one again and started to get the same errors again...  But this time, more things stopped working, including the external debug terminal.  Note: On the last rebuild, I soldered on a power connector (the one from Dig-key mentioned in hardware manual).  Maybe I screwed up and solder went to far and shorted something?   So I thought I would reflash the whole thing, to see if that helped.  The debug Terminal is still dead, the module is running hot, but I can still talk to it and program it with Arduino Sketch.  Currently I have one on it that allows me to run one command at a time and see the results...

                     

                    So my current solution, was to order a new Edison with Breakout board, which is currently sitting in my mailbox in town.  Hope to pick it up sometime today.   Once I get it setup again, I will continue to test some of the different Arduino capabilities, but I must say I may put any additional I2C testing on hold in case it can somehow actually brick the the Edison.

                     

                    So unfortunately I am not sure how much my issues are related to your issues here.  If they are, as I mentioned, I can still talk to most of it.  Will be happy to try different things to see if it helps...  Not sure what yet?

                     

                    Other things that I wonder about, include.  Are there other I2C devices (internal to Edison or on breakout board) that are connected to this buss (6).  What are their I2C addresses?  Is there already some PU enabled for them?  I know on my Arduino version of the board, I have SCL/SDA signals that are high and I have not installed any PU resistors when I was doing that test, and the I2C subsystem turns off the PU resistors associated with the IO pins on the Arduino board.

                     

                    Likewise wonder if there are internal I2C devices, could our I2C tests some how get misinterpreted by them and and maybe reprogram something not to work right?  If so is there a way to reprogram all of them back to their defaults?

                     

                    Again sorry that I don't have any real answers, more like more questions.

                    Kurt

                    • 7. Re: I partially bricked i2c-6... and my Edison
                      DimitriM

                      deium wrote:

                       

                      The Intel Arduino board has only i2c-6, the Intel Edison mini breakout board exposes i2c-1 and i2c-6

                       

                      This is true. Still, they are there and thus can be claimed without raising an exception, although with no effect what so ever.

                       

                      root@edison:~# ls /dev/i2c*

                      /dev/i2c-1  /dev/i2c-3  /dev/i2c-5  /dev/i2c-7

                      /dev/i2c-2  /dev/i2c-4  /dev/i2c-6  /dev/i2c-8

                      • 8. Re: I partially bricked i2c-6... and my Edison
                        deium

                        I think there is more to be discovered in the /sys/class/gpio for what is actually there and what the capabilities are.

                        • 9. Re: I partially bricked i2c-6... and my Edison
                          Susutawari

                          I have same problem of "the date has been reset to the old value" on same build (edison-rel1-maint-weekly_build_16_2014-10-14_14-56-19).

                          • 10. Re: I partially bricked i2c-6... and my Edison
                            deium

                            Susutawari, without a battery to hold the RTC, your date will  reset.  On a normal boot, Edison will sync to network time, which you can then push to the RTC with hwclock --systohc

                            • 11. Re: I partially bricked i2c-6... and my Edison
                              Susutawari

                              Thank you deium, I solved it using hwclok command with power supply on V_VBAT_BKUP pin like this:

                              R0012784.JPG

                              After 1st booting:

                              1. >date -s "2014-12-04 23:42"

                              2. >hwclock -w (same as --systohc)

                              3. Turn on power supply for V_VBAT_BKUP pin.

                              4. Un-plug power cable such as USB cable.

                              5. Re-plug power cable and start 2nd booting.

                               

                              After 2nd booting:

                              1. >hwclock -s (same as --hctosys)

                              2. I can see correct time with "date" command.

                               

                              Do you know where is the best way to execute "hwclock -s" at booting?

                              • 12. Re: I partially bricked i2c-6... and my Edison
                                deium

                                because my edison is synced to UTC, I need to wait till the network is up, and timesyncd is in sync with google's time servers.  Therefore I have my system('hwclock --systohc') inside my daemon.  My daemon is scheduled to be before user can login.