11 Replies Latest reply on Sep 11, 2015 11:54 AM by Maaaah

    Yocto read-only file system

    Maaaah

      Hi,

       

      we have done a preproduction run on Edison on a custom board for a wearable device. We based this run on edison's image ww05-15. Some parts have now come back from our testing partners with read-only Edison's that won't allow us to set their wireless configurations in the field (since it needs to save the connection information). It is a guess that the read-only Edison is done because of pulling power or improper shutdown. Since the 10 second hold on the power button pulls power instead of issuing a poweroff it's really likely that our Edison's always have a bad power down. I'm hoping someone could give a step by step solution for restoring the Edison's we got back, a suggestion for how to recover this state on deployed products in the field and a suggestion that has Edison not enter this state in our future production runs.

       

      Thanks,

       

      J

        • 1. Re: Yocto read-only file system
          Intel_Peter

          Hello Maaaah,

           

          That behavior is very strange on an Edison, however, I believe they may be able to be recovered by reflashing them. Have you already tried to reflash your board? If so, then, there are a couple things you can try; you could use the Flash Tool Lite (which you can find as well as a guide on how to use it in IoT - Intel® Edison Board Download | Intel® Developer Zone) or you could try the flashall method.

           

          Peter.

          • 2. Re: Yocto read-only file system
            Maaaah

            Hi Peter,

             

            I can flash it. I need to recover it. We had a minimal release of 60 units and 2 came back with this problem, so I wouldn't say that it's all that strange. It totally looks like a power down problem, we've seen reproducible problems with the the hold button not issuing a true poweroff or shutdown and having the filesystem corrupted. Can you walk through the recovery steps, and make suggestions on how to minimize this problem? I can patch through our phone application if you can suggest ways to make the Edison more resilient to flaky power offs.

             

            Thanks,

             

            M

            • 3. Re: Yocto read-only file system
              Intel_Peter

              Did you try the flashall method? I mean, does that behavior continues after recovering the board with flashall? If so, that's very strange any since flashall should solve any software related issues.

               

              Peter.

              • 4. Re: Yocto read-only file system
                Maaaah

                Hi Peter,

                 

                I'm pretty sure that will put our base image on, and does not allow us to recover anything in the field or ensure it doesn't happen again.

                 

                J

                • 5. Re: Yocto read-only file system
                  Intel_Peter

                  I see, I'm sorry I completely misunderstood you before. Could you please tell me a little more about the issue, when and how did it happen? What does your application does? I just want to understand a little more about the environmental factor to see if it could have affected.

                   

                  Peter.

                  • 6. Re: Yocto read-only file system
                    Maaaah

                    Thanks Peter,

                     

                    It's a medical, battery powered wearable. We've sent out about 60 to partners to test and got 2 back after a month with this failure. We have a hot-swap on the batteries that might be wonky, but we've seen the built in powerdown that comes with the yocto image doesn't wait for filesystem writes to complete and causes some file corruptions. I kind of feel this might be similar.

                     

                    Cheers,

                     

                    J

                    • 7. Re: Yocto read-only file system
                      Maaaah

                      Hi Peter,

                       

                      we just had another returned. Can you provide a way to recover the device to read/writable state without wiping it?

                       

                      Thanks,

                       

                      j

                      • 8. Re: Yocto read-only file system
                        Frederick Blais

                        Maaaah

                         

                        I had this problem before (Ubilinux - Read-only file system )

                         

                        To solve the problem, I used these commands :

                        umount /dev/mmcblk0p8

                        e2fsck /dev/mmcblk0p8

                        reboot

                         

                        I hope this will help you

                        • 9. Re: Yocto read-only file system
                          Maaaah

                          Thanks for your note Frederick,

                           

                          I did read through your discussion before. I think the Yocto image is different ( I think it mounts to /dev/mmcblk0p7, doing umount /dev/mmcblk0p8 results in umount: can't umount /dev/mmcblk0p8: Invalid argument ) and it fails at e2fsck when in a read only state (sorry I just flashed the device to start over) and gives significant warnings when the drive is still ok in RW. Since flashing does revert to a read/write state, I know I'm just missing something programmatically, and going with Yocto was probably a bad choice.

                           

                          Out of interest, I think my root cause is from a bad power design. I think we might have some cases in the field that battery contacts are intermittent. Any chance your device failed from pulling power too quickly/often?

                           

                          Thanks,

                           

                          Jeff

                          • 10. Re: Yocto read-only file system
                            Frederick Blais

                            I'm 100% sure it is related to removing power while writing to flash.

                             

                            It happened to me twice when I made a program that did not want to exit, I had to remove the power to reboot the board. After start-up, the filesystem was R-O.

                            I remember seeing errors in dmesg

                             

                            • 11. Re: Yocto read-only file system
                              Maaaah

                              Thanks Frederick,

                               

                              I'm afraid that this happens a lot on our device. It seems strange that the OS partition goes read-only and our /home doesn't; the /home gets written to all the time, but the OS should only get .conf's written on wifi p2p pairings, hmmm maybe journal writes to the OS. Either way, without an automatic recovery I think we'll have to switch off the Edison for our wearables with battery power.

                               

                              Thanks for your time,

                               

                              jeff