14 Replies Latest reply on Jun 30, 2010 1:46 AM by michael.riepen

    SCC firmware and driver setup problems

    pspillai

      I have been setting up sccKit 1.2.2 on the MCPC according to the setup instructions an have run into a couple of issues.  First a minor caveat: the instructions ask you to run "rm -rf ..." as root to delete files -- this is obviously very dangerous, and a simple typo can render the system useless.  Be careful.

       

      The first part (installing the two tarballs, setting up the config file) went smoothly.  The installation of the driver and firmware using install.csh, had some issues.  The install script tries to copy the driver kernel module to /lib/modules/.../kernel/drivers/mcedev, but this directory does not exist on a clean system.  One needs to manually create this directory before running the script.  Next, the script tries to download firmware files to the BMC.  This failed with an insufficient space error.

       

      It turns out that the usb memory stick was not mounted (actually, not even visible) on the BMC of the system here.  It seems that the front panel USB ports are not working properly on our system.  I tried other memory sticks, and nothing showed up in /proc/bus/usb/devices on the BMC. Finally, I opened up the chassis and used the USB port on the main board next to the BMC card -- this worked.  After resetting the BMC, the USB drive was mounted.  After this, the install script seems to run fine.

       

      After doing the power off MCPC, power on SCC, power on MCPC sequence, the new driver is loaded, and a funny device shows up in the output of lspci (Intel memory controller id c148), so I assume the hardware and driver are working.  The instructions then suggest either root or a user can run the sccBmc -i to train the interface.  However, by default the device (/dev/crbif0rb0) is only accessible by root -- one needs to run "chmod a+rw /dev/crbif0rb0" to allow users access.

       

      Now, I am having issues with the sccBmc -i command.  It just runs and runs, spewing lots of unexpected packet and timeout waiting for read request messages.  It keeps increasing "inc-delay" until 60, then says training unsuccessful, and starts again at 0.  (Actually, the first time through, the MCPC crashed; I redid all of the setup, including firmware download, and the second run is looping right now).  Any ideas what this could be?

       

      - Babu

        • 1. Re: SCC firmware and driver setup problems
          michael.riepen

          First of all, thanks for your valuable feedback. We didn't catch this, as we are only updating existing systems here... I'll add the automatic creation of /lib/modules/.../kernel/drivers/mcedev to the install.csh script. It will be part of the next release...

           

          Yes, "Intel memory controller id c148" is our Rocky Lake board (c048 would be Copperridge)... The access rights of the driver should have been updated by the the file 97-mcedev.rules in /etc/udev/rules.d! Could you please check if this file exists on your system (it is usually generated during step 3 of the Ubuntu installation).

           

          Seeing unexpected packets during "sccBmc -i" is strange... Could you attach a log file of the failed sccBmc training (preferably after re-booting the whole platform)? Thanks!

           

          • 2. Re: SCC firmware and driver setup problems
            pspillai

            I do not seem to have a 97-mcedev.rules file.  I don't see anything in the install.csh script that creates or references this file, so perhaps this is just another clean install vs existing system update issue?  I will clean boot everything and capture a log of the sccBmc -i output shortly.  Thanks,

             

            - Babu

            • 3. Re: SCC firmware and driver setup problems
              pspillai

              Here are log files of the trainting run.  bmc.txt is the output at the BMC console when powering up the scc.  sccBmc.out.txt is the sccBmc -i output; I stopped the run part way through the "slowest" training.  I tried running again, but this time the MCPC crashed after a while (got past inc-delay of 30).  Any thoughts on what the problem might be?  Thanks,

               

              - Babu

              • 4. Re: SCC firmware and driver setup problems
                michael.riepen

                This is not part of the install.csh scriptset. It's a part of the MCPC installation. The file is contained in the ubuntu tar archive and the installation is documented in chapter 3 of the Ubuntu installation guide. Okay, thanks for generating the data!

                • 5. Re: SCC firmware and driver setup problems
                  michael.riepen

                  Wow, that looks strange... It seems that the MCPC requests are looped back to the MCPC, containing a "shift" of the header to the payload section. Hence the unexpected packets (with broken header). Could you please try to execute sccPowercycle prior to executing "sccBmc -i"? Thanks!

                  • 6. Re: SCC firmware and driver setup problems
                    tedk

                    That ubuntu.tar archive is posted on this site ... go to Intel SCC Software ... the link is on the left navigation. The file is call SCC_FIles_for_MCPC_Installation. Possibly we could use a better name ... but if you click on it, you should be able to download ubuntu.tar.bz2.

                    • 7. Re: SCC firmware and driver setup problems
                      tedk

                      Babu, What usb memory stick are you referring to?

                      • 8. Re: SCC firmware and driver setup problems
                        pspillai

                        Yes, you are correct.  My mistake.  I somehow skipped over step 3, which sets up 97-mcedev.rules and modules.conf.  So the magic values in modules.conf presumably were not being used, either.  Anyway, I fixed this and clean booted.  Now, sccBmc -i runs for a bit (fails fast training, starts extended training) and then crashes the host.  I clean booted again, and it crashed again.  Any thoughts?  Thanks,

                         

                        - Babu

                        • 9. Re: SCC firmware and driver setup problems
                          michael.riepen

                          Are you working with the recommended Ubuntu setup? Could you look into your /var/log/messages and find the section (after booting) where the crbif driver gets loaded? The section should look similar to this:

                           

                          Jun 21 14:03:24 bwlrcklabpc01 kernel: [    9.503522] mcedev Id: $Id: mcedev_main.c 16541 2010-06-03 07:55:03Z jbrummer $
                          Jun 21 14:03:24 bwlrcklabpc01 kernel: [    9.503554] mcedev_probe found device: VID 0x8086, DID 0xc148, SVID 0x8086, SDID 0xc048, BUS 0x03, DEVFN 0x00.0

                          Jun 21 14:03:24 bwlrcklabpc01 kernel: [    9.503573] mcedev 0000:03:00.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16

                          Jun 21 14:03:24 bwlrcklabpc01 kernel: [    9.503576] crbif_init: $Id: crbif_main.c 16538 2010-05-31 12:58:11Z jbrummer $ (debuglevel 0000)

                          Jun 21 14:03:24 bwlrcklabpc01 kernel: [    9.503668] rckcrb_init: BITSID: 0x20100608

                          Jun 21 14:03:24 bwlrcklabpc01 kernel: [    9.503680] MIP/MOP size: 64/64KB (13)

                          Jun 21 14:03:24 bwlrcklabpc01 kernel: [    9.503805] IRQ 31/crbif: IRQF_DISABLED is not guaranteed on shared IRQs

                          • 10. Re: SCC firmware and driver setup problems
                            pspillai

                            There was a tiny usb dongle labeled "DeLOCK" attached to the front USB port.  It seems to be a 2 GB USB flash device.  It was not visible or mounted by the BMC, however.  So the install script step to copy the update and firmware files to the BMC (to path /media/usb) failed.  I pulled this out and put it on another port (inside the box), and the BMC mounted it correctly to /media/usb on the next reboot.  The BMC can't seem to see anything plugged into the front panel ( I tried with some other USB stick I had).

                            • 11. Re: SCC firmware and driver setup problems
                              pspillai

                              I believe I have the recommended setup -- it is a clean 64-bit ubuntu 10.04 install.  Here is the relevent bit of /var/log/messages:

                               

                              Jun 24 12:26:58 scc kernel: [ 15.433283] mcedev Id: $Id: mcedev_main.c 16541 2010-06-03 07:55:03Z jbrummer $

                              Jun 24 12:26:58 scc kernel: [ 15.433322] mcedev_probe found device: VID 0x8086, DID 0xc148, SVID 0x8086, SDID 0xc048, BUS 0x03, DEVFN 0x00.0

                              Jun 24 12:26:58 scc kernel: [ 15.433340] mcedev 0000:03:00.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16

                              Jun 24 12:26:58 scc kernel: [ 15.433342] crbif_init: $Id: crbif_main.c 16538 2010-05-31 12:58:11Z jbrummer $ (debuglevel 0000)

                              Jun 24 12:26:58 scc kernel: [ 15.433508] rckcrb_init: BITSID: 0x20100608

                              Jun 24 12:26:58 scc kernel: [ 15.433533] MIP/MOP size: 64/64KB (13)

                              Jun 24 12:26:58 scc kernel: [ 15.433751] IRQ 30/crbif: IRQF_DISABLED is not guaranteed on shared IRQs

                               

                              I think it looks good.

                               

                              - Babu

                               

                              • 12. Re: SCC firmware and driver setup problems
                                michael.riepen

                                Hmm, could you have another look in /var/log/messages and find out if (after aborting a failed "sccBmc -i") there are any error messages in the logfile?

                                • 13. Re: SCC firmware and driver setup problems
                                  pspillai

                                  I got a successful sccBmc -i run!  I ended up changing the host machine -- on the original machine, the hardware is detected and the driver loads, but always gives packet errors.  On the new machine, it just worked.  The software setup is identical (I transplanted the harddisk to the new machine), so it must be the motherboard, chipset, or BIOS differences.  Perhaps we need to start a list of known good and bad systems / motherboards?

                                   

                                  - Babu

                                  • 14. Re: SCC firmware and driver setup problems
                                    michael.riepen

                                    Babu, thanks for the update! I'm glad to hear that your Rocky Lake system is fully functional... Yes, it would be good to have a list of incompatible hardware! However, the best thing would be to try if we can modify the firmware or software to support that hardware as well. Could you please compile a "diff" of both platforms (e.g. "sudo dmidecode" on both PCs or a manual hardware list)? Did you for example use the same PCIe card? We also found out that the PCIe card does not work in each PCIe slot. So it would be interesting to know in which slot you had / have the PCIe card. Thanks in advance!