1 2 Previous Next 15 Replies Latest reply on Jul 14, 2011 5:16 PM by

    Machines that were once provisioned are no longer provisioned

    klmarcellus

      We are currently experiencing issues with our vPro Machines that we manage through SCCM 2007 SP1.  About three months ago we deployed over 300 HP DC7800 machines running AMT v.3.2.2.  Most of them provisioned successfully and we were able to use Out of Band Management on them.  However recently we have noticed that many machines that were once provisioned are no longer provisioned.  The AMT status for the machines show "Unknown"  but the oobmgmt.log shows "Device Already Provisioned".  We are currently unable to use Out of Band Management on most of the machines.

       

      Not sure where to begin troubleshooting.  Need help.

       

      Thank you,

       

      Kristine Marcellus

        • 1. Re: Machines that were once provisioned are no longer provisioned
          Trevor.Sullivan

          Kristine,

           

          The first thing I would do is run meinfowin.exe on one of the clients that you believe is unprovisioned, and see what the provisioning status of the AMT firmware is. Once we know this, we can determine whether or not the problem lies somewhere within the ConfigMgr client software.

           

          Trevor Sullivan

          Systems Engineer

          OfficeMax Corporation

          • 2. Re: Machines that were once provisioned are no longer provisioned
            klmarcellus

            Hi Trevor,

             

            I ran the tool on the client.  Here is the information:

             

            Intel(R) MEInfo Win Version: 2.5.0.1032

            BIOS Version:                786F1 v01.26

            Intel(R) AMT code versions:
                    Flash:                       3.2.2
                    Netstack:                    3.2.2
                    Apps:                        3.2.2
                    Intel(R) AMT:                3.2.2
                    Sku:                         14
                    VendorID:                    8086
                    Build Number:                1033
                    Recovery Version:            3.2.2
                    Recovery Build Num:          1033
                    Legacy Mode:                 False

            Link status:                 Link up
            Cryptography fuse:           Enabled
            Flash protection:            Enabled
            Last reset reason:           Power up
            Setup and Configuration:     Completed
            BIOS Mode:                   Post Boot
            Dedicated Mac Address:       00-22-64-a4-44-4d
            Host Mac Address:            00-22-64-a4-44-4c
            FWU Override Counter:        Always
            FWU Override Qualifier:      Always
            FW on Flash Desc Override:   Disable
            Kedron Driver Version:       Not Available
            Kedron HW Version:           Not Available
            UNS Version:                 3.2.0.1018
            LMS Version:                 3.0.10.1018
            HECI Version:                3.0.30.1086

            • 3. Re: Machines that were once provisioned are no longer provisioned
              Trevor.Sullivan

              Kristine,

               

              It appears that, based upon the "Setup and configuration: Completed" message, that the device truly is provisioned. It would logically conclude that you would receive the "Device already provisioned" message in your oobmgmt.log.

               

              Can you right-click on the system resource and run an AMT discovery? In the context menu, select Out of Band Management --> Discover Management Controllers. When you run this, look at the amtopmgr.log file on ConfigMgr site server, and see what messages come up.

               

              Trevor Sullivan

              Systems Engineer

              OfficeMax Corporation

              • 4. Re: Machines that were once provisioned are no longer provisioned
                klmarcellus

                Hi Trevor,

                 

                Attached are the messages from the amtopmgr.log.

                 

                Thanks,

                 

                Kristine

                • 5. Re: Machines that were once provisioned are no longer provisioned
                  Trevor.Sullivan

                  Kristine,

                   

                  If you run "Update collection membership" on a collection where this AMT system is contained, wait for the query to complete, then refresh the view, what does the AMT status field show as? You might need to go to the View menu to select the AMT status column, but I'm sure you're already familiar with this

                   

                  Trevor Sullivan

                  Systems Engineer

                  OfficeMax Corporation

                  • 6. Re: Machines that were once provisioned are no longer provisioned
                    wryork

                    Can you clear the CMOS battery to reset one of the devices you are having problems with and see if we can get the system to reprovision again through the SCCM agent (as done before)?  This would not be the ultimate fix but I would simply like to know if these devices will re-provision again.

                    • 7. Re: Machines that were once provisioned are no longer provisioned
                      klmarcellus

                      Hi Trevor,

                       

                      I Updated Collection Membership on the collection and did a refresh and the AMT status is now showing Detected.

                       

                      Kristine

                      • 8. Re: Machines that were once provisioned are no longer provisioned
                        klmarcellus

                        Hi Bill,

                         

                        After clearing the CMOS battery we were able to get the machine to reprovision successfully.  We are now able to use Out of Band Managment tools on the machine.

                         

                        Kristine

                        • 9. Re: Machines that were once provisioned are no longer provisioned
                          wryork

                          I have a theory that the Digest password on AMT and SCCM OOB console got out of sync.  If that did actually happen, this would be the behavior I believe.  SCCM could no longer manage the device since it does not have the admin password it uses to manage the device (and this would most like change the status in SCCM to Detected).  The CMOS clear sets everything back to factory default (admin disgest password would be admin, SCCM would know this, and then during provisioning it would modify it and use this new password for management).  Now we need to figure out how this would happen.

                          • 10. Re: Machines that were once provisioned are no longer provisioned
                            Trevor.Sullivan

                            Bill,

                             

                            That's kinda the direction I was going in ... but I wanted to do a little more discovery, first.

                             

                            The thing is ... I think we're going to start running into this issue also. Bear with my theoretical thought process here: Since there is an entry in the ConfigMgr database for AMT that's maintained separately from the ConfigMgr resource, there is a record elsewhere that joins the ConfigMgr resource, and the AMT information (including the randomized digest password). If the ConfigMgr resource record is disjoined from that machine, and a new one is created for the same system (new resource ID, SCCM GUID, etc), then the AMT information will no longer be tied to the ConfigMgr resource.

                             

                            This scenario could happen if, for example, a system is reimaged, which is (or at least can be) a somewhat common occurrence. It could also happen if the ConfigMgr client fails on a system, and needs to be removed / repaired.

                             

                            In my opinion, there needs to be some sort of maintenance task that reguarly checks for "stale" AMT records, and attempts to determine if there is a matching ConfigMgr resource to join it to. Theoretically, this shouldn't be hard to do, though I don't have my ConfigMgr database in front of me to investigate right at the moment.

                             

                            Trevor Sullivan

                            Systems Engineer

                            OfficeMax Corporation

                            • 11. Re: Machines that were once provisioned are no longer provisioned
                              wryork

                              As for reimaging the system, a process should be followed to perform a full unprovision in SCCM to clear the MEBx, remove the AMT OU Object and revoke the certificate for that system name.  Then after the reimaging process, let the SCCM agent perform the normal inband agent based provisioning.  Think of it as a process performed to clean out the AD for old computer objects.  Now for the SCCM agent reinstall, I was not aware this would break the functionality of SCCM to AMT.  This is something I will have to test and report back.  Thanks.

                              • 12. Re: Machines that were once provisioned are no longer provisioned
                                Trevor.Sullivan

                                Bill,

                                 

                                I respectfully disagree.

                                 

                                Since AMT is an out-of-band technology, I think that refreshing an operating system on a client should not require AMT to be unprovisioned and reprovisioned. In fact, I would expect that AMT would facilitate the reimaging of a system, and not cause additional complication to the process.

                                 

                                The method that we are currently using to name our systems, at least at HQ, is based on the serial number of the system. Every time a system is reimaged, it would retain the same hostname, so refreshing it would not cause a hostname mismatch.

                                 

                                Since our system refreshes do not involve renaming a device, I would expect to not have to reprovision a device. On the other hand, if the hostname does change, I realize that it would need to be reprovisioned.

                                 

                                Trevor Sullivan

                                Systems Engineer

                                OfficeMax Corporation

                                • 13. Re: Machines that were once provisioned are no longer provisioned
                                  wryork

                                  I don't think you are disagreeing but merely asking for what you would "like" SCCM product to do.  Unfortunately, reality today is that an unprovision is required if you - and rename the system.  Good feedback for Microsoft and I'd like for you to make note of it through your Microsoft contacts.  We can do the same on our side.

                                   

                                  Intel has worked on developing tools for other ISV console to help with the syncing of names when the OS and AMT names mismatch (Reflector: http://communities.intel.com/docs/DOC-1431).  But this would not help if the system was provisioned with SCCM and associated certificates are loaded in the firmware and AD Objects created for specific AMT names.  Since SCCM applies a cert to each AMT device and it is specific to the name of the AMT system, we need to make sure this certificate name matches.

                                   

                                  Today's reality is that you should perform the process step in SCCM to unprovision the AMT device before re-imaging a system.  Maybe someone in the community has written some tools to help with this manual step in SCCM.  And probably more appropriate for new thread since we are starting to diverge from Kristine's issue and getting more into product improvements.

                                  • 14. Re: Machines that were once provisioned are no longer provisioned
                                    wryork

                                    Kristine,

                                    Can you try to unplug the system so it will reboot the MEBx?  I beleive the problem you are seeing is related to an issue that was discovered with Kerberos.  After 25 continuous days of the MEBx being on (remember that could include the OS off but MEBx still running in S5 state) and the machine stops accepting Kerberos credentials.  The problem has been root caused and firmware is being worked to address.  This will be released trhough your OEM.  Please validate that this power cycle addresses your issue.  Thanks.

                                    1 2 Previous Next