Failed to call CheckCertificate provider method (oobmgmt.log)

[idata]

I am getting an error in my oobmgmt.log on a unprovisioned AMT 2.5.0 ConfigMgr client that says "Failed to call CheckCertificate provider method, 80041001".

Any ideas what the source of this issue could be?

Trevor Sullivan

Systems Engineer

OfficeMax Corporation

[idata]

Here is the output from MEinfowin.

Intel(R) MEInfo Win Version: 2.5.0.1032

BIOS Version: 68MCU Ver. F.15

Intel(R) AMT code versions:

Flash: 2.5.0 Netstack: 2.5.0 Apps: 2.5.0 Intel(R) AMT: 2.5.0 Sku: 12 VendorID: 8086 Build Number: 1084 Recovery Version: 2.5.0 Recovery Build Num: 1084 Legacy Mode: False

Link status: Link down

Cryptography fuse: EnabledFlash protection: EnabledLast reset reason: Firmware ResetSetup and Configuration: Not startedBIOS Mode: Post BootDedicated Mac Address: 00-16-d4-f0-f2-59Host Mac Address: 00-16-d4-f0-f2-58FWU Override Counter: NeverFWU Override Qualifier: AlwaysFW on Flash Desc Override: DisableKedron Driver Version: 11.1.0.86Kedron HW Version: 0.0.40UNS Version: 2.5.7.1083LMS Version: 2.5.10.1083HECI Version: 2.5.29.1090

FYI, this error was showing up on an Intel-provided HP Compaq 6910p, which appears to have an OEM build of Vista on it. It has a lot of customized local policy settings in Windows that may be interfering with things, so I am going to reload it with our standard build.

I just wanted to document this error message as something that may come up ...

Trevor Sullivan

Systems Engineer

OfficeMax Corporation

[idata]

The CheckCertificate provider method being referred to appears to be a WMI method on the CCM_PrvsMethodProvider class in the root\ccm WMI namespace. This is a static method and it takes a single string parameter, which I believe is the root certificate hash to be matched in the firmware.

I'm not sure why this would have been failing, but perhaps for some reason, it didn't like the hash it was fed?

Trevor Sullivan

Systems Engineer

OfficeMax Corporation

[idata]

Hmmm, well I upgraded the system to AMT 2.6.1, and loaded Windows 7 Beta Build 7000 on it, and I'm still getting the same error.

This message is also being logged just before the original error I posted:

<![LOG[Raising event:<p> 

[SMS_CodePage(437), SMS_LocaleID(1033)]instance of SMS_OOBMgmt_StartConfig_Failure{ClientID = "GUID:0BF82776-D18D-4F30-AC91-FBA6D47F0CDC";DateTime = "20090217171109.097000+000";ErrorCode = "1";FailureCategory = "Failed to enumerate certificate hash, please check if the BIOS contains valid certificates.";MachineName = "vproclient";ProcessID = 1964;SiteCode = "123";ThreadID = 336;};]LOG]!>

Here's the latest MEinfowin after upgrading it:

Intel(R) MEInfo Win Version: 2.5.0.1032

BIOS Version: 68MCU Ver. F.15

Intel(R) AMT code versions:

Flash: 2.6.1 Netstack: 2.6.1 Apps: 2.6.1 Intel(R) AMT: 2.6.1 Sku: 12 VendorID: 8086 Build Number: 1030 Recovery Version: 2.6.1 Recovery Build Num: 1030 Legacy Mode: False

Link status: Link down

Cryptography fuse: EnabledFlash protection: EnabledLast reset reason: Power upSetup and Configuration: Not startedBIOS Mode: Post BootDedicated Mac Address: 00-16-d4-f0-f2-59Host Mac Address: 00-16-d4-f0-f2-58FWU Override Counter: NeverFWU Override Qualifier: AlwaysFW on Flash Desc Override: DisableKedron Driver Version: 12.1.0.14Kedron HW Version: 0.0.40UNS Version: 2.6.8.1030LMS Version: 2.6.11.1030HECI Version: 2.6.30.1014

Trevor Sullivan

Systems Engineer

OfficeMax Corporation

[idata]

Consider this Powershell output from my Windows Vista Enterprise SP1-based Dell Latitude D630C running AMT 2.6.2:

PS C:\Users\User> $Prvs = [wmiclass]"root\ccm:CCM_PrvsMethodProvider"

PS C:\Users\User> $Prvs.CheckCertificate("test")__GENUS : 2__CLASS : __PARAMETERS__SUPERCLASS :__DYNASTY : __PARAMETERS__RELPATH :__PROPERTY_COUNT : 4__DERIVATION : {}__SERVER :__NAMESPACE :__PATH :dvcCert : testErrorCode : 1ErrorMsg : None certificate is valid between device and server certific ate hash.ReturnValue : 1

.... and this output from the HP Compaq 6910p running AMT 2.6.1:

PS C:\Users\i081225> $Prvs = [wmiclass]"root\ccm:CCM_PrvsMethodProvider"

PS C:\Users\i081225> $Prvs.CheckCertificate("test")__GENUS : 2__CLASS : __PARAMETERS__SUPERCLASS :__DYNASTY : __PARAMETERS__RELPATH :__PROPERTY_COUNT : 4__DERIVATION : {}__SERVER :__NAMESPACE :__PATH :dvcCert : testErrorCode : 1ErrorMsg : Failed to enumerate certificate hash, please check if the BI  OS contains valid certificates.ReturnValue : 1

Trevor Sullivan

Systems Engineer

OfficeMax Corporation

[idata]

And then again (on the Latitude D630C), when provided with a correct / valid / trusted root certificate hash:

PS C:\Users\i081225> $Prvs.CheckCertificate("74 2c 31 92 e6 07 e4 24 eb 45 49 54

2b e1 bb c5 3e 61 74 e2".Replace(" ",""))__GENUS : 2__CLASS : __PARAMETERS__SUPERCLASS :__DYNASTY : __PARAMETERS__RELPATH :__PROPERTY_COUNT : 4__DERIVATION : {}__SERVER :__NAMESPACE :__PATH :dvcCert : 742c3192e607e424eb4549542be1bbc53e6174e2ErrorCode : 0ErrorMsg :ReturnValue : 0

Trevor Sullivan

Systems Engineer

OfficeMax Corporation

[idata]

Well, this issue is apparently a known problem with AMT 2.5 to AMT 2.6 upgrades. The certificate hashes are actually missing from the firmware, which is causing this method on the ConfigMgr client to fail.

http://communities.intel.com/thread/2031?tstart=0 http://communities.intel.com/thread/2031?tstart=0

Here is some Powershell code you can run to enumerate embedded hashes via the HECI driver interface:

<!--[if gte mso 9]> Normal 0 false false false EN-US X-NONE X-NONE MicrosoftInternetExplorer4 <![endif]--><!--[if gte mso 9]> <w:LsdException Locked="false" Priority="69" SemiHidden="false" UnhideWhenUsed="false" Name="Medium ...

[idata]

I have the exact same issue on a Dell Optiplex 755. It was AMT firmware 3.0 upgraded to 3.2.2. We are using an internal certificate, and have keyed in the hash manually. Every time the SCCM client tries to provision the system in-band, I get the exact same error. I have tried everything I can think of, but the error remains.

I can supply log files, exact error quotes, anything necesssary. This has been driving me crazy for over a week, any suggestions are welcome.

Thanks.

[idata]

John,

Can you try running the Powershell code I posted? For what it's worth, I've upgraded a bunch of Dell Optiplex 755's from 3.0 to 3.2.1 or 3.2.2, and haven't seen this issue on them.

Let me know if it outputs anything (take a screenshot of your Powershell instance).

Trevor Sullivan

Systems Engineer

OfficeMax Corporation

[Steven_D_Intel1]

When you key the hash into the MEBX on the DELL, make sure it is the hash from the root certificate of your internal PKI and not the hash from the provisioning certificate itself

This may be why you are seeing this error

May also be worth checking that SCCM did actually accept your provisioning certificate when you configured the Out of Band Management Component. I have seen couple of examples where you specify the certificate, it appears like SCCM accepted it, and then when you look in AMTOPMGR.LOG file you find SCCM rejected it (wrong domain, missing OID or OU...)

Hope this helps

[idata]

Hi Trevor,

Thanks for responding. I have installed powershell, but I have absolutely no experience with it, can you point me to the steps required to run the command you posted?

In the meantime, here's the output of MEInfoWin

C:\ME>meinfowin

Intel(R) MEInfo Win Version: 2.5.0.1032

BIOS Version: A12

Intel(R) AMT code versions:

Flash: 3.2.2 Netstack: 3.2.2 Apps: 3.2.2 Intel(R) AMT: 3.2.2 Sku: 12 VendorID: 8086 Build Number: 1033 Recovery Version: 3.2.2 Recovery Build Num: 1033 Legacy Mode: False

Link status: Link up

Cryptography fuse: EnabledFlash protection: EnabledLast reset reason: Firmware ResetSetup and Configuration: In processBIOS Mode: Post BootDedicated Mac Address: 00-1a-a0-e6-5c-d0Host Mac Address: 00-1a-a0-e6-5c-ceFWU Override Counter: AlwaysFWU Override Qualifier: AlwaysFW on Flash Desc Override: DisableKedron Driver Version: Not AvailableKedron HW Version: Not AvailableUNS Version: 3.0.4.1089LMS Version: 3.0.10.1089HECI Version: 3.0.30.1086

This is the relevant portion of the OOBmgmt log

Raising event:

[SMS_CodePage(850), SMS_LocaleID(4105)]instance of SMS_OOBMgmt_StartConfig_Failure

ErrorCode = "1";

FailureCategory = "None certificate is valid between device and server certificate hash."; MachineName = "XXX"; ProcessID = 180; SiteCode = "AAA"; ThreadID = 4092;};

We have manually added the hash several times and have followed the technet articles on how to do so. The only thing I can think of is the name of the hash in the BIOS, we have typically left it blank, could that have any effect?

Thanks!!

John

[idata]

John,

Open a Powershell instance, and simply paste in the code I provided.

Alternatively, download the free http://www.powergui.org Quest PowerGUI tool, then save the code to a .ps1 file, and hit the "Play" button to execute it.

Trevor Sullivan

Systems Engineer

OfficeMax Corporation

[idata]

Sdavies, Thanks for responding.

Originally we made the mistake of keying in the hash from the provisionsing certificate, but a few days ago I discovered that and we them began entering the hash from the root CA instead. In the amtpopmgr log, I get entries like this:

Found new provision server certificate with hash

Get ROOT HASH of provision server

Is the ROOT HASH value the same as the one that should be in the BIOS of the client PC?

I also get an error whenever I try to discover OOB controllers on the collection:

AMT Discovery Worker: Wakes up to process instruction files SMS_AMT_OPERATION_MANAGER 2/27/2009 2:03:16 PM 10664 (0x29A8)

CAMTDiscoveryWSMan::DoDetectAMTVersion: recv failed: 10054 SMS_AMT_OPERATION_MANAGER 2/27/2009 2:03:16 PM 11768 (0x2DF8)Server unexpectedly disconnected when TLS handshaking. SMS_AMT_OPERATION_MANAGER 2/27/2009 2:03:16 PM 11768 (0x2DF8)**** Error 0x3b2b904 returned by ApplyControlToken SMS_AMT_OPERATION_MANAGER 2/27/2009 2:03:16 PM 11768 (0x2DF8)Server unexpectedly disconnected when TLS handshaking. SMS_AMT_OPERATION_MANAGER 2/27/2009 2:03:16 PM 11768 (0x2DF8)**** Error 0x3b2b904 returned by ApplyControlToken SMS_AMT_OPERATION_MANAGER 2/27/2009 2:03:16 PM 11768 (0x2DF8)AMT Discovery Worker: Wait 20 seconds... SMS_AMT_OPERATION_MANAGER 2/27/2009 2:03:16 PM 10664 (0x29A8)session params ::16992 , 111001 SMS_AMT_OPERATION_MANAGER 2/27/2009 2:03:16 PM 11768 (0x2DF8)ERROR: Invoke(get) failed: 80020009argNum = 0 SMS_AMT_OPERATION_MANAGER 2/27/2009 2:03:16 PM 11768 (0x2DF8)Description: The I/O operation has been aborted because of either a thread exit or an application request. SMS_AMT_OPERATION_MANAGER 2/27/2009 2:03:16 PM 11768 (0x2DF8)Error: Failed to get AMT_SetupAndConfigurationService instance. SMS_AMT_OPERATION_MANAGER 2/27/2009 2:03:16 PM 11768 (0x2DF8)session params : 16992 , 111001 SMS_AMT_OPERATION_MANAGER 2/27/2009 2:03:16 PM 11768 (0x2DF8)ERROR: Invoke(get) failed: 80020009argNum = 0 SMS_AMT_OPERATION_MANAGER 2/27/2009 2:03:16 PM 11768 (0x2DF8)Description: The I/O operation has been aborted because of either a thread exit or an application request. SMS_AMT_OPERATION_MANAGER 2/27/2009 2:03:16 PM 11768 (0x2DF8)

I was ignoring this for the time being because I think it refers to OOB provisioning, and we're trying to get around that.

Thanks!

[idata]

Hi Trevor,

I ran the powershell command, and it comes back with a match for the hash we entered, but this has me thinking it's the wrong certificate hash we are entering. Because of the structure of this organization, I'm having difficulty getting direct access to the root CA server. Is there anyway to get the hash I'm looking for from the certificates mmc on the provisioning server? Or is it in the amtpopmgr log file as shown in the C/P in my previous post?

Thanks!

[idata]

Open the properties on your provisioning certificate, go to the certificate chain, open the root CA's certificate, go to the details tab, and find the Thumbprint.

Here is a quick demonstration: http://blip.tv/file/1823217 http://blip.tv/file/1823217

Trevor Sullivan

Systems Engineer

OfficeMax Corporation

[idata]

I did as you suggested, and as it turns out, we were entering the wrong hash. For future information, the hash we needed to use was listed in the amtpopmgr log file in the entry

Get ROOT HASH of provision server .

After entering the correct hash, the oobmgmt log on the client now shows it was provisioned succesfully. I'm going to wait to see the change on the object in the collection before I get to excited, but this defnitely looks good. I'll keep you posted and thanks to both of you for the help.

John

[idata]

John,

I'd really recommend purchasing an SSL certificate for the purposes of AMT provisioning. When you consider that you can get one from Godaddy for only $90 per year, it's really worth it to avoid having to touch each and every machine.

Trevor Sullivan

Systems Engineer

OfficeMax Corporation

[idata]

Hi Trevor,

I've already made that recommendation. Once I get this lab setup running, I'll leave it to them to decide how to proceed.

In the meantime, even though the oobmgmt log shows the client was succesfully configured, it still shows as "detected" in the collection. The amtpopmgr log still shows the same errors as I had previously posted. Am I missing something? Once it has been configured in-band, should it not automatically change to provisioned after the system has had a chance to process it? The collection is set to automatically provision computers, but as this one is still just detected, I assume it hasn't tried to provision it yet.

Any suggestions are greatly appreciated.

John

[idata]

John,

I'd recommend installing http://support.microsoft.com/kb/960804 Microsoft KB 960804, which is an update rollup for out-of-band management in ConfigMgr. See if this has any effect on the issue you're having.

FYI, sometimes you have to do "Update Collection Membership" (or even run a separate Management Controller discovery task), instead of simply refreshing the collection in order to get the AMT provisioning status to change.

If you have an AMT client that's stuck as "Detected" I believe it's possible that ConfigMgr may have provisioned it in the past, but the resource was deleted from the database, and now it can't connect to it again. I believe that ConfigMgr uses digest authentication using a randomly generated password in its database, that is associated with the respective AMT client. If you delete the resource record for a provisioned AMT client, it will remain provisioned, but ConfigMgr won't be able to manage it any longer. Don't quote me on any of this though, because it's only based on what I have lightly observed over time.

Trevor Sullivan

Systems Engineer

OfficeMax Corporation

[idata]

I'll look into installing that rollup. In the meantime, I have updated and refreshed several times, and the client still shows as detected. During the course of troubleshooting, we changed the admin password in the oob component configuration, could that have any effect? We changed the password on the client machine as well. I'd change it back but unfortunately, the SCCM console enforces a strong password for that account, so it wont accept the default of "admin" anymore.

John

[idata]

John,

If you changed the MEBx password manually on an AMT client, then you need to specify that password under the Provisioning Accounts tab of the OOB Component Configuration screen. During a provisioning attempt, ConfigMgr first tries any passwords specified in that area, then it tries the default password of "admin".

It's possible that this is the reason you are seeing Detected instead of Not Provisioned. If ConfigMgr can see an AMT device responding during a management controller discovery, but is unable to authenticate to it with the default remote MEBx password, then logically, it would probably show Detected. This isn't an exact scenario I've tested though, so don't quote me

-----------------

The reason that I recommended purchasing an SSL certificate earlier, was because this would allow you to reset the MEBx back to factory defaults (pull the CMOS battery and disconnect power), and not ever have to login to the MEBx locally (to enter your internal root CA hash). This is more or less a "best practice" because it takes the entire issue of a changed MEBx password completely out of the equation.

Trevor Sullivan

Systems Engineer

OfficeMax Corporation