Intel vPro® Platform
Intel Manageability Forum for Intel® EMA, AMT, SCS & Manageability Commander
2827 Discussions

Certificate Error Provisioning All AMT Devices

idata
Employee
4,732 Views

Hey guys,

I have a brand new Dell Optiplex 755 running BIOS A11 and AMT Firmware 3.2.1. I'm having trouble provisioning it. Everything works up until the certificate request is made from out certificate server, however. I'm getting the below messages in the amtproxymgr.log (not amtopmgr.log) on the ConfigMgr site server.

I had one of the guys on our server team check out the certificate server, and it is creating multiple certificates for the same client, and automatically approving them (as is proper), but for some reason, the site server is rejecting the certificate during the verification of the certificate chain. Our internal root CA certificate is in the Trusted Root CA store on the site server, and I have successfully provisioned other clients before.

I have also verified that this is not the self-signed certificate issue, because I have manually unprovisioned the device in SMB mode, and also pulled the CMOS battery to reset back to factory defaults. The same behavior is persisting.

DNS also is not a problem, as I have verified the forward and reverse records for the client from the site server. DHCP option 15 is also set properly. If either of these were the issue, we wouldn't be getting as far as we are in the provisioning process.

Found instruction file: D:\SMS\inboxes\amtproxymgr.box\{50830F19-8E2D-410A-A75B-EC5F0A32F96E}.apx

 

Processing Instruction: RCT 1;1;62151;3.2.1;vproclient.vprodemo.com;SMS_AMT_OPERATION_MANAGER_PROV;

 

Request certificate task begin to read Site Control File.

 

Changes to the site control file settings detected.

 

Request certificate task success to read parameters from Site Control File.

 

Request certificate task success to connect to the SQL database.

 

ERROR: CertCreateCertificateContext failed: 0x80093102, msg=ASN1 unexpected end of data.~

 

Error: CTaskRequestClientCert::RevokeExistedCertificate failed to get serial number from the certificate binary.

 

Request certificate task disConnected to the SQL database.

 

INFO: Enter process request 1

 

INFO: Save Request

 

INFO: Add new request

 

Certificate for vproclient.vprodemo.com has been retrieved.

 

ERROR: CertGetCertificateChain(...) failed: 0x1000040

 

ERROR: HandleDisposition failed: the root certificate of the CA is not at the Trust List!

 

INFO: Enter process request 3

 

INFO: Delete Request

 

INFO: Request to delete found

 

STATMSG: ID=7601 SEV=E LEV=M SOURCE="SMS Server" COMP="SMS_AMT_PROXY_COMPONENT" SYS=PROVSERVER SITE=123 PID=8536 TID=2220 GMTDATE=Thu Jan 08 21:28:22.411 2009 ISTR0="vproclient.vprodemo.com" ISTR1="certserver.vprodemo.com" ISTR2="" ISTR3="" ISTR4="" ISTR5="" ISTR6="" ISTR7="" ISTR8="" ISTR9="" NUMATTRS=0

 

Failed to run instruction: RCT 1;1;62151;3.2.1;vproclient.vprodemo.com;SMS_AMT_OPERATION_MANAGER_PROV;

 

Finished Executing Instruction: RCT 1;1;62151;3.2.1;vproclient.vprodemo.com;SMS_AMT_OPERATION_MANAGER_PROV;

 

Thanks,

Trevor Sullivan

Systems Engineer

OfficeMax Corporation

0 Kudos
20 Replies
idata
Employee
1,629 Views

I am experiencing this same issue on a HP Compaq dc7900 running AMT 5.0.2. I'd appreciate some feedback on this problem ... any ideas on where it's failing? The certificate authority apperas to be functioning properly.

Which certificate is not in which trust list? I've verified that my internal root and subordinate CA certificates are in their proper locations on the site server. I've also verified that the proper Verisign root and subordinate CA certificates are in their proper locations.

Trevor Sullivan

Systems Engineer

OfficeMax Corporation

0 Kudos
idata
Employee
1,629 Views

Here is what the amtopmgr.log file is showing:

>>>>>>>>>>>>>>>Provision task begin<<<<<<<<<<<<<<<<p> 

Provision target is indicated with SMS resource id. (MachineId = 62378 vproclient.vprodemo.com)

 

AMT Provision Worker: 1 task(s) are sent to the task pool successfully.~

 

AMT Provision Worker: Wait 20 seconds...

 

Found valid basic machine property for machine id = 62378.

 

Warning: Currently we don't support mutual auth. Change to TLS server auth mode.

 

The provision mode for device vproclient.vprodemo.com is 1.

 

Attempting to establish connection with target device using SOAP.

 

Found matched certificate hash in current memory of provisioning certificate

 

Create provisionHelper with (Hash: 0CE62E1E26D22E86F2C31BB6D95471C968C9903B)

 

Set credential on provisionHelper…

 

Try to use provisioning account to connect target machine vproclient.vprodemo.com...

 

HTTP digest authentication failed with status = 401.~

 

Fail to connect and get core version of machine vproclient.vprodemo.com using provisioning account # 0.

 

Try to use default factory account to connect target machine vproclient.vprodemo.com...

 

Succeed to connect target machine vproclient.vprodemo.com and core version with 5.0.2 using default factory account.

 

GeneralInfo.GetProvisioningState finished with HResult = 0x0, status = 0x0, clientError = 0.~

 

Get device provisioning state is In Provisioning

 

Passed OTP check on AMT device vproclient.vprodemo.com.

 

Machine vproclient.vprodemo.com will be added and published to AD and OU is LDAP://.

 

Send request to AMT proxy component to add machine vproclient.vprodemo.com to AD.

 

Successfully created instruction file for AMT proxy task: D:\SMS\inboxes\amtproxymgr.box~

 

Processing provision on AMT device vproclient.vprodemo.com

 

Send request to AMT proxy component to generate client certificate. (MachineId = 62378)

 

Successfully created instruction file for AMT proxy task: D:\SMS\inboxes\amtproxymgr.box~

 

Wait 20 seconds to find client certificate for AMT device vproclient.vprodemo.com being generated again…

 

Auto-worker Thread Pool: Current size of the thread pool is 1

 

AMT Provision Worker: Wakes up to process instruction files

 

AMT Provision Worker: Wait 20 seconds…

 

RETRY(1) - Validate client certificate for AMT device vproclient.vprodemo.com being generated.

 

Wait 20 seconds to find client certificate for AMT device vproclient.vprodemo.com being generated again...

 

AMT Provision Worker: Wakes up to process instruction files

 

AMT Provision Worker: Wait 20 seconds...

 

RETRY(2) - Validate client certificate for AMT device vproclient.vprodemo.com being generated.

 

Wait 20 seconds to find client certificate for AMT device vproclient.vprodemo.com being generated again...

 

AMT Provision Worker: Wakes up to process instruction files

 

AMT Provision Worker: Wait 20 seconds…

 

RETRY(3) - Validate client certificate for AMT device vproclient.vprodemo.com being generated.

 

Wait 20 seconds to find client certificate for AMT device vproclient.vprodemo.com being generated again…

 

AMT Provision Worker: Wakes up to process instruction files

 

AMT Provision Worker: Wait 20 seconds...

 

RETRY(4) - Validate client certificate for AMT device vproclient.vprodemo.com being generated.

 

Wait 20 seconds to find client certificate for AMT device vproclient.vprodemo.com being generated again…

 

AMT Provision Worker: Wakes up to process instruction files

 

AMT Provision Worker: Wait 20 seconds...

 

RETRY(5) - Validate client certificate for AMT device vproclient.vprodemo.com being generated.

 

Error: Missed device certificate. To provision device with TLS server or Mutual authentication mode, device certficate is required. (MachineId = 62378)

 

Error: Can't finish provision on AMT device vproclient.vprodemo.com with configuration code (0)!

 

>>>>>>>>>>>>>>>Provision task end<<<<<<<<<<<<<<<</span>

Trevor Sullivan

Systems Engineer

OfficeMax Corporation

0 Kudos
idata
Employee
1,629 Views

Another piece of information: I've disabled support for the Intel WS-MAN Translator v1.1 Build 552, which is installed on the same site server. While I'm trying to figure this problem out, I want to reduce the number of variables present.

Trevor Sullivan

Systems Engineer

OfficeMax Corporation

0 Kudos
Tony_C_Intel
Employee
1,629 Views

Hi Trevor,

I ran into a similar issue a while back, although I don't quite remember if it was with an HP dc7800 or a Dell Optiplex 755. I, too, saw the multiple "validate certificate for AMT device" errors and I noticed that the certificates generated for the AMT client were quickly revoked with every RETRY message being logged. I was reusing the client name and when I previously performed a full-unprovision, the AD object was not deleted from the AMT OU container. Shortly after I manually deleted the stale object from AMT OU, provisioning went through successfully. Might be worth to take a look to see if the same object already exists in AMT OU or not.

0 Kudos
idata
Employee
1,629 Views

Tony,

Thanks for the idea. I think my machine is hosed up at the moment, because it completed first-stage provisioning, so I'll have to go reset the CMOS again tomorrow morning, when I get back into the office. Good thinking though!

Trevor Sullivan

Systems Engineer

OfficeMax Corporation

0 Kudos
William_Y_Intel
Employee
1,629 Views

Trevor,

Remember that a CMOS clear re-surfaces that self-signed cert issue. Make sure after you CMOS clear the system, you go through one of the methods to clear the self-signed cert problem that exist on AMT FW <3.2.2. I would clear everything (OU, Certs, etc) and try to re-establish provisioning. Are you working with only on root CA? Have you validated that single root CA is in the trusted root store of the local SCCM system? You might try to remove and re-add as well. From the logs ou posted, it appears that it can't validate the root CA for the cert being generated.

0 Kudos
idata
Employee
1,629 Views

Bill,

I didn't realize that resetting the CMOS reset the self-signed certificate issue. I will try that again.

I have a root CA and a subordinate CA internally. As I stated earlier, I verified that the root and subordinate CA certificates are in their respective containers in the computer's certificate store. I could delete them and re-add them I suppose, but the certificates both show as valid (no red X), if I open them up.

I agree that it looks like it can't validate the root CA, but I just have a hard time understanding why, considering the root CA certificate doesn't expire for a few years at least ... 2012 I think. I don't know why it would suddenly just stop working. Could the certificate configuration in the translator have impacted those somehow? I suppose I could peruse through the source code .... but I really don't feel like it.

Trevor Sullivan

Systems Engineer

OfficeMax Corporation

0 Kudos
William_Y_Intel
Employee
1,629 Views

Ah, that is another piece of the puzzle. If you have a root and subordinate CA, there is a known issue in SCCM that has trouble validating the chain (subordinate in the intermediate store and the Root CA in the Trusted Root Store). The work-around for this issue is to load both the Root CA and the Subordinate CA into the trusted root store (local computer store). For some reason, SCCM does not look into the intermediate store as it should. We have reported this issue to Microsoft. Give that a try (after manually fixing the self signed certificate) and see if provisioning will succeed.

0 Kudos
idata
Employee
1,629 Views

Actually, I'm pretty sure the subordinate CA certificate is already in the computer's Trusted Root CA store, but I will definitely verify that. I did actually run into that issue (in another thread) when trying to use a Windows Vista or Windows 7 system to establish a Serial-over-LAN session.

I'll post back with more information tomorrow.

Trevor Sullivan

Systems Engineer

OfficeMax Corporation

0 Kudos
idata
Employee
1,629 Views

I'm having this issue across the board. I'm working with two separate AMT 5.0.x systems, and they are both failing to provision with these same error messages. I re-imported the subordinate and the root CA certificates, and it is still failing.

The certificate chain can't be validated.

Trevor Sullivan

Systems Engineer

OfficeMax Corporation

0 Kudos
Terry_C_Intel
Employee
1,629 Views

Have you reviewed the errors\explanations at http://technet.microsoft.com/en-us/library/cc161803.aspx?

You mention that the setup\process was working fine previously, yet not now. Just to validate - no changes made to server\infrastructructure?

In the logs - references to "vproclient.vprodemo.com" - and the TLS certificates being issued to this FQDN? Are you working in a production or test environment? (vprodemo.com is a test\demonstration environment used by Intel)

0 Kudos
idata
Employee
1,629 Views

Terry,

I don't recall exactly when the problem started occurring, but I believe that it was before the holidays, and after I had installed the WS-MAN Translator. I don't specifically recall ever getting a successful provision after installing the Translator.

Yes, I have reviewed the documentation you referenced. I don't believe it to be any of those issues. My schannel.dll is the correct version.

Trevor Sullivan

Systems Engineer

OfficeMax Corporation

0 Kudos
Terry_C_Intel
Employee
1,629 Views

Is this a test or production environment?

In the log - specific client noted is vproclient.vprodemo.com. You also noted "I had one of the guys on our server team check out the certificate server, and it is creating multiple certificates for the same client"

I see the 5 retries to obtain the certificate which you state was created. To clarify - this didn't happen until the Translator was added on?

Will you run the AMTSCAN tool and provide a summary of the data captured about the client(s) having issues? Information on the AMTSCAN tool, including link for download, is available at http://communities.intel.com/docs/DOC-2062.

Were any other systems provisioned with this particular setup, before the issue arose? Are those client systems responding to subsequent commands?

Has the server been restarted before\after the issue started to occur? Have you tried restarting the services in the following order:

  • Stop Translator
  • Stop IIS
  • Start IIS
  • Start Translator

Once service restarts completed - any reported errors on service starts? What about the amtopmgr.log?

0 Kudos
idata
Employee
1,629 Views

Terry,

Sorry for the incomplete response.

This is a production Configuration Manager infrastructure. The reason you are seeing vproclient.vprodemo.com, is because I scrubbed the log files for company-specific information that would be undesirable to disclose.

The 5 certificate retries are occurring as a result of the failure to validate the certificate chain in the amtproxymgr.log file. The same behavior (in amtopmgr.log) occurs when the CA is configured to not automatically approve certificate requests. This is not the case however, as I verified with our server team, that the CA is automatically approving certificate requests. And yes, I don't believe that this behavior started occurring until after the Translator was installed ... I cannot confirm this 100% though, because my ConfigMgr log files have rotated at least several times since then.

I do have a system (Dell OptiPlex 755, iAMT 3.2.1, BIOS A11) that was provisioned by the same site server that I'm currently experiencing the problem with, yes. The system works fine, and I can control it through the Microsoft OOB console, as well as the reference tools included with Intel AMT DTK. Because of this, it is inherent that kerberos authentication is working as expected.

---------------------------------------------------------

I still have two follow-up items:

  1. Let me get back to you with the output from the iAMT executable that you referenced.
  2. I will also investigate restarting the IIS service, but as I said above, the Intel WS-MAN Translator integration is disabled from within Configuration Manager.

     

Trevor Sullivan

Systems Engineer

OfficeMax Corporation

0 Kudos
idata
Employee
1,629 Views

We're getting an error on our subordinate certificate authority logged very frequently (probably for each provisioning attempt).

The "Windows default" Exit Module "Notify" method returned an error. The requested property value is empty. The returned status code is 0x80094004 (-2146877436). The Certification Authority was unable to send an email notification for EXITEVENT_CERTISSUED to ???.

Trevor Sullivan

Systems Engineer

OfficeMax Corporation

0 Kudos
idata
Employee
1,629 Views

I just found a thread over on the Microsoft Technet forums from October 2008 by some guy named Matt Royer It sounds like he knows what he's talking about.

http://social.technet.microsoft.com/Forums/en-US/configmgrgeneral/thread/91580f69-2007-4070-bf89-99c4d7d120ef/ http://social.technet.microsoft.com/Forums/en-US/configmgrgeneral/thread/91580f69-2007-4070-bf89-99c4d7d120ef/

Matt, could you possibly expand on what your issue was back then? What exactly did you mean by an "expired CRL"?

The errors that you and I have experienced are slightly different, but it appears that there may be an issue related to our subordinate CA configuration somehow.

Trevor Sullivan

Systems Engineer

OfficeMax Corporation

0 Kudos
Matthew_R_Intel
Employee
1,629 Views

For this issues...

Error: CTaskRequestClientCert::RevokeExistedCertificate failed to get serial number from the certificate binary.

... the CRL or http://en.wikipedia.org/wiki/Certificate_revocation_list Certificate Revocation List was expired on the Subordinate/Issuing CA.

I would take a look at the following TechNet Articles.

--Matt Royer

0 Kudos
idata
Employee
1,629 Views

Matt,

I actually had the server team check this out, and our CRL isn't expired (still not sure what that means).

I opened a ticket with Microsoft earlier today on this issue.

Trevor Sullivan

Systems Engineer

OfficeMax Corporation

0 Kudos
idata
Employee
1,629 Views

FYI, this is still an issue. I could use some recommendations ...

Trevor Sullivan

Systems Engineer

OfficeMax Corporation

0 Kudos
idata
Employee
1,515 Views

So, the issue was related to an expired CRL on the subordinate CA. There are two locations on the subordinate CA that the CRL is stored, and one of them was out of date. The CRL is stored in c:\inetpub\wwwroot\certsrv (I think), and also c:\windows\system32\certsrv\certenroll. The copy of the CRL in the former location was correct, but for some reason, the CRL was being pulled from the System32 location. This was validated by using the command: certutil -urlfetch -verify vProClientCert.cer.

I've attached two log files with the output from the certutil command, before and after fixing the problem. In the badlog.txt file, you'll see a lot more errors about failing the revocation check than in the goodlog.txt.

Trevor Sullivan

Systems Engineer

OfficeMax Corporation

0 Kudos
Reply