1 2 Previous Next 20 Replies Latest reply: Feb 16, 2009 6:14 AM by Trevor.Sullivan

Certificate Error Provisioning All AMT Devices

Trevor.Sullivan Jan 14, 2009 5:33 PM

Hey guys,

I have a brand new Dell Optiplex 755 running BIOS A11 and AMT Firmware 3.2.1. I'm having trouble provisioning it. Everything works up until the certificate request is made from out certificate server, however. I'm getting the below messages in the amtproxymgr.log (not amtopmgr.log) on the ConfigMgr site server.

I had one of the guys on our server team check out the certificate server, and it is creating multiple certificates for the same client, and automatically approving them (as is proper), but for some reason, the site server is rejecting the certificate during the verification of the certificate chain. Our internal root CA certificate is in the Trusted Root CA store on the site server, and I have successfully provisioned other clients before.

I have also verified that this is not the self-signed certificate issue, because I have manually unprovisioned the device in SMB mode, and also pulled the CMOS battery to reset back to factory defaults. The same behavior is persisting.

DNS also is not a problem, as I have verified the forward and reverse records for the client from the site server. DHCP option 15 is also set properly. If either of these were the issue, we wouldn't be getting as far as we are in the provisioning process.

Found instruction file: D:\SMS\inboxes\amtproxymgr.box\{50830F19-8E2D-410A-A75B-EC5F0A32F96E}.apx
Processing Instruction: RCT 1;1;62151;3.2.1;vproclient.vprodemo.com;SMS_AMT_OPERATION_MANAGER_PROV;
Request certificate task begin to read Site Control File.
Changes to the site control file settings detected.
Request certificate task success to read parameters from Site Control File.
Request certificate task success to connect to the SQL database.
ERROR: CertCreateCertificateContext failed: 0x80093102, msg=ASN1 unexpected end of data.~
Error: CTaskRequestClientCert::RevokeExistedCertificate failed to get serial number from the certificate binary.
Request certificate task disConnected to the SQL database.
INFO: Enter process request 1
INFO: Save Request
INFO: Add new request
Certificate for vproclient.vprodemo.com has been retrieved.
ERROR: CertGetCertificateChain(...) failed: 0x1000040
ERROR: HandleDisposition failed: the root certificate of the CA is not at the Trust List!
INFO: Enter process request 3
INFO: Delete Request
INFO: Request to delete found
STATMSG: ID=7601 SEV=E LEV=M SOURCE="SMS Server" COMP="SMS_AMT_PROXY_COMPONENT" SYS=PROVSERVER SITE=123 PID=8536 TID=2220 GMTDATE=Thu Jan 08 21:28:22.411 2009 ISTR0="vproclient.vprodemo.com" ISTR1="certserver.vprodemo.com" ISTR2="" ISTR3="" ISTR4="" ISTR5="" ISTR6="" ISTR7="" ISTR8="" ISTR9="" NUMATTRS=0
Failed to run instruction: RCT 1;1;62151;3.2.1;vproclient.vprodemo.com;SMS_AMT_OPERATION_MANAGER_PROV;
Finished Executing Instruction: RCT 1;1;62151;3.2.1;vproclient.vprodemo.com;SMS_AMT_OPERATION_MANAGER_PROV;

Thanks,

Trevor Sullivan

Systems Engineer

OfficeMax Corporation

1. Re: Error Provisioning Dell Optiplex 755

Trevor.Sullivan Jan 12, 2009 7:57 AM (in response to Trevor.Sullivan)

I am experiencing this same issue on a HP Compaq dc7900 running AMT 5.0.2. I'd appreciate some feedback on this problem ... any ideas on where it's failing? The certificate authority apperas to be functioning properly.

Which certificate is not in which trust list? I've verified that my internal root and subordinate CA certificates are in their proper locations on the site server. I've also verified that the proper Verisign root and subordinate CA certificates are in their proper locations.

Trevor Sullivan
Systems Engineer
OfficeMax Corporation
Like Show 0 Likes (0)

Actions
2. Re: Error Provisioning Dell Optiplex 755

Trevor.Sullivan Jan 12, 2009 7:56 AM (in response to Trevor.Sullivan)

Here is what the amtopmgr.log file is showing:

>>>>>>>>>>>>>>>Provision task begin<<<<<<<<<<<<<<<
Provision target is indicated with SMS resource id. (MachineId = 62378 vproclient.vprodemo.com)
AMT Provision Worker: 1 task(s) are sent to the task pool successfully.~
AMT Provision Worker: Wait 20 seconds...
Found valid basic machine property for machine id = 62378.
Warning: Currently we don't support mutual auth. Change to TLS server auth mode.
The provision mode for device vproclient.vprodemo.com is 1.
Attempting to establish connection with target device using SOAP.
Found matched certificate hash in current memory of provisioning certificate
Create provisionHelper with (Hash: 0CE62E1E26D22E86F2C31BB6D95471C968C9903B)
Set credential on provisionHelper…
Try to use provisioning account to connect target machine vproclient.vprodemo.com...
HTTP digest authentication failed with status = 401.~
Fail to connect and get core version of machine vproclient.vprodemo.com using provisioning account #0.
Try to use default factory account to connect target machine vproclient.vprodemo.com...
Succeed to connect target machine vproclient.vprodemo.com and core version with 5.0.2 using default factory account.
GeneralInfo.GetProvisioningState finished with HResult = 0x0, status = 0x0, clientError = 0.~
Get device provisioning state is In Provisioning
Passed OTP check on AMT device vproclient.vprodemo.com.
Machine vproclient.vprodemo.com will be added and published to AD and OU is LDAP://<LDAPPath>.
Send request to AMT proxy component to add machine vproclient.vprodemo.com to AD.
Successfully created instruction file for AMT proxy task: D:\SMS\inboxes\amtproxymgr.box~
Processing provision on AMT device vproclient.vprodemo.com…
Send request to AMT proxy component to generate client certificate. (MachineId = 62378)
Successfully created instruction file for AMT proxy task: D:\SMS\inboxes\amtproxymgr.box~
Wait 20 seconds to find client certificate for AMT device vproclient.vprodemo.com being generated again…
Auto-worker Thread Pool: Current size of the thread pool is 1
AMT Provision Worker: Wakes up to process instruction files
AMT Provision Worker: Wait 20 seconds…
RETRY(1) - Validate client certificate for AMT device vproclient.vprodemo.com being generated.
Wait 20 seconds to find client certificate for AMT device vproclient.vprodemo.com being generated again...
AMT Provision Worker: Wakes up to process instruction files
AMT Provision Worker: Wait 20 seconds...
RETRY(2) - Validate client certificate for AMT device vproclient.vprodemo.com being generated.
Wait 20 seconds to find client certificate for AMT device vproclient.vprodemo.com being generated again...
AMT Provision Worker: Wakes up to process instruction files
AMT Provision Worker: Wait 20 seconds…
RETRY(3) - Validate client certificate for AMT device vproclient.vprodemo.com being generated.
Wait 20 seconds to find client certificate for AMT device vproclient.vprodemo.com being generated again…
AMT Provision Worker: Wakes up to process instruction files
AMT Provision Worker: Wait 20 seconds...
RETRY(4) - Validate client certificate for AMT device vproclient.vprodemo.com being generated.
Wait 20 seconds to find client certificate for AMT device vproclient.vprodemo.com being generated again…
AMT Provision Worker: Wakes up to process instruction files
AMT Provision Worker: Wait 20 seconds...
RETRY(5) - Validate client certificate for AMT device vproclient.vprodemo.com being generated.
Error: Missed device certificate. To provision device with TLS server or Mutual authentication mode, device certficate is required. (MachineId = 62378)
Error: Can't finish provision on AMT device vproclient.vprodemo.com with configuration code (0)!
>>>>>>>>>>>>>>>Provision task end<<<<<<<<<<<<<<<

Trevor Sullivan
Systems Engineer
OfficeMax Corporation
Like Show 0 Likes (0)

Actions
3. Re: Error Provisioning Dell Optiplex 755

Trevor.Sullivan Jan 12, 2009 8:05 AM (in response to Trevor.Sullivan)

Another piece of information: I've disabled support for the Intel WS-MAN Translator v1.1 Build 552, which is installed on the same site server. While I'm trying to figure this problem out, I want to reduce the number of variables present.

Trevor Sullivan
Systems Engineer
OfficeMax Corporation
Like Show 0 Likes (0)

Actions
4. Re: Error Provisioning Dell Optiplex 755

TerryCutler Jan 12, 2009 9:53 AM (in response to Trevor.Sullivan)

Have you reviewed the errors\explanations at http://technet.microsoft.com/en-us/library/cc161803.aspx?

You mention that the setup\process was working fine previously, yet not now. Just to validate - no changes made to server\infrastructructure?

In the logs - references to "vproclient.vprodemo.com" - and the TLS certificates being issued to this FQDN? Are you working in a production or test environment? (vprodemo.com is a test\demonstration environment used by Intel)
Like Show 0 Likes (0)

Actions
5. Re: Error Provisioning Dell Optiplex 755

Trevor.Sullivan Jan 12, 2009 12:03 PM (in response to TerryCutler)

Terry,

I don't recall exactly when the problem started occurring, but I believe that it was before the holidays, and after I had installed the WS-MAN Translator. I don't specifically recall ever getting a successful provision after installing the Translator.

Yes, I have reviewed the documentation you referenced. I don't believe it to be any of those issues. My schannel.dll is the correct version.

Trevor Sullivan
Systems Engineer
OfficeMax Corporation
Like Show 0 Likes (0)

Actions
6. Re: Error Provisioning Dell Optiplex 755

TerryCutler Jan 12, 2009 12:26 PM (in response to Trevor.Sullivan)
Is this a test or production environment?

In the log - specific client noted is vproclient.vprodemo.com. You also noted "I had one of the guys on our server team check out the certificate server, and it is creating multiple certificates for the same client"

I see the 5 retries to obtain the certificate which you state was created. To clarify - this didn't happen until the Translator was added on?

Will you run the AMTSCAN tool and provide a summary of the data captured about the client(s) having issues? Information on the AMTSCAN tool, including link for download, is available at http://communities.intel.com/docs/DOC-2062.

Were any other systems provisioned with this particular setup, before the issue arose? Are those client systems responding to subsequent commands?

Has the server been restarted before\after the issue started to occur? Have you tried restarting the services in the following order:
Stop Translator
Stop IIS
Start IIS
Start Translator

Once service restarts completed - any reported errors on service starts? What about the amtopmgr.log?
Like Show 0 Likes (0)

Actions
7. Re: Error Provisioning Dell Optiplex 755

Trevor.Sullivan Jan 12, 2009 1:04 PM (in response to TerryCutler)
Terry,

Sorry for the incomplete response.

This is a production Configuration Manager infrastructure. The reason you are seeing vproclient.vprodemo.com, is because I scrubbed the log files for company-specific information that would be undesirable to disclose.

The 5 certificate retries are occurring as a result of the failure to validate the certificate chain in the amtproxymgr.log file. The same behavior (in amtopmgr.log) occurs when the CA is configured to not automatically approve certificate requests. This is not the case however, as I verified with our server team, that the CA is automatically approving certificate requests. And yes, I don't believe that this behavior started occurring until after the Translator was installed ... I cannot confirm this 100% though, because my ConfigMgr log files have rotated at least several times since then.

I do have a system (Dell OptiPlex 755, iAMT 3.2.1, BIOS A11) that was provisioned by the same site server that I'm currently experiencing the problem with, yes. The system works fine, and I can control it through the Microsoft OOB console, as well as the reference tools included with Intel AMT DTK. Because of this, it is inherent that kerberos authentication is working as expected.

---------------------------------------------------------

I still have two follow-up items:

Let me get back to you with the output from the iAMT executable that you referenced.
I will also investigate restarting the IIS service, but as I said above, the Intel WS-MAN Translator integration is disabled from within Configuration Manager.

Trevor Sullivan
Systems Engineer
OfficeMax Corporation
Like Show 0 Likes (0)

Actions
8. Re: Error Provisioning Dell Optiplex 755

tlchan Jan 12, 2009 5:25 PM (in response to Trevor.Sullivan)

Hi Trevor,

I ran into a similar issue a while back, although I don't quite remember if it was with an HP dc7800 or a Dell Optiplex 755. I, too, saw the multiple "validate certificate for AMT device" errors and I noticed that the certificates generated for the AMT client were quickly revoked with every RETRY message being logged. I was reusing the client name and when I previously performed a full-unprovision, the AD object was not deleted from the AMT OU container. Shortly after I manually deleted the stale object from AMT OU, provisioning went through successfully. Might be worth to take a look to see if the same object already exists in AMT OU or not.
Like Show 0 Likes (0)

Actions
9. Re: Error Provisioning Dell Optiplex 755

Trevor.Sullivan Jan 12, 2009 5:38 PM (in response to tlchan)

Tony,

Thanks for the idea. I think my machine is hosed up at the moment, because it completed first-stage provisioning, so I'll have to go reset the CMOS again tomorrow morning, when I get back into the office. Good thinking though!

Trevor Sullivan
Systems Engineer
OfficeMax Corporation
Like Show 0 Likes (0)

Actions
10. Re: Error Provisioning Dell Optiplex 755

wryork Jan 12, 2009 7:08 PM (in response to Trevor.Sullivan)

Trevor,
Remember that a CMOS clear re-surfaces that self-signed cert issue. Make sure after you CMOS clear the system, you go through one of the methods to clear the self-signed cert problem that exist on AMT FW <3.2.2. I would clear everything (OU, Certs, etc) and try to re-establish provisioning. Are you working with only on root CA? Have you validated that single root CA is in the trusted root store of the local SCCM system? You might try to remove and re-add as well. From the logs ou posted, it appears that it can't validate the root CA for the cert being generated.
Like Show 0 Likes (0)

Actions
11. Re: Error Provisioning Dell Optiplex 755

Trevor.Sullivan Jan 12, 2009 7:33 PM (in response to wryork)

Bill,

I didn't realize that resetting the CMOS reset the self-signed certificate issue. I will try that again.

I have a root CA and a subordinate CA internally. As I stated earlier, I verified that the root and subordinate CA certificates are in their respective containers in the computer's certificate store. I could delete them and re-add them I suppose, but the certificates both show as valid (no red X), if I open them up.

I agree that it looks like it can't validate the root CA, but I just have a hard time understanding why, considering the root CA certificate doesn't expire for a few years at least ... 2012 I think. I don't know why it would suddenly just stop working. Could the certificate configuration in the translator have impacted those somehow? I suppose I could peruse through the source code .... but I really don't feel like it.

Trevor Sullivan
Systems Engineer
OfficeMax Corporation
Like Show 0 Likes (0)

Actions
12. Re: Error Provisioning Dell Optiplex 755

wryork Jan 12, 2009 7:42 PM (in response to Trevor.Sullivan)

Ah, that is another piece of the puzzle. If you have a root and subordinate CA, there is a known issue in SCCM that has trouble validating the chain (subordinate in the intermediate store and the Root CA in the Trusted Root Store). The work-around for this issue is to load both the Root CA and the Subordinate CA into the trusted root store (local computer store). For some reason, SCCM does not look into the intermediate store as it should. We have reported this issue to Microsoft. Give that a try (after manually fixing the self signed certificate) and see if provisioning will succeed.
Like Show 0 Likes (0)

Actions
13. Re: Error Provisioning Dell Optiplex 755

Trevor.Sullivan Jan 12, 2009 9:46 PM (in response to wryork)

Actually, I'm pretty sure the subordinate CA certificate is already in the computer's Trusted Root CA store, but I will definitely verify that. I did actually run into that issue (in another thread) when trying to use a Windows Vista or Windows 7 system to establish a Serial-over-LAN session.

I'll post back with more information tomorrow.

Trevor Sullivan
Systems Engineer
OfficeMax Corporation
Like Show 0 Likes (0)

Actions
14. Re: Error Provisioning Dell Optiplex 755

Trevor.Sullivan Jan 13, 2009 11:19 AM (in response to Trevor.Sullivan)

I'm having this issue across the board. I'm working with two separate AMT 5.0.x systems, and they are both failing to provision with these same error messages. I re-imported the subordinate and the root CA certificates, and it is still failing.

The certificate chain can't be validated.

Trevor Sullivan
Systems Engineer
OfficeMax Corporation
Like Show 0 Likes (0)

Actions

1 2 Previous Next

Go to original post