For those who have Provisioned Intel AMT Systems, you may wonder what takes place in the background. This article is for you! The process has often been covered at a high level, but here the technical details are provided. Hopefully this helps you understand the inner workings, and provide you information when troubleshooting Provisioning issues. And for those of you who are technically minded, it's also neat to know! This information was compiled working on issues and running through provisioning processes from Symantec Support.

Introduction

Often the Provisioning process for Intel vPro systems has been described as complex. This comes from the fact that the Provisioning process was designed with high security in mind. Since the initial release we have improved success rates by working with Intel to make the process more user friendly without compromising the high level of security. To this end this document will explain the process of Provisioning from a technical level, providing an unfiltered view of the process, also without compromising its security.

Provisioning Flow

The following process assumes that Altiris Out of Band Management and Intel SCS are install, configured, and ready to go. This process follows the flow of Provisioning and what data points, technologies, and methods are used. The level of details is meant to be a resource when working with Provisioning or troubleshooting Provisioning issues, so not all details are available for this process. Note the following points before moving through the process:

• The console items in the Altiris Console under View > Solutions > Out of Band Management > Provisioning are not tied to the Altiris database like most of the rest of the Altiris Console. They connect through a virtual Website (AMTSCS under the Default Website of the SCS Server) to the IntelAMT database.
• Data from two databases (IntelAMT and Altiris) are used during the Provisioning process.

The following articles can assist if you need information on these:

• http://juice.altiris.com/article/1314/altiris-and-intel-reg-vpro-copy-technology-evaluators-guide
• Administrator's and Reference Guides: http://www.altiris.com/support/documentation look under Out of Band Management

1. The server is loaded with a security key or certificate. See the following two items for how these keys are loaded:
   1. For a PID PPS, either keys are randomly generated or imported into the IntelAMT database. Specifically they reside in the table csti_pid_map. Once created/imported, they are available for verifying authentication from an incoming provisioning request from AMT.
   2. For TLS-PKI (certificate-based Remote Configuration) a certificate is loaded onto the server. See this article for details: http://juice.altiris.com/article/4496/obtaining-and-applying-a-verisign-remote-configuration-certificate.
2. The clients need the matching keys loaded onto them. This is done differently depending on the type:
   1. For PID PPS the keys are set by one of the following methods: the OEM sets it, it's entered manually into the Intel ME, or inputted via a one-touch USB flash drive. The PID and PPS are written into the firmware to be used as the authentication credentials when it looks for a provisioning server.
   2. For Remote Configuration (TLS-PKI) at the factory predefined hashes are burned into the firmware for the following certificate vendors (more to come in subsequent versions of AMT). This means AMT already has authentication keys to begin the provisioning process direct from the factory.

• VeriSign
• Komodo
• GoDaddy

1. The client machine, once it has it's keys and has been connected to the network and power, uses one of two methods to find the Provisioning Server:
   1. The IP address of the server can be manually put into the Intel ME, including what port the SCS listener is configured for (default 9971). When this is done, the AMT client will transmit its Hello message directly to the IP Address and port.
   2. The client will transmit its message on port 9971 to the name of ‘ProvisionServer'. If Out of Band Management, Intel SCS, and DNS have been properly setup DNS will route the packet to the Notification Server.
2. The Notification Server is set to listen for AMT Provisioning traffic on port 9971, but can be configured to use a different port if so desired in the Altiris Console under View > Solutions > Out of Band Management > Configuration > Provisioning > Configuration Service Settings > General. The top options labeled: ‘Listen port:".|
   
3. When SCS, via the service AMTConfig (process AMTConfigWinService.exe) receives the incoming "hello" packet, it initiates an authentication request with the client to complete the authentication process, the beginning of which was stored in the packet. Once authentication completes successfully, the process moves on.
4. The service, AMTConfig, catches the incoming packet and logs the data in the IntelAMT database, in the table csti_amts. This table contains all the relevant data for this system's identity.
   
5. Once the system has been logged into the IntelAMT database, Intel SCS uses the database entries under csti_configuration to initiate what's known as the props script. This script is what will assist in the provisioning process. In Altiris case, it is oobprov.exe, located by default at C:\Program Files\Altiris\OOBSC\oobprov.exe. For an example of how Intel SCS knows about this, see this data snippet from the csti_configuration table:
   
6. On a busy SCS server you can look at Task Manager and see multiple instances of oobprov.exe running. The default settings allow 10 threads to work on provisioning requests at any given time. These threads will interface with the Altiris Database via the Altiris Agent on the local server system. In a standard setup the local system is also the Notification Server.
7. OOBPROV runs a SQL query to fetch the Fully Qualified Domain Name (FQDN) for the system it is to provision. The query is based off the following data points:
   1. UUID passed to it via Intel SCS, Source is as follows: Database: IntelAMT, Table: csti_amts, Data Source: "Hello" packet from AMT system, Values used: uuid
   2. Database: Altiris, Data-class: OOB Capability, Table: Inv_OOB_Capability, Data Source: Out of Band Discovery Task, Values used: _ResourceGuid - UUID
   3. Database: Altiris, Data-class: AeX AC Location, Table: Inv_AeX_AC_Location, Data Source: Basic Inventory Agent, whether from Basic Inventory function or Hardware Inventory from Inventory Solution, Values used: _ResourceGuid - Fully Qualified Domain Name
8. The Query accomplishes the following: It takes the UUID from csti_amts, uuid and looks for a match in Inv OOB Capability, uuid. If a match is made, it takes the _ResourceGuid from the same table and makes a match of the same columns name to AeX AC Location. With the match it then reads the values stored under Fully Qualified Domain Name (I'm not sure why they didn't just label this column FQDN...).
9. Next, oobprov.exe hands back the FQDN it's read from AeX AC Location, Fully Qualified Domain Name and passes it to SCS. SCS takes this value and inserts it into the IntelAMT database at csti_amts, fqdn for the matching resource.
10. Next, oobprov.exe fetches the automatic profile set within Out of Band Management Solution. This is done in the Altiris Console under View > Solutions > Out of Band Management > Configuration > Provisioning > Intel AMT Systems > Resource Synchronization. This policy needs to be enabled for this step to work, and a default profile configured and selected under the dropdown labeled ‘Intel AMT 2.0+ to profile:'.
11. The profile provides the operational data for management of the AMT system. After AMT accepts the profile, the Provisioning process is now complete. Before this step, AMT functionality is not available on this system, and after this step only properly authenticated functions will be able to use Intel vPro on the target provisioned systems.

Troubleshooting

The following items can be considered break points for this process. If you've done provisioning you may have run into the symptoms produced by the following items. These are compiled as common areas of trouble in this process.

• The "Hello" packets only transmit for 24 hours, on a back-off schedule, before stopping altogether. If the Server is unable to provision in that time, with IP refreshes becoming more frequent, the system can be in a limbo state. See this article for steps to rectify: http://juice.altiris.com/article/3612/using-intels-rct-tool-restart-amt-hello-packets-enterprise-provisioning
• IP Address changes, refreshes within DHCP during a system's build process can leave SCS with an out of date IP Address for a system that needs provisioning. Coupled with the preceding issue this can leave the system in an unprovisioned state, leaving no ability of the SCS to contact the system to finish the process.
• Remote Configuration certificate is not properly installed on the server, producing authentication failure messages in the AMT logs.
• Oobprov.exe is unable to fetch the FQDN. The AMT system needs the Altiris Agent installed, have sent Basic Inventory when it had a valid FQDN (for example a system in the process of being built might not have a valid FQDN yet), OOB Discovery Task downloaded and executed, and data populated into the OOB Capability data class from the task in order for oobprov.exe to be able to fetch the FQDN. Conversely you can use the option in Resource Synchronization labeled, ‘Use DNS IP resolution to find FQDN when assigning profiles'.

A good resource for troubleshooting issues can be found here:

• http://juice.altiris.com/book/3699/troubleshooting-altiris-manageability-toolkit-vpro-technology

Conclusion

Knowing the underline mechanisms can help when troubleshooting or even when planning your environment. While not all details are provided here, the most essential are.

In part 3 we covered troubleshooting common Provisioning Console issues. In part 4 we now focus on those components operating in the background during provisioning. With a functioning install and console, and when the issue appears to be server-related (In part 1 we covered troubleshooting the locale AMT system) now any issues seen must be evaluated on the server side. This article covers this process in a Problem - Cause - Solution format.

Introduction

The server components constitute a lot of ‘background' processes that support what is only seen as Altiris Console points. Much of what goes on in the background is invisible to the user save as a change in status. If setup correctly, machines simply provision. It's when they do not provision that a user should understand the server components so that proper troubleshooting can be accomplished. Note that this covers the symptoms of server-component problems. Some of the symptoms do overlap client-side issues, but in this process we are assuming we've confirmed that the client systems is functioning as expected. If you are unsure, see Part 1 of this article series.

Symptoms

The following symptoms are seen on the Server. Please note that some of the symptoms may appear to be both client and server related making it difficult to know where the issue lies. Use Part 1 in conjunction with this article if necessary in troubleshooting these issues.

• No update to Intel AMT Systems Node - At times this node can abruptly appear stagnant with no new systems coming in and no provisioning taking place
• No Systems Appearing - The Intel AMT Systems node may stay blank even after connecting systems in Setup Mode onto the Network.
• FQDN Not Acquired - Once the SCS receives a hello message, it needs to acquire the FQDN, and if this fails the machine will remain in an unprovisioned state
• No systems Provisioning - This can occur where systems show up in the system, but none of them provision
• Properties Script Failed - This is a common error to be covered separately, though many of the above symptoms end up throwing this particular error

In addition to the symptoms, the following tools were used to troubleshoot the issues to find out which particular issue afflicted the Server:

1. AMT Logs
2. OOB Trace Loggging
3. Wireshark

See Part 1 in this article series on how to use these. These will be referenced in the below items.

No update to Intel AMT Systems Node

Problem

The typical symptom is an abrupt stop to updates on this node. For example if you have a number of provisioned systems, with systems added as systems are brought up on the network, and abruptly they stop updating or being added, this is indicative of this issue.

Tools:

AMT Logs - No updates to this log occur.

Cause

AMTConfig Service - The AMTConfig service has stopped, crashed, or is in a hung state. This isn't common in version 3.0 of SCS or higher.

Resolution

Check that the AMTConfig Service is running.

• 1. Go to Services Manager under Administrative Tools.
• 2. Check the Service named AMTConfig to make sure it is running.
• 3. If the service is not running, start it. If the service is running, try restarting it just in case it's in an hung state.
• 4. Once the service is up and running again (if this is the issue) provisioning should start occurring.

No Systems Appearing

Problem

The symptom is that no machines appear in the Intel AMT Systems list when the page is refreshed over a period of time when new systems are expected. The page ties directly into the IntelAMT database to populate the systems, so if the list isn't updating on the page, the list is also not updating in the database.

Tools:

AMT Logs - I. No entries found

II. No entries found

III. Invalid PID Map error

Wireshark - II. On the client the "Hello" packet is sent, but on the server it never arrives.

Cause

The causes vary. See below for known causes for this issue:

1. I. AMTConfig Service - The AMTConfig service has stopped, crashed, or is in a hung state. This isn't common in version 3.0 of SCS or higher.
2. II. "Hello" packets - The routing of "hello" packets is not configured correctly, so clients can't reach the Provision Server.
3. III. PID rejected - The PID provided in the "Hello" packet is not contained as a valid security key in the IntelAMT database. This is only seen in the AMT Log found in the Provisioning Console under Logs, selecting the ‘Log' icon.

Resolution

See the steps to follow for the above causes.

1. I. AMTConfig Service
   • 1. See the resolution to the section No update to Intel AMT Systems Node.
2. II. "Hello" Packets
   • 1. In the Provisioning console go to the DNS Configuration node. Does the ‘Test' button allow Provisionserver to resolve back to the IP of the Notification Server?
   • 2. If yes, go to the segment of the network the client is on and try to ping the name ‘Provisionserver'. Does the IP resolve?
   • 3. If answer to either question above is NO, a CNAME record needs to be created on each DNS Server to route to the IP address of the Notification Server.
3. III. PID rejected
   • 1. In the Provisioning Console go to the Security Keys node under the Configuration Service Settings. The list of unused PID and PPS combinations are listed.
   • 2. In the IntelAMT database, within the csti_pid_map table all used and unused security keys are listed. The ones with a value ‘True' in the ‘Used' column will not show up in the console.
   • 3. Either import the keys if the OEM placed the AMT systems in TLS-PSK Setup Mode through the import button in the Security Keys page, or manually enter the PID PPS.

FQDN Not Acquired

Problem

One or more Intel AMT Systems are registering in Intel SCS, but they never show an FQDN and never move out of the ‘Unprovisioned' status. In the AMT Log often these systems show the error ‘Properties Script Failed' (note that the cause of this error can be many, and this issue is but one of them).

NOTE! If no system is provisioning the issue may not be FQDN related. See No Systems Provisioning in this article for more information.

Tools:

AMT Logs - Properties Script Failed messages

OOB Trace - Unable to locate FQDN (Fully Qualified Domain Name) entries

Cause

Intel SCS calls the Out of Band Provisioning or Properties script oobprov.exe to do a number of things. The first thing it does is obtain an FQDN for the machine needing provisioning. If it fails to obtain an FQDN Provisioning will fail and the computer will remain in an unprovisioned state until oobprov.exe can successfully locate the FQDN.

Resolution

To find the FQDN, oobprov.exe runs through a number of checks. The suggested method is to have the Altiris Agent installed and have run the OOB Discovery Task (located in the Altiris Console under View > Solutions > Out of Band Management > Configuration > Out of Band Discovery > Out of Band Discovery). This populates the Altiris database so it has both an FQDN in the AeX AC Location data class and the UUID in the Inv_OOB_Capability data class. If this data is not available, another option is to check DNS resolution as a method. In the Altiris Console look under the Resource Synchronization node, within the Intel AMT Systems folder. As shown below, this option enables oobprov.exe to use DNS IP resolution as a method.

NOTE the warning found directly below the checkbox: Warning! Using DNS for IP to FQDN resolution might lead to incorrect profile mapping. Make sure your DHCP server is configured correctly to give update the DNS server for dynamic addresses.

No systems Provisioning

Problem

Systems are added regularly to the Intel AMT Systems node, but they never provision. This includes never getting an FQDN (see the above section for more information), though the cause may not be the inability of oobprove.exe to obtain the FQDN.

Tools:

AMT Logs - Provisioning Script Failed messages

OOB Trace - No references to oobprov.exe

Cause

If not an FQDN mapping issue, this issue stems from a timeout value in the IntelAMT database being set to 0. In the IntelAMT database, in the table csti_configuration, under the column Props_script_timeout if the value is 0 IntelSCS will timeout before it even has a chance to call oobprov.exe.

Resolution

Normally only one row exists in this table. The following SQL query will properly update this value to the default level. The default is 180 and should be set.

USE IntelAMT

UPDATE csti_configuration

SET props_script_timeout = 180

WHERE use_props_script = 'True'

Execute the script within SQL Query Analyzer or SQL Enterprise Studio to update the value.

Properties Script Failed

Problem

This message can mean a number of things, including the symptoms described in the preceding two section. This message can continually appear into the AMT logs as provisioning is attempted over and over.

Cause

The causes of this issue vary. The basic explanation is that when oobprov.exe is called, if it returns anything other than success, the resulting error message in the AMT logs is ‘Properties Script Failed'.

Resolution

See the above two sections for the symptoms No Systems Provisioning and FQDN Not Acquired, but for additional information see the following article:

• http://juice.altiris.com/article/2982/troubleshooting-properties-script-failed-out-band-management-solution

Conclusion

This concludes the troubleshooting section for the Provisioning process. For the most common issues, the resolutions and steps presented in the first four parts of this series will resolve them. I also hope the methodology here helps explain how the background processes are working. In the next parts of this series we'll cover troubleshooting issues with the management components after systems have been successfully provisioned.