From iCSO SW Engineering – Intel® vPro™ Packet Decoder – The First of Many
Posted by Timothy Duncan Jan 28, 2008
Part 3: Handling common provisioning, configuration, and Intel SCS errors
Posted by Terry Cutler Oct 24, 2007This is the third and final part of this series (at least for now). The previous two posts include
Basicsand
Common Intel SCS errors
BEFORE GOING ANY FURTHER - PLEASE READ AND ENSURE THE FOLLOWING
At this point, you have ensured the infrastructure is setup correctly and have attempted to troubleshoot the common Intel SCS errors as listed in the SCSconsole log file. Intel vPro systems are being recognized and listed in the SCSconsole. However, strange or unexpected behavior continues to occur - whether during provisioning, maintenance, or other activities. If Intel SCS has been included in a system management console or a provisioning script provider with whom you are working - AND - further debug analysis is needed, the following points may help. The debug log output may be one of the datapoints requested to replicate and remediate issues.
Before we go on - please note that these steps require modifications to the Microsoft Windows Registry on the system labeled as "ProvisionServer". That system will be running the AMTconfig service. Enabling the debug logging features will require root drive access and space to capture and store the log outputs. The logs will be stored at the root of C:
Ready to create an Intel SCS debug log?
SCS debug logging is off by default. If enabling for troubleshooting purposes, be sure to disable when done troubleshooting. The following steps will require a new registry key and string value to be added. Once these changes have been made - restart the AMTconfig service. At most, two log files will appear on the root of c: drive. The first is scs_win_server.log the second is scs_server.log. The second commonly appears only after errors have occurred.
Create the following registry key on the service's machine:
HKEY_LOCAL_MACHINE\SOFTWARE\INTEL\AMTConfServer\Log with string value "LogLevel"="V"
Click on the following image to view the entire image
Logging levels can be set to 'V' for verbose, 'W' for warnings and errors, 'E' for only errors.
Once the debug log capture is complete, remove the LogLevel entry from the registry and restart the AMTconfig service.
This concludes the three part Intel SCS troubleshooting. If the community is experiencing additional events or has additional questions - please comment\reply.
Part 2: Handling common provisioning, configuration, and Intel SCS errors
Posted by Terry Cutler Oct 24, 2007
Part 1: Handling common provisioning, configuration, and Intel SCS errors
Posted by Terry Cutler Oct 24, 2007This blog is divided into 3 sections - understanding the basics,
addressing common Intel SCS errors, and
how to generate an Intel SCS debug log.
If only solutions were perfect, errors resolved automatically, and tuning was never required nor needed. Then again, that's what many of us get paid to do and handle. The intent here is to focus on common Intel® vPro™ configuration and provisioning errors with Intel Setup and Configuration Services (SCS). More importantly, the article intent is to provide some insight on the correction needed or tasks to handle common errors.
The Basics
Deploying Intel® vPro™ enabled solutions presents many working parts. In a lab environment - these "always" work well. In a production environment, determining the cause of an error could be difficult. Generally speaking, to isolate the scenario take into consideration the management console, the vPro configuration services (e.g. Intel® SCS), the OEM firmware and drivers, and the infrastructure. The lab environment comes in handy to isolate components and aspects, especially when so many variables are present.
In stepping through each item, consider the following basic points:
OEM hardware and drivers - Check the update page for the latest BIOS and Firmware on the platform. The BIOS update will often include the Intel® AMT firmware. The drivers to be checked are mentioned Management Engine Interface (MEI), Local Management Service (LMS), Serial over LAN (SoL), and User Notification Service (UNS). NOTE: UNS applies to AMT 3.0 and higher versions.
Intel® SCS version - Don't know what version if running? Check the AMTconfig service properties or version listed in the SCSconsole. More on SCS and AMT versions
here. Version 3.2 is the latest. If running version 1.x, an update to version 3.x is recommended. Check first with preferred system management vendor on supported setups, upgrade paths, and so forth.
Infrastructure - Ensure a ProvisionServer DNS record exists for the target DNS domain, and that this pointer record resolves to the server running AMTconfig (e.g. Intel SCS). Ensure proper resolution of the DNS entry for the FQDN of ProvisionServer (e.g. ProvisionServer.company.com)
Verbose Logging for SCS events - Within the SCSconsole, access the Change the Log Level to "Verbose" mode. This will log all informational, warning, and error messages and events in the SCS log. This is good to see when a hello packet is received, when the ProvisionServer attempts to provision the target system, and so forth. When changing this setting, you may also want to decrease the log retention level to a few days or shorter timeframe than the default value. Depending on the number of systems managed or attempting to provision, setting the log level to "verbose" may rapidly grow the size of the IntelAMT database.
Image of SCSconsole and setting logging to verbose mode 
Image of SCSconsole viewing log events in verbose mode 
Remote Configuration - What is it? How does it work? When will it be available?
Posted by Terry Cutler Aug 29, 2007This article provides an overview. If you have questions, comments, and so forth - please respond.
Overview of Remote Configuration
With the launch AMT 3.0 on the Weybridge platform (e.g. Intel® vPro™ platform launched Aug 27, 2007), Intel is introducing an additional mechanism for enterprise provisioning and configuration: Remote Configuration (RCFG) formerly known as Zero-Touch Configuration (ZTC). This process provides an alternative approach to the existing pre-shared key (e.g. TLS-PSK) configuration or provisioning - commonly referred to as USB one-touch.
Enterprise configuration of Intel® vProTM of provides a central configuration control of the management engine. For environments using small-medium business (SMB) configuration, RCFG is not applicable. RCFG applies only to enterprise mode of provisioning of configuration the management engine.
In the diagram below - Remote Configuration accomplishes the transition from Factory Default to Setup Mode.
RCFG allows an IT administrator using an RCFG-enabled setup and configuration application (SCA), such as Intel Setup and Configuration Service (SCS), to remotely initiate a secure and authenticated communication channel with the Intel® AMT device in order to perform setup and configuration of the device. RCFG is based on a Mutual Transport Layer Security (MTLS) handshake with an Intel® Client Setup Certificate on the provisioning server and a self-signed certificate on the Intel® vPro™ client system. The Intel® vPro™ client system has a series of certificate hashes which identify approved certificate authorities.
During the MTLS handshake, the SCA sends a certificate to the Intel AMT system - this certificate is used to authenticate the SCA to the Intel AMT system. An enterprise customer or environment must obtain this SSL Server certificate from a commercial Certificate Authority (CA), with certificate properties as defined below. The chain of trust of this certificate must include as its root one of the root certificates which are on the list of root certificate hashes in the firmware image of the Intel AMT system - i.e., it is a trusted CA. The default firmware image supplied by Intel contains a default of list certificate hashes which can be edited by the OEM. In essence, this is the same as noting trusted certificate authorities for internet browsers.
The certificate which the customer purchases must be marked as capable of Intel AMT configuration in one of the following two ways:
OID value in EKU field = 2.16.840.1.113741.1.2.3
OU value in Subject field = "Intel(R) Client Setup Certificate"
These requirements, along with additional processes mention below, will prevent an attacker from attaining a certificate from one a less secure server of the end customer and using it to configure Intel AMT systems.
RCFG will be first available on AMT 3.0, released Aug 2007. Backwards support on AMT 2.x platforms will include AMT 2.2 (Intel vPro platform released in 2006) and AMT 2.6 (Intel Centrino Pro platform release in 2007). It is best to confirm with your preferred OEM on their plans to support the respective Intel AMT releases.
What is TLS-PSK?
Before diving deeper into Remote Configuration, a base understanding or awareness of the current provisioning process might be helpful. TLS-PSK refers to Transport Layer Security Preshared Key. Essentially - a pre-shared secret (actually, a triplet pair) is shared between the Intel AMT client and the Setup and Configuration Application (SCA).
Since this article focuses on Remote Configuration - only an overview of the predecessor is provided
Keys are generated and stored in the database
Keys are transferred to a USB key
The USB key is inserted into the Intel vPro Clients, which load a pre-shared secret record at boot
The client boots, negotiates an IP address and requests the IP of "ProvisionServer"
The client sends a "hello" packet to the ProvisionServer. The packet includes the public portion of the pre-shared key along with client specific\unique information.
The ProvisionServer matches the public portion of the pre-shared key (e.g. Provisioning Identifier), and performs a handshake based on the private key (provisioning passphrase). Once the handshake and TLS session obtained - the configuration data is transferred as part of the Intel AMT profile.
What is Required for Remote Configuration?
DHCP is required, with option 15 enabled and configured. This will return the DNS suffix with the IP lease. The DNS suffix will be used in validating the configuration certificate.
An Intel® Client Setup Certificate must be obtained, and associated with the ProvisionServer. In multiple ProvisionServers due to different DNS domains, one certificate per each is needed. More on this certificate will be shared later in this article.
An OEM update package must be obtained for the Intel® AMT firmware, the Management Engine Interface (MEI) drive and Local Management Service (LMS) driver.
An OEM platform with certificate hashes embedded in the non-volatile RAM of the Intel® AMT management engine, or the ability to update per the previous item.
An updated ISV client agent must be obtained, which will support the Remote Configuration process to allow agent initiation.
Version 3.1 or higher of Intel® Setup and Configuration Service (SCS) is required within the ProvisionServer. For ISVs implementing their own Setup and Configuration Application (SCA) - recommend checking with them directly (Hint: A quick check of the AMTconfig service will reveal the version number of SCS.)
The base certificate hashes loaded on AMT client are listed below. Again - it may be best to confirm with your preferred OEM on what certificate hashes they have chosen to support, make active, and so forth. There is space for 20 certificate hashes, thus allowing custom hashes to be created - especially for environments which manage their own enterprise PKI. However, a full UnProvision of the client will restore the base certificate hashes along with setting all base certificate hashes to active.
AMT Version | VeriSign G1 | VeriSign G3 | GoDaddy | Comodo | Starfield |
AMT 2.2 | Yes | Yes | Yes | Yes | Yes |
AMT 2.6 | Yes | Yes | Yes | Yes | Yes |
AMT 3.0 | Yes | Yes | Yes | Yes | No |
The exact fingerprints of the certificates can be provided if needed. Post a comment at the end of the article.
How does Remote Configuration Work?
Two approaches are included in the design: Agent Initiated and Bare-metal (aka agent-less).
AMT 3.0 and beyond will support both.
AMT 2.2 and 2.6 will support only Agent Initiated.
The initial actions and communications occur during a preparation stage which is unique between the agent initiated and bare-metal approaches. The preparation phase ends with a hello packet sent to the ProvisionServer announcing the Intel® AMT client request for a remote configuration session. Next is an authentication stage between the ProvisionServer and the Intel® AMT client for the setup of a mutual transport layer security (MTLS) session. Once the MTLS session is established, a valid FQDN and UUID map is obtained, the configuration process commences and the assigned Intel® AMT profile is sent to the client.
The following subsections provide some additional detail on the preparations phases and the mutual authentication for Remote Configuration. The final phase of Intel® AMT profile assignment, although not shown in this article, is similar to the existing pre-shared key approach once the session is encrypted (see TLS-PSK section above). The MTLS session established for Remote Configuration is separate from TLS and MTLS sessions defined in the Intel® AMT profile and used for management traffic security.
Agent Initiated Preparation Phase
Targeted for all Intel® AMT platforms supporting Remote Configuration, this process is best used after the host operating system has been installed along with the ISV client management agent to support Remote Configuration. The steps below assume that existing Intel® AMT version is 2.0 or 2.1 for Intel® vProTM systems, or Intel® AMT version 2.5 for Intel® Centrino® Pro systems. This implies that no certificate hashes have been stored on the Intel® AMT client, and an upgrade is needed for the BIOS, firmware, drivers, and so forth
Per the numbering in the diagram above, the agent initiated preparation process is as follows:
The ISV client management suite sends an updated BIOS, Intel® AMT firmware, and Intel® AMT drivers (e.g. MEI, LMS), and the latest ISV client agent. All of these updates are in support of the Remote Configuration process. In addition, the Intel® AMT firmware could be updated out-of-band (supported with AMT 2.1 and higher). The Intel® AMT firmware update package places a set of certificate hashes into the non-volatile RAM (NVRAM). More on the certificate hashes is mentioned in the next section.
The ISV client agent will then query the BIOS and firmware to check the configuration mode, configuration state, and Intel® AMT version. It will obtain the system's UUID and AMT version, passing this information to the ProvisionServer via the management console.
The management console creates a One Time Password (OTP), which is sent to the client agent and subsequently to the NVRAM. The management console also send the OTP to the ProvisionServer (running Intel® SCS). Note: Only the agent initiated process creates an OTP.
The ISV client agent sends a command to the Intel® AMT management engine, via the Management Engine Interface (MEI), to open the network interface. Once the network interface is open, the management engine obtains an IP address and queries for the ProvisionServer. The interface is open for 6 hours and will close if provisioning does not complete. The process can be restarted by the ISV client agent if needed.
Once the ProvisionServer IP address is obtained from DNS, a hello packet is sent containing the UUID and a self signed certificate created via the local certificate hash.
Bare-metal (e.g. Agent-Less) Initiated Preparation Phase
Targeted for Intel® AMT 3.0 and higher systems, this preparation phase allows the provisioning process to start before a host operating system has been loaded or if the client management agent is not used (i.e. agent-less).
!http://communities.intel.com/openport/servlet/JiveServlet/downloadImage/38-10209-1032/BarePrep.png!
Per the numbering in the diagram above, the bare-metal preparation process is as follows:
When the system first boots, a self-signed certificate is created based on the existing certificate hashes loaded by the OEM at the time of manufacturing. Only one of the hashes will be set to active, in accordance to the matching Intel® Client Setup certificate within the ProvisionServer's local certificate store. (There is an assumption here that the OEM activates or the IT administrator activates the appropriate certificate hash.)
Intel® AMT obtains a DHCP IP lease which includes option 15 (DNS domain suffix), and queries DNS for ProvisionServer within that DNS domain (e.g. ProvisionServer.Loc1.com).
The hello packet is sent to the ProvisionServer with the Intel® AMT client's UUID and self-signed certificate included.
*_Authentication Phase_*
After completing one of the two paths for preparation, the authentication between ProvisionServer and the Intel® AMT client commences. For this process to complete, the ProvisionServer has a valid Intel® Client Setup Certificate local to the system and associated for the Remote Configuration process. The Intel® AMT client has received a DHCP IP lease with option 15 included (the DNS domain suffix). The Intel® AMT client has generated a self-signed certificate based on the active certificate hash within the local NVRAM store. The active certificate hash matches the fingerprint of the Intel® Client Setup Certificate - which will be validated during the process. If agent initiated, the Intel® AMT client, ProvisionServer, and management console all have a matching One Time Password (OTP).
!http://communities.intel.com/openport/servlet/JiveServlet/downloadImage/38-10209-1034/RCFGstep2.gif!
Per the numbering in the animated image above, the mutual authentication process of Remote Configuration is as follows:
The ProvisonServer requests the self-signed certificate of the Intel® AMT client.
The Intel® AMT management engine requests the Intel® Client Setup Certificate from the ProvisionServer. Based on the self-signed certificate form the client, the ProvisionServer generates TLS key 1 and encrypts this using the public key obtained from client's self-signed certificate. The encrypted TLS key 1, Intel® Client Setup Certificate, and PEM file are then sent to the management engine. The PEM file defines the chain of trust. (An enterprise security architect will also be familiar with how to create a PEM file.)
At this point, the Intel® AMT client does some validation. Extracts and stores Key 1. Using the PEM file and Intel® Client Setup certificate, the management engine extracts the root certificate, generates a certificate hash and validates to the local active certificate hash. NOTE: If the two hashes do not match, the process stops. Validates the OU assignment of the Intel® Client Setup certificate to the DNS suffix received via DHCP IP lease with option 15. For this reason, each ProvisionServer MUST have a unique Intel® Client Setup certificate (see next section). A wildcard certification (e.g. *.company.com) is supported (AMT version 2.6 and beyond)
If the previous validation steps complete successfully, the Intel® AMT management engine creates TLS key 2, encrypts with the public key of the Intel® Client Setup Certificate obtained from the ProvisionServer, and transmits.
With TLS key 1 and key 2 obtained by both the ProvisionServer and the Intel® AMT management engine, mutual authentication has occurred and an MTLS session is established.
At this point, the configuration process occurs where the FQDN and UUID are matched, the assigned Intel® AMT profile is sent to the management engine, and the changes are committed.
Conclusion
Remote Configuration adds to the existing Pre-shared key configuration, providing two main approaches to preparing and configuring the Intel® AMT management engine. Remote Configuration will be supported first on Intel® AMT 3.0 (codenamed Weybridge) systems, followed by firmware upgrades to support current Intel® vProTM and Centrino® Pro systems. These firmware upgrades are Intel® AMT versions 2.2 and 2.6 respectively, and it is best to confirm with you preferred OEM whether they plan to support.
In addition the firmware upgrades, the ISV client management agent will require an update to support the agent initiated process. Updates to the Intel® AMT drives will also be required - specifically the MEI and LMS drivers.
For systems already deployed, an update package from the OEM will be required to load the appropriate certificate hashes and\or to set one of the hashes to active. The active hash on the Intel® AMT client will be used to create a self-signed certificate used in the authentication process. Certificate hashes outside the base 5 can be added to the OEM system, with space for up to 20 total. However, a full UnProvision of the client will restore the base certificate hashes along with active\inactive indicators.
Using SSL Server authentication certificates, certificate hashes, Mutual TLS authentication and encryption, updated firmware and drivers, and other key components - Remote Configuration establishes a trusted relationship to securely transmit the provisioning data without physically touching the Intel® AMT client system. This secure session is separate from the TLS configuration and sessions of an enterprise provisioned Intel® AMT client, which is defined in the Intel® AMT profile.
Each release of the Intel®
AMTprovides a few additional features. The good news is that Altiris handles the abstraction and interface or capabilities in heterogeneous environments. However, for troubleshooting, deployment, product selection, and other decisions - it may help to provide a summary of features and capabilities within each generation of the platform.
Yet this raises some questions: What version of Intel® AMT is running on the client system? What features does the version support? What are the dependencies on the Intel® Setup and Configuration Service (SCS) for Altiris Out-of-Band provisioning?
All good questions - let's step through each.
What version of Intel® AMT do I have?
Production Intel® AMT systems support either version 2.1 (for Intel® vProTM desktop systems) or 2.5 (for Intel® Centrino® Pro systems). First generation Intel® vProTM systems were codenamed Averill. First generation Intel® Centrino® Pro systems were codenamed Santa Rosa. Weybridge is the codename of the second generation Intel® vProTM desktop platform, and will start with Intel® AMT version 3.0.
If ever a question of what exact version of the MEBx (management engine
BIOSextension), the login screen will reveal the answer. This screen is typically access via Ctrl-P during the POST boot process. The picture below comes from an Intel® Centrino® Pro systems, running v2.15.15.0000 of the MEBx.
Click to view.
Versions and Features of Intel® AMT
The table below provides a summary of Intel® AMT platform versions and support features. Remote configuration, formerly called Zero Touch Configuration, will be released in September timeframe for Averill and Santa Rosa systems. A future article will address this topic.
Details on Intel® AMT versions beyond that which is stated will be shown later. There is an Intel® AMT 1.0 version, yet not branded as Intel® vProTM. Details will not be shared.
A common question is raised of migration paths from one Intel® AMT family to the next. For example - will Averill systems be firmware upgradeable to Weybridge. No. However, the functionality will continue to grow and the capabilities will be integrated into the Altiris console. Thus a mixed environment is supported.
Will you find all these items supported in the current Altiris console for out-of-band manageability? No. This is due to the underlying configuration service and management that is needed, and is presently provided by Intel® SCS. This will be addressed more in the next section. If you happen to download and test the Early Access version of
OOB, RTSI, and RTCI
portals.altiris.com/eap, you will see the additional functionalities.
Does this mean that Santa Rosa systems will not work with the current production environment? In a wired mode, they will perform all of the functions of Averill systems. Some key questions and consideration - Notebooks are powered by AC (wall plug) or DC (battery). In addition, with the wireless network support management of profiles and configurations is needed.
Before continuing to the next sections on power policies and supported states for wired versus wireless environments, a quick review of what hardware asset data is collected might help. See this document for a refresher.
Intel® AMT Power Policies
Adding Intel® AMT to notebooks presents some new possibilities and considerations. The tables below address some permutations of wired vs. wireless, running on AC or DC power, and whether the host system is healthy (e.g. system on and operating system running), sick (e.g. system on yet operating system failed or unavailable), or asleep (whether standby, hibernate, or off). The power states of Intel® are defined in a policy, with nomenclature that might confuse at first. (e.g. S0, S3, S5, and H0 power states). Review mention of power policies and states in .this article
In the case of Intel® Centrino® Pro systems, the power policy is even more extensive. Below is a screen shot of the MEBx menu for power policy. A similar list of options will appear in the Intel® AMT profile.
[this article|http://juice.altiris.com/files/u6338/power_state_policy.jpg]
Intel® AMT over Wired
Since Intel® AMT is on by default and consuming power, if powered by battery it may be better to turn the management engine off. This explains the "No" in the right column of the table below. What about the "No" for system isolation and agent presence is the system the host system is asleep and powered via AC? If the host system is off, what agent or virus outbreak is being prevented? That is why these states are not supported nor needed - they both require the host to be running (not necessarily the host operating system).
Wired mode also assumes the system is connected directly to the management network. If a remote site is connected to the main site via a VPN appliance, this is effectively a virtual extension of the managed network. However, if the target system requires a software agent running above the host operating system to support VPN, this is different and will be addressed in the next section.
Intel® AMT over Wireless
Similar to the last table, the next view addresses Intel® AMT over wireless. The use cases are the same, as are AC or DC selections, and the state of the host system. The key difference is whether Intel® AMT has wireless access. The wired
NIC is on by default and built into the hardware. The wireless NIC actually requires the host system to be on.
Wireless allows mobility, including outside of the managed network. Does this remove the manageability and security of the Intel® AMT platform? If a VPN connection to the managed network requires a Layer 3 (L3) VPN (virtual private network) agent running on top of the host operating system, does out-of-band management still apply?
Technologically, the L3 VPN functionality could be embedded into the Intel® AMT management engine. However, do the variety or vendors and approaches, this presents a major validation and certificate challenge. Plus, the number of potential updates and patches would not be favorable. Therefore, if the system is connected via wireless (or wired) outside of the managed network, and using an L3 VPN agent to connect in - there are some consideration of supported functionality. This does not apply to situations where a VPN appliance is between the Intel® AMT system and the managed network - since that is effectively a virtual extension of the physical managed network.
Intel® AMT 2.5 supports wireless profiles. Thus if the Intel® AMT device is inside a managed environment, connecting via an 802.11 b/g/n network with defined SSID and configurations, the system can be managed out-of-band.
Note: 802.11i, 802.1x, and NAC are supported in the Intel® AMT profile for Centrino® Pro environments. More on this in a latter section
802.11n is draft version today. For best compatibility, use 802.11 b/g networks
With the perceived constraints of Intel® AMT over wireless, does out-of-band still apply? Yes.
If the device is within a managed network environment - whether wireless or wired - and has been configured (e.g. provisioned) correctly, the system is still manageable if the host is on. (e.g. healthy or sick). As mentioned in an earlier tech-tip, system data is still being collected at boot. Plus, if a network filter policy is pushed down to the Intel® AMT device - part of the System Isolation capability (formerly called Circuit Breaker), that policy remains in effect until removed.
If the system is on, yet the host operating system is not functional and the system is inside a managed network environment (whether wired or wireless) - the system is manageable.
The last item may raise some concern - if the system is wireless and host is asleep (S1 through S5 power state), why are all use cases not supported? Remember that the wireless network driver for Intel® AMT requires the host to be on (H0), whether or not the operating system is running. Only the wired NIC is powered in the asleep state - if a connection is available.
Intel® SCS versions
Within the Altiris OOB server install, a service labeled "AMTconfig" is running. The version number of this windows service refers to the Intel® SCS version number. The Altiris Out-of-Band management console (under Provisioning > Configuration Service Settings) will show additional options and capabilities with higher versions of Intel® SCS. Yet what is the relation and mapping of Intel® SCS to Intel® AMT?
The general concept is this: The major version number of Intel® SCS (again - check the AMTconfig service version number) indicates all versions of Intel® AMT supported at or below that number. Therefore, Intel® SCS 3.0 would support AMT 2.0 through 3.0 versions (i.e. Averill, Santa Rosa, and Weybridge). Of course, there are always expections to the rule. Intel® SCS version 1.x supports Intel® AMT 2.1 and below, and Remote Configuration will be supported with a version of Intel® SCS slightly above 3.0. (More on Remote Configuration in a near future article)
The opinions expressed on this site are mine alone and do not necessarily reflect the opinions or strategies of Intel Corporation or its worldwide subsidiaries.
