The Task Server contains AMT function tasks that give you the ability to integrate AMT functionality into Task Server Jobs. This allows you to use AMT in conjunction with Software Delivery, Scripting, and any other Task Server supported function. Understanding how to troubleshoot the AMT side of a Task Server job will help resolve issues so that AMT can be utilized. This includes the following technologies:

• System Defense - Network Filtering
• Reliable Power Management
• IDE redirect for boot redirection

Introduction

This is the concluding article for the series: Troubleshooting the Altiris Manageability Toolkit for vPro Technology. The first four articles covered the setup and configuration of AMT systems, while parts 5 and 6 covered RTCI and RTSM respectively. This final article discusses troubleshooting the AMT integration into Task Server when issues arise.

As an introduction, the actual SOAP or API calls made to the AMT system is invoked through Real-Time Console Infrastructure, the same as when they are invoked through the Real-Time tab for RTSM. Though the calls are from the same place, how those calls are made differ. The following subjects will be covered:

• Determining Cause of Failure
• AMT Detection Issues
• Authentication Issues

Determining Cause of Failure

Often you'll known the general symptom that tells you a job or task in Task Server didn't execute as expected. For example a power management task may have shown as run but the AMT system never woke up. A failure is not shown except deep within a series of status windows.

To determine the returned error, use the following steps. Task Server's actual failure code is buried deep in a series of status windows, as shown in the screenshot after the steps.

1. Under the Task or Job that failed, double-click on the general status row for the specific execution attempt.
2. If within a job, double-click on the line that represents the task or AMT function that failed.
3. Note the numbers of successes versus failures. Click the ‘View Report' link.
4. Now you'll get a grid with the status of the Task, including the status and return code, if present.

AMT Detection Issues

When Task Server reaches a Task that involves AMT, it makes direct calls to AMT in those systems targeted in the task or job. Detecting AMT and subsequently executing the scheduled function requires success at both junctures. The following sections discuss potential issues and solutions in this process.

Power State Unknown

One common problem we see is when a power management task fails due to the failure message: Generic error, FromState detected as unknown:14. This will cause the power action to fail. The causes vary, but the following list contains the most common:

• System unreachable - The target system is not available on the network
• AMT failed to be detected - See the subsequent section ‘AMT not detected'
• Authentication failed - See the subsequent section ‘Authentication Troubleshooting'
• AMT is unavailable - If a system is not provisioned, or AMT is not functioning on that system

Use the following process to determine what the issue is:

1. If RTSM is available, try connecting to the target system using RTSM, specifying the same credential profile.
2. If that fails, try manually putting in credentials until you find one that works.
3. If Step 1 succeeds, try creating a different connection profile with only AMT functions provided.
4. If no RTSM is available, still try the profile with only AMT functions to see if it works.
5. Try other AMT functions, such as Collect Intel AMT Inventory to see if they succeed.
6. If other functions succeed, try using another method to reboot the system to reset the power state stored in the Intel ME. One way to accomplish this is using the Task Server Power Management Agent to send down a standard reboot command to the PC.
7. If no other AMT functions are successful, AMT might not be properly setup on this system. Ask the question: Has this system gone through the provisioning process?
8. If unknown, use the Out of Band Discovery Task to see if AMT is available and to identify what state it is in. See the steps provided under the ‘AMT Not Detected' section following.
9. If all else fails (generally this is on a system-by-system basis, rarely do a collection of systems encounter this level of this issue) try reprovisioning the system by fully unprovisioning and going through the provisioning process again.

AMT Not Detected

Normally a non-vPro system will receive the return code that AMT was not detected. This is accurate, but when it happens to valid managed vPro systems, the issue must be troubleshot to determine why the applying Task Server cannot detect AMT on the system. Out of Band Discovery is a great way to determine what state the system is in. Use the following steps to take stock of the systems:

1. In the Altiris Console, browse to View > Solutions > Out of Band Management > Configuration > Out of Band Discovery > and select the ‘Out of Band Discovery' policy.
2. Enable the policy if it is not yet enabled. If it is enabled, set a schedule to run the discovery again so you have updated information on your systems.
3. On the AMT system in question, go to the Altiris Agent and bring up the Agent UI by double-clicking on the system tray icon or by launching C:\Program Files\Altiris\Altiris Agent\AeXAgentActivate.exe.
4. Highlight the ‘Out of Band Discovery Package.
5. Click the ‘Out of Band Discovery' link under Application Tasks.
   
6. Once completed, now check back at the server and double-click the system within a collection to bring up Resource Manager.
7. Click on the Inventory tab and browse to Out of Band Management, and select the data class OOB Capability. This will give you the details of AMT.

If AMT is disabled, it needs to be enabled in the BIOS. A BIOS update from the vendor may provide you a remote way to enable AMT, by using Software Delivery for example. If it is all enabled, next check the provisioning status. Provision as necessary.

Authentication Issues

As with RTSM, Task Server uses the same basic authentication method when executing against a computer. Task Server also includes another option to add additional credentials to the execution to be used when contacting the protocol, which is AMT in this case.

Authentication Methods

Since RTCI controls the authentication, much of the same method is used whether the execution of an AMT command is issues from the Real-Time console or from Task Server, however there are some differences.

Runtime Profile - The Runtime profile contains he following information:

• All known good credentials used to connect via RTSM to a system
• The Intel SCS AMT password sent to systems when provisioning occurs
• Previously successfully used credentials from past RTSM sessions
• Previously successfully used credentials from a Task that succeeded

User-defined Profiles - Profiles can be created that specifically provide credentials for the four types of technologies:

• WMI digest or Domain account
• AMT digest or Kerberos-authenticated user
• ASF digest or Domain account
• SNMP community strings

Task-specified Credentials - When a user setups up a job or task, the user can specify specific credentials to be used when executing AMT-related functions through the profile interface. This option is per job or task, and applies to all AMT functions invoked during the job or task. The Interface allows this as shown in the following screenshot:

Authentication Troubleshooting

The following method will help identify issues and offer ways to work-around and solutions. These have been compiled through experience when troubleshooting issues with failed authentication with Task Server.

1. First, how do you determine if your task or job is failing due to authentication? Use the previous section under Introduction labeled ‘Determining Cause of Failure'.
2. In the Altiris Console browse to View > Solutions > Real-Time Console Infrastructure > Configuration > select Manage Credentials Profiles, or in the Task click the ‘Run Now', and on the subsequent page click on the pencil icon next to the credential profile being used.
3. Where does the green checkmark fall? This is the default profile that will be used when connecting via a Task Server task.
4. Create a new profile by clicking the blue + on the icon bar in the right-hand pane.
5. Under the Intel® AMT tab check the box ‘Enable this technology in the profile'.
6. Supply the admin user credentials set when the managed vPro systems were provisioned.
7. Under the WMI tab also check the box as above and provide a user that has admin privileges to the target system.
8. Give the profile a name and then save it.
9. Back at the main screen check the box under the ‘Default' column until the green check-mark uses your new Profile, or if you are in a job interface select the profile to be used for the run. Note that this does not require you to make it the default profile, allowing another profile to remain the default credentials.
10. Run the task or job to see if the authentication failure has been resolved.
11. If it is not, try rerunning with the Runtime Profile. This contains all known good authentication attempts to the system from either Task Server or RTSM.
12. In one case we supplied only AMT credentials in the Profile which allowed it to authenticate to AMT while a multiple protocol authentication profile failed. If your Task or Job does not contain any of the other protocols, this is recommended.

Conclusion

This concludes the Troubleshooting article series for the Altiris Manageability Toolkit for Intel vPro Technology, version 6. While this doesn't cover all issues, it should resolve most of the common issues we've seen.

Formerly known as Web Admin for Windows, Real-Time System Manager provides a powerful set of functions for IT specialists. In part 5 of this article series we covered the main points for Real-Time Console Infrastructure troubleshooting. As a natural extension of RTCI, Real-Time System Manager troubleshooting is covered in this article as part 6. With an emphasis on credentials and connection methods, this article provides information to overcome the most common issues seen when using the Real-Time tab for direct, one-to-one computer interaction.

Introduction

Real-Time System Manager provides a powerful tool for directly connecting to a system agentlessly with functionality available through WMI and Intel AMT. This article covers the issues associated with general functions seen with both technologies but with emphasis on the AMT functions. The following sections cover areas of troubleshooting:

• Connection Issues
• Authentication Issues
• IDE Redirect (IDER)
• Network Filtering

Connection Issues

Under the current architecture the FQDN is the primary method for connecting and authenticating to AMT on remote systems. If the FQDN the Real-Time tab is using does not resolve in DNS, then AMT connectivity and thus functionality will not be available. FQDN connectivity issues are the number one issues we see with RTSM connections to AMT.

Invalid FQDN

To view what FQDN the Real-Time is using, use the ‘Hardware Management' node in the RTSM tree. The following screenshot shows what AMT is using:

In this example my system is in a workgroup and reported only the hostname as the FQDN, which DNS had no trouble resolving. If this fqdn is not reachable via DNS, we won't be able to connect to the AMT functionality.

NOTE: We use several methods, including IP address, for WMI. WMI functionality may show correctly when AMT is absent in this situation

Use these steps to see the FQDN is the issue:

1. Open the Real-Time tab for the AMT system you are managing.
2. Once the tree loads, open the Real-Time System manager folder, open Administrative Tasks, and click on ‘Hardware Management'.
3. Once the page loads, if AMT is missing as an available technology, take note of the name displayed as in the screenshot above.
4. Go to Start, Run, type in cmd, and click OK.
5. Type in nslookup <name displayed>. In the above example it would read:
   1. Nslookup dellvpro
6. Can DNS resolve this address? If no, we'll need to fix the issue in one of the following ways.
7. FIX DNS and/or the Altiris record: If DNS can be fixed, this is the preferred method. The difficulty is finding out why the Altiris Agent reported the incorrect record. Once DNS is fixed, have the Altiris Agent run Basic Inventory. The table location we pull this out of for management in RTSM is Inv_AeX_AC_Location, column: Fully Qualified Domain Name.
8. Use the ‘Manage' node available in RTSM (see the below screenshot): By putting in the IP address of the system, we'll use the IP to lookup the FQDN and not make any assumptions.
   
9. Update the Servers HOSTS or LMHOSTS files to contain the mapping to the invalid name. For example find the LMHOSTS file, edit it and add a line <IP ADDRESS> <FQDN>, as in this example:
   1. 10.10.10.1 Dellvpro

Real-Time unable to connect

If WMI and AMT functions are unavailable, you'll get a message when you click on the Real-Time tab indicating that the functionality isn't available. See the following screenshot:

Note: If you use another product such as Dell or HP's plug-ins to this tab, you'll simply not have the ‘Real-Time System Manager' node underneath Real-Time Consoles.

The number one reason this occurs is due to a firewall being engaged. Firewalls need to allow AMT traffic through. If a firewall is enabled, use the following details to resolve the AMT issue:

1. Create an inclusion in the firewall properties.
2. Allow the following ports, based off your environment:
   1. 16992 - For non-TLS encrypted traffic - if you are not using TLS this is the port that will be used for communication
   2. 16993 - For TLS-enabled, encrypted AMT traffic - If https is required for communication with AMT, this port will be used
   3. 16994 - For a note, AMT provisioning uses this port for sending out the ‘hello' packet during the configuration process - this will be used if you initiate a reprovision from RTSM
3. Another options is to disable the firewall when you need to manage the system via RTSM.
4. Unfortunately WMI has a known issue with the Windows firewall where the dynamic ports WMI uses after initiation will be blocked. It's a bug in WMI that has been addressed in Vista. Previous Operating Systems do not have a resolution at this time.

The other issue we've seen is where the system is simply unavailable for one reason or another. AMT is available if the system is off but still connected to the network, but WMI or if the system is unplugged from power or off the network RTSM obviously cannot function. Verify that the system is available if nothing resolves this issue.

Authentication Issues

Another common issue concerns authentication to the system via the Real-Time tab. First, let me discuss the methods RTSM uses to authenticate to a target system.

Authentication Methods

Runtime Profile - The Runtime profile contains he following information:

• All known good credentials used to connect via RTSM to a system
• The Intel SCS AMT password sent to systems when provisioning occurs
• Previously successfully used credentials from past RTSM sessions

User-defined Profiles - Profiles can be created that specifically provide credentials for the four types of technologies:

• WMI digest or Domain account
• AMT digest or Kerberos-authenticated user
• ASF digest or Domain account
• SNMP community strings

Manually entered credentials - When RTSM tries to connect, if the default profile set in the RTCI configuration fails to authenticate, the left-hand tree will still load but each node will prompt the user for credentials. A user can put in an AMT account, Domain user, or digest user that has rights on the target system. When authentication succeeds, these credentials are then stored in the Runtime Profile for the target system.

Troubleshooting Authentication

The following method will help identify issues and offer ways to work-around and solutions. These have been compiled through experience when troubleshooting issues with failed authentication with RTSM.

1. In the Altiris Console browse to View > Solutions > Real-Time Console Infrastructure > Configuration > select Manage Credentials Profiles.
2. Where does the green checkmark fall? This is the default profile that will be used when connecting via the Real-Time tab.
3. Create a new profile by clicking the blue + on the icon bar in the right-hand pane.
4. Under the Intel® AMT tab check the box ‘Enable this technology in the profile'.
5. Supply the admin user credentials set when the managed vPro systems were provisioned.
6. Under the WMI tab also check the box as above and provide a user that has admin privileges to the target system.
7. Give the profile a name and then save it.
8. Back at the main screen check the box under the ‘Default' column until the green check-mark uses your new Profile.
9. Test to see if this new profile is successful. Note that you'll need to launch IE fresh to use the new settings.
10. If it is not, try entering credentials in manually when you hit the system under the Real-Time tab. See the screenshot below for the connection icon to switch between WMI and AMT authentication. If two show in this area, both technologies are available but not authenticated.
    
11. In one case we supplied only AMT credentials in the Profile which allowed it to authenticate to AMT while a multiple protocol authentication profile failed.
12. Check the collection you are launching Resource Explorer from. Sometimes the identity of the system is incorrect. For AMT you can launch RTSM from the Provisioned collections populated with the Resource Synchronization.

IDE Redirect (IDER)

IDE Redirect allows a system to be remotely booted to a file, drive, or virtual disc. There are a number of potential issues to be aware of when working with IDER in a vPro environment. The below items include well-known issues and their resolutions.

Redirection Invalid Parameter

When initiating an IDER (IDE Redirect) session to an external source such as an .iso file, the following error appears in the console:

Power management operation failed.
Redirection session start has failed. See logs for more details.

The Notification Server log shows the following error:

Log File Name: C:\Program Files\Altiris\Notification Server\Logs\a.log
Priority: 2
Date: 3/9/2007 2:51:05 PM
Tick Count: 10617218
Host Name: <>
Process: w3wp.exe (2436)
Thread ID: 5412
Module: AltirisNativeHelper.dll
Source: RTCI.Trace
Description: RedirectionProvider::StartIDER - RedirectionProvider::StartIDER - IMR_IDEROpenTCPSession: IMR_RES_INVALID_PARAMETER

This is caused by Intel's redirection library requiring a correct floppy device to initiate an IDER session (either floppy image or real removable device). Real-Time System Manager 6.2 can work around this. If you put floppy.img file into Program Files\Altiris\RTSM\UIData folder, then the issue will not occur.

IDER or SOL Disabled

In some instances Intel vPro systems are arriving from the OEM with IDER and SOL disabled in the BIOS. When disabled, neither of these functions work from any management engine, including RTSM. Correcting this oversight is not easy, especially if the OEMs do not offer a solution by a firmware or BIOS update. Use the following method to resolve the issue:

1. Go to the Support site for the OEM for the systems.
2. Browse to the drivers and downloads section for the exact model (note that sometimes the model will differ based on possessing or not possessing vPro technology).
3. Check the firmware updates for a new BIOS.
4. Check the documentation for any new BIOS versions that include vPro to see if they've corrected this.
5. Contact your OEM if they have not and request a status!
6. The only other recourse is to develop an update yourself or manually update the settings by visiting the system.

Conclusion

This should account for the most common issues we've seen, and allow you to successfully use RTSM with AMT technology, avoiding those issues.

For general questions about Remote Configuration, please review the following article - http://communities.intel.com/docs/DOC-1490

Earlier in 2007, I wrote two brief articles about Remote Configuration.

• http://communities.intel.com/openport/blogs/proexpert/2007/08/29/remote-configuration-what-is-it-how-does-it-work-when-will-it-be-available
  
• http://juice.altiris.com/article/2161/remote-configuration-preview
  

The embedded video below is a summary of how Remote Configuration works in an Altiris environment. The target environment has a VeriSign Intel(R) Client Setup Certificate loaded. Intel AMT 2.2 and 2.6 systems are provisioned using Agent Initiated approach. The Intel AMT 3.0 system is provisioned using the baremetal approach (could have done this via agent initiated... yet wanted to show both methods)

More content\details on acquiring external certificates, or creating a custom internal certificate and adding the associated certificate hash to the clients... can be provided if needed.

Here's an interesting one on USB provisioning -- just published in Known Issues, Best Practices, and Workarounds:

What is the maximum number of PID/PPS pairs that can be used during USB provisioning?

Much of the recent news around Microsoft System Center has been focus on the upcoming release of Microsoft System Configuration Manager SP1. However, did you know that MS System Center Operations Manager (SCOM) and System Center Essentials (SCE) also have support for vPro Client management through the use of the Intel Client Manageability Pack for SCOM & SCE?

First a little background...

What is MS System Center Operations Manager? SCOM is the third generation of Microsoft's monitoring solution. Operations Manager provides an easy-to-use monitoring environment that monitors thousands of servers, applications, and clients to provide a comprehensive view of the health of an organization's IT environment.

What is MS System Center Essentials? Microsoft System Center Essentials 2007 is a new management solution in the System Center family of IT systems management products. Essentials is specifically designed for IT professionals working in midsize businesses who often face IT challenges similar to those of larger enterprises - troubleshooting end user problems, automating management tasks, managing multiple systems, and diagnosing and resolving IT problems.

So where does the Intel Client Management Pack for SCOM and SCE fit in?

The Intel Client Management Pack for Microsoft System Center Operations Manager and System Center Essentials enables the users of these applications to take advantage of the advanced, hardware-based system management capabilities of vPro. These capabilities will help reduce the cost of discovering, managing and securing desktop and mobile PCs in the enterprise and thereby improving compliance with corporate policies. Information Technology support staff can now rapidly and remotely communicate with, power up, reboot, control, inventory assets, and remediate vPro capable clients even if the PC is powered off or the operating system is not functioning.

For more information and download of the Intel Client Manageability Pack for SCOM and SCE, please visit: http://softwarecommunity.intel.com/articles/eng/3681.htm

Matt Royer

Let the Journey begin.. On my way to MMS (las vegas). Hopefully to make it to the show floor in a few hours and start to scope out the Intel booth. I am looking forward to talking with IT shops that are using vPro & the folks that are about to take on their journey. If you are also interested to learn more make sure you stop by the booth.

As soon as I arrive I’ll shoot a quick video of the booth & post on the Expert Center.. .

Here are some new updates for the Known Issues, Best Practices, and Workarounds wiki:

*Updated results for the USB key provisioning matrix

*BIOS password screen unavailable on HP systems during SOL session