The Task Server contains AMT function tasks that give you the ability to integrate AMT functionality into Task Server Jobs. This allows you to use AMT in conjunction with Software Delivery, Scripting, and any other Task Server supported function. Understanding how to troubleshoot the AMT side of a Task Server job will help resolve issues so that AMT can be utilized. This includes the following technologies:

• System Defense - Network Filtering
• Reliable Power Management
• IDE redirect for boot redirection

Introduction

This is the concluding article for the series: Troubleshooting the Altiris Manageability Toolkit for vPro Technology. The first four articles covered the setup and configuration of AMT systems, while parts 5 and 6 covered RTCI and RTSM respectively. This final article discusses troubleshooting the AMT integration into Task Server when issues arise.

As an introduction, the actual SOAP or API calls made to the AMT system is invoked through Real-Time Console Infrastructure, the same as when they are invoked through the Real-Time tab for RTSM. Though the calls are from the same place, how those calls are made differ. The following subjects will be covered:

• Determining Cause of Failure
• AMT Detection Issues
• Authentication Issues

Determining Cause of Failure

Often you'll known the general symptom that tells you a job or task in Task Server didn't execute as expected. For example a power management task may have shown as run but the AMT system never woke up. A failure is not shown except deep within a series of status windows.

To determine the returned error, use the following steps. Task Server's actual failure code is buried deep in a series of status windows, as shown in the screenshot after the steps.

1. Under the Task or Job that failed, double-click on the general status row for the specific execution attempt.
2. If within a job, double-click on the line that represents the task or AMT function that failed.
3. Note the numbers of successes versus failures. Click the ‘View Report' link.
4. Now you'll get a grid with the status of the Task, including the status and return code, if present.

AMT Detection Issues

When Task Server reaches a Task that involves AMT, it makes direct calls to AMT in those systems targeted in the task or job. Detecting AMT and subsequently executing the scheduled function requires success at both junctures. The following sections discuss potential issues and solutions in this process.

Power State Unknown

One common problem we see is when a power management task fails due to the failure message: Generic error, FromState detected as unknown:14. This will cause the power action to fail. The causes vary, but the following list contains the most common:

• System unreachable - The target system is not available on the network
• AMT failed to be detected - See the subsequent section ‘AMT not detected'
• Authentication failed - See the subsequent section ‘Authentication Troubleshooting'
• AMT is unavailable - If a system is not provisioned, or AMT is not functioning on that system

Use the following process to determine what the issue is:

1. If RTSM is available, try connecting to the target system using RTSM, specifying the same credential profile.
2. If that fails, try manually putting in credentials until you find one that works.
3. If Step 1 succeeds, try creating a different connection profile with only AMT functions provided.
4. If no RTSM is available, still try the profile with only AMT functions to see if it works.
5. Try other AMT functions, such as Collect Intel AMT Inventory to see if they succeed.
6. If other functions succeed, try using another method to reboot the system to reset the power state stored in the Intel ME. One way to accomplish this is using the Task Server Power Management Agent to send down a standard reboot command to the PC.
7. If no other AMT functions are successful, AMT might not be properly setup on this system. Ask the question: Has this system gone through the provisioning process?
8. If unknown, use the Out of Band Discovery Task to see if AMT is available and to identify what state it is in. See the steps provided under the ‘AMT Not Detected' section following.
9. If all else fails (generally this is on a system-by-system basis, rarely do a collection of systems encounter this level of this issue) try reprovisioning the system by fully unprovisioning and going through the provisioning process again.

AMT Not Detected

Normally a non-vPro system will receive the return code that AMT was not detected. This is accurate, but when it happens to valid managed vPro systems, the issue must be troubleshot to determine why the applying Task Server cannot detect AMT on the system. Out of Band Discovery is a great way to determine what state the system is in. Use the following steps to take stock of the systems:

1. In the Altiris Console, browse to View > Solutions > Out of Band Management > Configuration > Out of Band Discovery > and select the ‘Out of Band Discovery' policy.
2. Enable the policy if it is not yet enabled. If it is enabled, set a schedule to run the discovery again so you have updated information on your systems.
3. On the AMT system in question, go to the Altiris Agent and bring up the Agent UI by double-clicking on the system tray icon or by launching C:\Program Files\Altiris\Altiris Agent\AeXAgentActivate.exe.
4. Highlight the ‘Out of Band Discovery Package.
5. Click the ‘Out of Band Discovery' link under Application Tasks.
   
6. Once completed, now check back at the server and double-click the system within a collection to bring up Resource Manager.
7. Click on the Inventory tab and browse to Out of Band Management, and select the data class OOB Capability. This will give you the details of AMT.

If AMT is disabled, it needs to be enabled in the BIOS. A BIOS update from the vendor may provide you a remote way to enable AMT, by using Software Delivery for example. If it is all enabled, next check the provisioning status. Provision as necessary.

Authentication Issues

As with RTSM, Task Server uses the same basic authentication method when executing against a computer. Task Server also includes another option to add additional credentials to the execution to be used when contacting the protocol, which is AMT in this case.

Authentication Methods

Since RTCI controls the authentication, much of the same method is used whether the execution of an AMT command is issues from the Real-Time console or from Task Server, however there are some differences.

Runtime Profile - The Runtime profile contains he following information:

• All known good credentials used to connect via RTSM to a system
• The Intel SCS AMT password sent to systems when provisioning occurs
• Previously successfully used credentials from past RTSM sessions
• Previously successfully used credentials from a Task that succeeded

User-defined Profiles - Profiles can be created that specifically provide credentials for the four types of technologies:

• WMI digest or Domain account
• AMT digest or Kerberos-authenticated user
• ASF digest or Domain account
• SNMP community strings

Task-specified Credentials - When a user setups up a job or task, the user can specify specific credentials to be used when executing AMT-related functions through the profile interface. This option is per job or task, and applies to all AMT functions invoked during the job or task. The Interface allows this as shown in the following screenshot:

Authentication Troubleshooting

The following method will help identify issues and offer ways to work-around and solutions. These have been compiled through experience when troubleshooting issues with failed authentication with Task Server.

1. First, how do you determine if your task or job is failing due to authentication? Use the previous section under Introduction labeled ‘Determining Cause of Failure'.
2. In the Altiris Console browse to View > Solutions > Real-Time Console Infrastructure > Configuration > select Manage Credentials Profiles, or in the Task click the ‘Run Now', and on the subsequent page click on the pencil icon next to the credential profile being used.
3. Where does the green checkmark fall? This is the default profile that will be used when connecting via a Task Server task.
4. Create a new profile by clicking the blue + on the icon bar in the right-hand pane.
5. Under the Intel® AMT tab check the box ‘Enable this technology in the profile'.
6. Supply the admin user credentials set when the managed vPro systems were provisioned.
7. Under the WMI tab also check the box as above and provide a user that has admin privileges to the target system.
8. Give the profile a name and then save it.
9. Back at the main screen check the box under the ‘Default' column until the green check-mark uses your new Profile, or if you are in a job interface select the profile to be used for the run. Note that this does not require you to make it the default profile, allowing another profile to remain the default credentials.
10. Run the task or job to see if the authentication failure has been resolved.
11. If it is not, try rerunning with the Runtime Profile. This contains all known good authentication attempts to the system from either Task Server or RTSM.
12. In one case we supplied only AMT credentials in the Profile which allowed it to authenticate to AMT while a multiple protocol authentication profile failed. If your Task or Job does not contain any of the other protocols, this is recommended.

Conclusion

This concludes the Troubleshooting article series for the Altiris Manageability Toolkit for Intel vPro Technology, version 6. While this doesn't cover all issues, it should resolve most of the common issues we've seen.

Sometimes the methods for dealing with hostile or infected systems on the network are drastic, resulting in lost productivity, time, and energy. In one example the IT staff would physically shut down the user's main network port, sealing off all production systems, test systems, etc, until the hostile machine could be dealt with. Phone calls results, requiring the user to deal personally with the affected system. Now take Intel AMT's System Defense. Remotely quarantine a hostile system and use Altiris to remediate it. System Defense, it puts the power in the hands of the administrator remotely.

Introduction

System Defense (formerly known as Circuit Breaker) allows network filtering at the level of AMT. Systems that have been compromised and are a threat to the network can be remotely quarantined, with certain ports and IP addresses available for remediation. For example the entire network can be filtered out except to the NS, and only those ports required for the Notification Server to remediate the client (install anti-virus, patches, remove harmful software, etc).

Note that testing is vital when using a mechanism that can potentially cut off a system from the network. The ease of remediating compromised systems remotely while quarantining from the main network will remain as long as the filters are properly configured. If not, the system may require a desk-side visit to bring back on the network.

System Defense

System Defense shows as Circuit Breaker in some versions of the Altiris Manageability Toolkit for Intel® vPro Technology. This feature allows a network filter to be placed at the hardware level via AMT. AMT will hijack the operating system's hold on the network connection and apply a secure filter based on a configuration file provided by the administrator.

See the following diagram for a representation of how System Defense (Network filtering) works:



This filter becomes a complete block that disallows any network communication in OR out, save those sources that are configured. Note that the parameters for allowing network communication are those of Sending IP Address and Port. This means that not only to systems have to be explicitly defined to be allowed through, but the ports they are using as well.

Use Cases

The following use cases will find real value with System Defense network filtering:

• Virus attack from an infected vPro client - This cuts off the ability of that virus to send packets out on the network
• Vulnerable vPro clients without anti-virus - Close off the ability of a virus from getting through to the vulnerable system
• Vulnerable vPro clients without critical patches or updates - Quarantine systems, but allow NS to remediate to bring the system up to corporate security standards
• Unauthorized Network use - plug a system that is found participating in unauthorized network use, whether it be unauthorized content, gross use of bandwidth for non-approved purposes, etc...
• For fun - Drive a fellow administrator crazy by applying and removing filters randomly from his computer (Just kidding, don't try this at home, or at work for that matter)

Task Server Integration

As of Real Time Console Infrastructure release 6.3 the Task Server now has a Task type of Network Filter. This exclusively uses Intel AMT System Defense to apply a comprehensive filter that only allows strict communication to and from the NIC. Because of Task Server's sequencing engine and collection targeting, jobs using this can be setup to do a large number of things, including patching, critical application install such as anti-virus, and other critical computer maintenance items required by the organization.

Task Server Jobs

As a primer for details in this article, see the following article series on Altiris Juice: http://juice.altiris.com/article/2088/utilizing-intel-vpro-amt-technology-with-task-server-introduction.

See the Introduction for more information on jobs. There are two major types of a Network Filtering job:

1. Apply a System Defense network filter, either the default filter allowing communication to the NS for remediation or a custom filter allowing access to necessary resources
2. Remove a System Defense network filter to open back up general network communication

See the following screenshot for the option when this Task type is created:

• The first radial button allows the application of a filter, either a custom or the default, with the added option of enabling anti-spoofing filter
• The second radial button simply applies a PING filter to the target systems
• The third and final radial button removes any filters previous applied to the system

Job Targeting

Because of the significance of System Defense and what it does to client computers, I'm going to cover how Task Server Jobs target systems. With a Task Server job you can add individual systems or whole collections of computers. Collections are either manually or dynamically defined and can have few or many systems therein. Multiple systems and collections can be attached to the running of a job, either on demand or by a schedule.

Since System Defense is essentially quarantining vPro Systems, any Task or Job should be tested in a lab environment to ensure workability. If a custom filter is used, the potential to decapitate vPro systems from the network becomes a very real, very severe consequence of improper filters. Take the scenario of having a custom filter that does not allow proper communication back to the Notification Server or another critical resource (like Task Server) in the remediation process. Once the trigger is pulled and the System Defense network filter has been applied, those systems now have insufficient network access to remediate, which may mean that a remote Task to remove the filter is unavailable. IF the job contained half the computers in the environment, the impact is huge.

I say again: Test every filter within every job to ensure everything works properly!

Filter Configuration

Real-Time System Manager allows you to create your own filter configuration files to use with a System Defense Task. In some instances it may be required to open additional ports or destination IPs for full remediation to occur. If you use Package Servers to deliver software you may need to allow communication to these systems.

Edit Network Filters Utility

A utility is provided to create, edit, or otherwise revise any filter file to be used by a System Defense Task. This filter is provided via the Altiris Knowledgebase.

Installation The ENF Utility

See the following article for both the guide in using the utility and to download the utility directly:

https://kb.altiris.com/article.asp?article=34891&p=1

The attached file is a zip. The file included Altiris_ENF_6_2.exe will install the utility on the computer it is executed on. The prerequisites for this utility include:

1. Windows 2000 Server or Windows 2003 Server
2. .NET 1.1
3. Notification Server 6.0 Sp3
4. At least Real-Time Console Infrastructure 6.2

Using the ENF Utility

Once the installation has run, the Altiris Console can now be used to edit the filters. It's found in the Altiris Console under View > Solutions > Real Time Console Infrastructure > Configuration > and click on ‘Edit Network Filters'. The console provided a spreadsheet of the current filters for the default filter file, as shown:



When you click the Edit pencil icon, a subsequent window will appear. This wizard will walk through editing of the filters. This same wizard is used to add new filters to the list. This wizard is robust and allows minute tuning of what ports are allowed, both for sending and receiving from the NS and from the host AMT computer. The wizard appears as follows:

The default file is called CBFilters.xml and is found at \Program Files\Altiris\RTSM\UIData\. Other files can be created and used in the System Defense Filtering Tasks. It is configurable per Task or Job instance.

NOTE: If you plan on making changes to the default filter file, it is recommended to browsing to the file and making a copy of it. The copy will be a backup to use in case the default file becomes corrupt through editing or for related recovery options.

The best way to know how to open which ports to enable the access you require is to consult the documentation for the application or mechanism you are trying to work with. For example the Task Server uses ports 50120 through 50124, and these ports need to be opened between the Task Server to be used and the client computer.

Conclusion

As previously indicated, make sure you test every system defense task and job you plan to use out in your environment. It's one thing to test against one or two systems where you can manually resolve any unforeseen problems, but if a targeted collection contains many systems and the job or task as an unforeseen issue, this can cut off all these systems from the necessary access to restore network functionality. So test, test, test, and test again before deploying large jobs using System Defense network filtering.

When used properly, this tool enables administrators to remotely deal with vulnerable or infected systems remotely, and stop unauthorized network use. With System Defense enable your administrators to more quickly deal with threats, and remediate in much less time.