Formerly known as Web Admin for Windows, Real-Time System Manager provides a powerful set of functions for IT specialists. In part 5 of this article series we covered the main points for Real-Time Console Infrastructure troubleshooting. As a natural extension of RTCI, Real-Time System Manager troubleshooting is covered in this article as part 6. With an emphasis on credentials and connection methods, this article provides information to overcome the most common issues seen when using the Real-Time tab for direct, one-to-one computer interaction.

Introduction

Real-Time System Manager provides a powerful tool for directly connecting to a system agentlessly with functionality available through WMI and Intel AMT. This article covers the issues associated with general functions seen with both technologies but with emphasis on the AMT functions. The following sections cover areas of troubleshooting:

• Connection Issues
• Authentication Issues
• IDE Redirect (IDER)
• Network Filtering

Connection Issues

Under the current architecture the FQDN is the primary method for connecting and authenticating to AMT on remote systems. If the FQDN the Real-Time tab is using does not resolve in DNS, then AMT connectivity and thus functionality will not be available. FQDN connectivity issues are the number one issues we see with RTSM connections to AMT.

Invalid FQDN

To view what FQDN the Real-Time is using, use the ‘Hardware Management' node in the RTSM tree. The following screenshot shows what AMT is using:

In this example my system is in a workgroup and reported only the hostname as the FQDN, which DNS had no trouble resolving. If this fqdn is not reachable via DNS, we won't be able to connect to the AMT functionality.

NOTE: We use several methods, including IP address, for WMI. WMI functionality may show correctly when AMT is absent in this situation

Use these steps to see the FQDN is the issue:

1. Open the Real-Time tab for the AMT system you are managing.
2. Once the tree loads, open the Real-Time System manager folder, open Administrative Tasks, and click on ‘Hardware Management'.
3. Once the page loads, if AMT is missing as an available technology, take note of the name displayed as in the screenshot above.
4. Go to Start, Run, type in cmd, and click OK.
5. Type in nslookup <name displayed>. In the above example it would read:
   1. Nslookup dellvpro
6. Can DNS resolve this address? If no, we'll need to fix the issue in one of the following ways.
7. FIX DNS and/or the Altiris record: If DNS can be fixed, this is the preferred method. The difficulty is finding out why the Altiris Agent reported the incorrect record. Once DNS is fixed, have the Altiris Agent run Basic Inventory. The table location we pull this out of for management in RTSM is Inv_AeX_AC_Location, column: Fully Qualified Domain Name.
8. Use the ‘Manage' node available in RTSM (see the below screenshot): By putting in the IP address of the system, we'll use the IP to lookup the FQDN and not make any assumptions.
   
9. Update the Servers HOSTS or LMHOSTS files to contain the mapping to the invalid name. For example find the LMHOSTS file, edit it and add a line <IP ADDRESS> <FQDN>, as in this example:
   1. 10.10.10.1 Dellvpro

Real-Time unable to connect

If WMI and AMT functions are unavailable, you'll get a message when you click on the Real-Time tab indicating that the functionality isn't available. See the following screenshot:

Note: If you use another product such as Dell or HP's plug-ins to this tab, you'll simply not have the ‘Real-Time System Manager' node underneath Real-Time Consoles.

The number one reason this occurs is due to a firewall being engaged. Firewalls need to allow AMT traffic through. If a firewall is enabled, use the following details to resolve the AMT issue:

1. Create an inclusion in the firewall properties.
2. Allow the following ports, based off your environment:
   1. 16992 - For non-TLS encrypted traffic - if you are not using TLS this is the port that will be used for communication
   2. 16993 - For TLS-enabled, encrypted AMT traffic - If https is required for communication with AMT, this port will be used
   3. 16994 - For a note, AMT provisioning uses this port for sending out the ‘hello' packet during the configuration process - this will be used if you initiate a reprovision from RTSM
3. Another options is to disable the firewall when you need to manage the system via RTSM.
4. Unfortunately WMI has a known issue with the Windows firewall where the dynamic ports WMI uses after initiation will be blocked. It's a bug in WMI that has been addressed in Vista. Previous Operating Systems do not have a resolution at this time.

The other issue we've seen is where the system is simply unavailable for one reason or another. AMT is available if the system is off but still connected to the network, but WMI or if the system is unplugged from power or off the network RTSM obviously cannot function. Verify that the system is available if nothing resolves this issue.

Authentication Issues

Another common issue concerns authentication to the system via the Real-Time tab. First, let me discuss the methods RTSM uses to authenticate to a target system.

Authentication Methods

Runtime Profile - The Runtime profile contains he following information:

• All known good credentials used to connect via RTSM to a system
• The Intel SCS AMT password sent to systems when provisioning occurs
• Previously successfully used credentials from past RTSM sessions

User-defined Profiles - Profiles can be created that specifically provide credentials for the four types of technologies:

• WMI digest or Domain account
• AMT digest or Kerberos-authenticated user
• ASF digest or Domain account
• SNMP community strings

Manually entered credentials - When RTSM tries to connect, if the default profile set in the RTCI configuration fails to authenticate, the left-hand tree will still load but each node will prompt the user for credentials. A user can put in an AMT account, Domain user, or digest user that has rights on the target system. When authentication succeeds, these credentials are then stored in the Runtime Profile for the target system.

Troubleshooting Authentication

The following method will help identify issues and offer ways to work-around and solutions. These have been compiled through experience when troubleshooting issues with failed authentication with RTSM.

1. In the Altiris Console browse to View > Solutions > Real-Time Console Infrastructure > Configuration > select Manage Credentials Profiles.
2. Where does the green checkmark fall? This is the default profile that will be used when connecting via the Real-Time tab.
3. Create a new profile by clicking the blue + on the icon bar in the right-hand pane.
4. Under the Intel® AMT tab check the box ‘Enable this technology in the profile'.
5. Supply the admin user credentials set when the managed vPro systems were provisioned.
6. Under the WMI tab also check the box as above and provide a user that has admin privileges to the target system.
7. Give the profile a name and then save it.
8. Back at the main screen check the box under the ‘Default' column until the green check-mark uses your new Profile.
9. Test to see if this new profile is successful. Note that you'll need to launch IE fresh to use the new settings.
10. If it is not, try entering credentials in manually when you hit the system under the Real-Time tab. See the screenshot below for the connection icon to switch between WMI and AMT authentication. If two show in this area, both technologies are available but not authenticated.
    
11. In one case we supplied only AMT credentials in the Profile which allowed it to authenticate to AMT while a multiple protocol authentication profile failed.
12. Check the collection you are launching Resource Explorer from. Sometimes the identity of the system is incorrect. For AMT you can launch RTSM from the Provisioned collections populated with the Resource Synchronization.

IDE Redirect (IDER)

IDE Redirect allows a system to be remotely booted to a file, drive, or virtual disc. There are a number of potential issues to be aware of when working with IDER in a vPro environment. The below items include well-known issues and their resolutions.

Redirection Invalid Parameter

When initiating an IDER (IDE Redirect) session to an external source such as an .iso file, the following error appears in the console:

Power management operation failed.
Redirection session start has failed. See logs for more details.

The Notification Server log shows the following error:

Log File Name: C:\Program Files\Altiris\Notification Server\Logs\a.log
Priority: 2
Date: 3/9/2007 2:51:05 PM
Tick Count: 10617218
Host Name: <>
Process: w3wp.exe (2436)
Thread ID: 5412
Module: AltirisNativeHelper.dll
Source: RTCI.Trace
Description: RedirectionProvider::StartIDER - RedirectionProvider::StartIDER - IMR_IDEROpenTCPSession: IMR_RES_INVALID_PARAMETER

This is caused by Intel's redirection library requiring a correct floppy device to initiate an IDER session (either floppy image or real removable device). Real-Time System Manager 6.2 can work around this. If you put floppy.img file into Program Files\Altiris\RTSM\UIData folder, then the issue will not occur.

IDER or SOL Disabled

In some instances Intel vPro systems are arriving from the OEM with IDER and SOL disabled in the BIOS. When disabled, neither of these functions work from any management engine, including RTSM. Correcting this oversight is not easy, especially if the OEMs do not offer a solution by a firmware or BIOS update. Use the following method to resolve the issue:

1. Go to the Support site for the OEM for the systems.
2. Browse to the drivers and downloads section for the exact model (note that sometimes the model will differ based on possessing or not possessing vPro technology).
3. Check the firmware updates for a new BIOS.
4. Check the documentation for any new BIOS versions that include vPro to see if they've corrected this.
5. Contact your OEM if they have not and request a status!
6. The only other recourse is to develop an update yourself or manually update the settings by visiting the system.

Conclusion

This should account for the most common issues we've seen, and allow you to successfully use RTSM with AMT technology, avoiding those issues.

The ability to provide access to the Real-Time tab of Resource Manager will enable administrators to provide this valuable tool to IT specialists or Helpdesk workers. Furthermore the ability to configure access to certain functions within the console will allow administrators to grant or restrict what users can do with Real-Time System Manager. This includes WMI functionality as well as powerful AMT functionality.

Introduction

Your environment will likely have a unique set of requirements on who can access what in Real-Time System Manager. It can be as simple as two levels of workers, from an administrator to an IT Specialist, to a complex system of access rights in a multi-tiered environment tightly controlled. No matter the environment, this article provides the details to customize access to the Real-Time tab, including WMI and AMT access rights.

RTSM contains limited functionality to configure access via WMI. AMT, on the other hand, can be configured at a function-granular level. Whether you're simply trying to give users full access to RTSM, or to provide access to only certain functions, this document assists to achieve this.

NS Role Security

The first item that must be enabled is creating a role or modifying an existing role to have rights to Real-Time System Manager at the general level. Without assignment to such a role, a user cannot gain access to RTSM.

Overview

Briefly I'll explain how NS Role and Scope security work together in Notification Server. Roles give feature access rights. For example in Software Delivery Solution there's a role object labeled ‘Item Tasks - Software Delivery Wizard'. The two options allow use of the Simple or Advanced Software Delivery Wizard. Without this right, the user cannot launch the Software Delivery Wizard, regardless if they have scope rights to the Wizard and Status node in the console.

Scope security is much like the Windows File-System security model. In the Altiris Console the left-hand tree can be accessed like the file system, applying security to folders or to nodes, as opposed to folders and files. Inherence allows security to be inherited from the containing folder, on up the chain until the root node is reached.

Role Configuration

The following steps show how to create a user with RTSM permissions.

1. In the Altiris Console, browse to View > Configuration > Server Settings > Notification Server Settings > Security Roles.
2. Select an existing Role or Right-click on the Security Roles folder and choose to create a new Role.
3. Under Privileges, find the following categories and check the indicated option. After the screenshot the items are details with description of the option:
   
   1. Altiris System Privileges - Use Real-Time System Management - This is the ability to use the product at the most basic and general level.
   2. Altiris Console Privileges - View Resources Tab - For this example I'm providing the user the ability to see collections so he or she can launch Resource Manager and use the Real-Time tab.
   3. Altiris Console Privileges - View Tasks Tab - Access to the ‘Manage' node allowing launch of Resource Manager requires this privilege.
   4. Item Tasks - Real-Time System Manager - Manage - This is access to the main tree for RTSM. Most functions are covered by this option.
   5. Item Tasks - Real-Time System Manager - Password Reset - Because of the nature of this function, it has been separated out as a single security role object in Notification Server but belongs to the Real-Time tree.
   6. Item Tasks - Real-Time System Manager - Port Check - The Port Check feature is normally accessed as a separate contextual item in the right-click menu, or launch from an icon under the Real-Time tab.
   7. Item Tasks - Real-Time System Manager - Trace Route - This is treated in the same way as Port Check.
   8. Item Tasks - Real-Time System Manager - Hardware Management - This is one of the objects in the tree that provides basic hardware function, which is greatly extended if the system is Intel vPro capable and Provisioned.
4. Click the Membership tab.
5. Use the blue + icon to add users and/or groups to the Role. These can be digest users or local computer groups, or Domain users or groups.
6. Click Apply to save the Role.

Note: The users will not have access yet to the Altiris Console as the scope-level security has not been set for the new Role. Complete the below NS Scope Security section to give access to the Altiris Console

NS Scope Security

Altiris Console

For Altiris Console access, scope security must be configured before a Role can access or login to the console. The security window is the same for any node, be it a folder or otherwise. The two screenshots below show the security window and the permission selection screens:

Note: Depending on the object type, the available permissions may differ

To allow access to the ‘Manage' Real-Time Console Infrastructure Task, follow these steps:

1. In the Altiris Console, browse under View > Tasks > Incident Resolution > Tools.
2. Right-click on the node ‘Manage' and choose Properties.
3. Click on the Security tab.
4. Click the ‘Add' button.
5. Select from the list Role name of your role (+ie:+ Role RTSM Workers) and click the ‘Select' button.
6. Check the option for ‘Full Control' and click ‘Select'.
   Note: Full Control does not give the user the ability to delete or otherwise manipulate the Manage node. This node can only be accessed for the function alone.
7. Click ‘Apply' to save the security changes made.

To access Collections so the users of the role can view collections so they can use the RTSM right-click contextual menu options for a listed resource, follow these steps:

1. In the Altiris Console, browse to View > Resources > Collections.
2. Depending on what collections you want to give the user access to, browse to a containing folder or an individual collection.
3. Right-click on the folder or collection and choose Properties.
4. Click on the Security tab.
5. Click the ‘Add' button.
6. Select from the list Role name of your role (+ie:+ Role RTSM Workers) and click the ‘Select' button.
7. Check the following options:
   1. Altiris System Permissions - Read
   2. Altiris Resource Management Permissions - Read Resource Data
   3. Altiris Resource Management Permissions - Read Resource Association
8. Click Select, and then click Apply on the permissions window.

Now we have allowed the user access to certain parts of the Altiris Console so they can execute Real-Time System Manager on managed systems. To restrict access to certain parts of the RTSM console, see the previous Role section for what options are available to you.

AMT Permissions

RTSM takes advantage of powerful functionality available in Intel vPro, AMT technology. Once a user has access to RTSM, their user account, if permitted, is used to connect to the remote system by WMI. An AMT connection can either use Kerberos integration or an inputted digest user when prompted. The credentials must be specified in the destination system's AMT Profile, otherwise authentication will fail.

To configure who has rights to AMT, follow these steps:

1. In the Altiris Console, browse to View > Solutions > Out of Band Management > Configuration > Provisioning > Configuration Service Settings > Provision Profiles.
2. Double-click on an existing profile, or create a new one.
3. Click on the ACL tab.
4. Click Add to add either a digest user or to use Domain users and groups with Kerberos integration.
5. Once a user is inputted, the ‘Realms' section allows or disallows access to different AMT functions. The boxes that are of importance to RTSM are:
   1. Circuit Breaker - Now known as System Defense, or Network Filtering
   2. Hardware Asset - For power management capabilities
   3. Redirection - To allow IDE Redirection
   4. Remote Control - Allows Serial Over LAN (SOL) remote connection
   5. Event Manager - Allows viewing of AMT logs
   6. General Info - Allows viewing of AMT data on the system
6. The ‘Access Permission' dropdown should be used to select either Network Access or Any. The Local Access option gives that user rights to log into the Intel ME locally when the system boots and isn't needed for RTSM function, however if you wish to allow the user to have access to both, choose ‘Any'.
   
7. Click OK to save the changes.

To apply the updated or new profile to an AMT system Provisioning must occurred. If the system was already provisioned with this same profile previously, a reprovision will update the profile.

This will not limit access to see the functions available in the Real-Time tab for AMT, but will throw a not authorized message if an applicable function is attempted with a user who does not have the rights to execute it.

Conclusion

The Real-Time tab, a one-to-one solution for system access, data gathering, or troubleshooting, provides a powerful tool to IT administrators and IT professionals alike. Providing this ability to users you do not want to have full access to Altiris is essential for any secure environment. With the additional ability to configure granular AMT rights for vPro capable and configured systems, an administrator has the ability to get very specific on what users or groups of what rights.

I just posted a new YouTube video on my own Intel AMT 3.0 computer that runs under my television. It runs Microsoft Media Center, has 4 cores, 4 tuners, 4 hard drives, 3 Gigs of RAM, 2 DVD's... Certainly the most powerful computer I have ever owned. Most importantly, it has Intel AMT 3.0 using an Intel DQ35JO motherboard. This is very useful for me to work on Intel AMT Commander on my spare time and also to remotely manage my computer from anywhere in the world.

If you guys have your own computer project that runs Intel AMT, please let me know. Better yet, if you have pictures it would be great to share with the community.

Ylian (Intel AMT Blog)

Hi all. I wanted to announce the release of the Intel AMT DTK v0.51 on the public web site. As usual, lots of improvements have been made since the last version thanks for much testing and feedback from users. There are a few things that are particularly interesting about this new release of the Intel AMT DTK and lets get right to it:

• Build-in C# WSMAN stack. As Intel AMT is transitioning to WSMAN calls for remote managibility, adding WSMAN support into the DTK has been increasly important. In the past, the DTK made use of WinRM, a Microsoft component that needed to be installed and configured. With version 0.51 of the DTK, I build my own WSMAN stack in C# right into the DTK stack. As a result, no more dependency on WinRM at all and no more compile problems. Additionaly, the DTK is now much faster at making WSMAN calls since all HTTP requests are now pipelined, and the DTK can connect to AMT computers that have invalid TLS certificates (a warning will be displayed of course). This is big news for anyone interested in WSMAN work. If you build your own managibility solution, I suggest you look at grabbing at least that part of the DTK source code.
• Intel AMT Flash Tool. This version of the DTK adds a new Intel AMT Flash Tool. It will help users correctly setup a USB flash key so that it can be use to provision Intel AMT computers. As many of you many know, Intel AMT will in the right conditions, read a setup.bin file in a USB flash key when booted and use the information to help setup Intel AMT. The setup.bin file must be at the very start of the USB key and this new tool with help with that. The new tool is based on a similar tool that has already been released on the Intel Pro Center.
• Intel AMT Reflector tool. Another new tool is a TCP connection reflector. It's a small generic tool that accepts connections and forwards the data back to the source IP address on a target port. It's useful for accessing Intel AMT from your own computer using a reflector on a different computer. I use it for recording some of my demonstration videos, but it can also be used by agents running localy that want to re-configure Intel AMT on itself. For example, detecting an OS name change and updating Intel AMT.

Many more changes and fixes have also been done, for example the terminal now correctly detects Serial-over-LAN disconnection, etc. For a full list, the DTK includes a change log.

Intel AMT DTK v0.51x Audio Blog (.mp3)

Ylian (Intel AMT Blog)

Years before I started working on Intel AMT, designers where creating a list of usages that would be enabled by Intel AMT. The list included, I presume, usages around 3PDS, remote reboot to BIOS, disk redirection, etc. Many of the Intel AMT usages that are promoted on the Intel web site. When I started work on the DTK, a personal challenge had always been to find new ways of using existing features to do different and sometimes unexpected things. Create new usages for Intel AMT that it was never originally designed to do. I now present my top 5 abuses of existing features.

TCP-over-Serial-over-LAN. The Intel AMT serial port I am told, was originally designed as an easy way to remotely take control of the BIOS and recovery OS remotely. Designers needed a way for BIOS to be able to send test display data to a remote console. A virtual serial port was a great solution. It so happens that in the original design, this serial port was always enabled and usable, even when the normal OS was running. This allows a serial agent to talk to a console while bypassing the OS’s network stack. This is interesting on its own and I started work on a serial agent of my own. Things took a weird twist when I started sending binary data and sending files over this serial port, making it very valuable. It’s only a few weeks later that I realized I could also send TCP traffic over this serial link, making it possible to contact TCP services on the Intel AMT computer even if the network stack was disabled. A few days later, I showcased the first demonstration of VNC-over-SOL, and turning this abuse of the serial port into an instant hit. To this day, VNC-over-SOL is still, one of the most impressive demonstrations of Intel AMT.

Reverse Watchdog. When Intel sales people demonstrate Intel AMT to customers, they often get asked if you can shutdown gracefully an Intel AMT computer using Intel AMT. The simple answer was no, Intel AMT will perform a brutal shutdown or reset upon request. To perform operations like a clean shutdown or reset, sleep or hibernation requires the involvement of the OS. You could tell a serial agent like Intel AMT Outpost to perform the shutdown, but that required opening the serial connection and could be a problem if you had to shutdown many computers. I needed a way to pass a small amount of information to a running Intel AMT agent on the PC, do it using SOAP/WSMAN only and if possible get confirmation of reception. We could store the command into 3PDS and have the agent read it periodically, but 3PDS required setup and that little amount of data would have required allocation of a 4K flash page. The solution came when looking at the agent presence feature. When a console creates a new agent, the agent can now register this agent locally. The agent also get the timeout of the agent in seconds (from 1 to 65535), this would be the key. By constantly trying to register a known GUID, Intel AMT Outpost could see if the agent existed or not. If suddenly the registration works, the timeout value would indicate that type of shutdown operation to perform. Better yet, the simple fact that registration occurred changes the state of the agent to “Running”, confirming to the console that the message was indeed received. Today the Intel AMT Terminal has “Agent Commands” in the remote control that allows a user to perform soft operations when the agent is running, even if the OS network stack is not working.

Mouse over serial. A few months back I started work on a smaller version of Intel AMT Outpost called Intel AMT Guardpost. The idea was that if a serial agent was going to be useful, it was going to need to run on a recovery OS, run in the background with no dependencies and with as little footprint as possible (Is it not annoying to have all there background processes running?). The C/C++ version of Intel AMT Outpost was on its way. One feature I always wanted to work on was a remote Windows command prompt; it took over a week to finally pull this off. I could now remotely shell to DOS and perform basic command line operations. I could also enter the command like editor with the “Edit” command at which point, the temptation to support the mouse-over-serial-over-LAN was a must have. Using the binary serial protocol, I added the support to the terminal in a few hours. To this day, it’s still a fun and amazing demonstration of outstanding remote manageability.

IDE-R within the OS. A few days after first enabling IDE-R within Intel AMT Commander, I stumbled upon something I had not noticed before. If an administrator where to start IDE redirection and the OS was to re-scan its plug & play devices, the additional floppy and CDROM drive would show up in Microsoft Windows. This was immediately interesting since transferring files over the serial port was limited to 115kb/sec a very slow speed in today’s world. With IDE-R, you can copy files at around CDROM 4x speed on a local network. All I needed was a way for Intel AMT Outpost to cause the OS to rescan its plug & play devices. A few hours later the “HWRESCAN” command was built and for the first time, an administrator could mount a CDROM remotely and install a patch as high speed without ever using the OS’s network stack. This feature also turned out to be an excellent compliment to VNC-over-SOL.

Fast data path using IDE-R. This is not an idea I never built into the DTK, but I wanted to add it to this list since it would also be an interesting was to use existing features in new ways. The serial-over-LAN feature turned out to be extremely valuable, but it is also slow. Serial ports are very inefficient. One way someone could speed things up is to use IDE-R as a fast by-pass to the OS. An administrator would mount a virtual floppy disk drive containing a single file. This file, would not really exist, it would contain different data each time it was read, making it possible to send data to an OS agent thru Intel AMT at much higher speeds. Also, since the floppy is a read/write device, the agent could write into the virtual file data that it wants to send to the console. It would be quite a bit of work to pull this off, but it certainly seems possible. Someone would just have to know the internal format of an .img file.

That’s my top 5. I realize this is probably a rather advanced blog article, but this is proof that you can have a lot of fun to any technologies.

Ylian (Intel AMT Blog)

Check out this excellent example of activating vPro and a usage model in a creative way to achieve a relitivly quick Break Even point and Return on Investment .... vPro ROI Analysis- Reducing IT Resource Needs & Service Costs Through Intel Core2

Abstract: Intel's investigation into the savings offered by Intel vPro technology was conducted in a globally distributed environment. The test environment consisted of 39 training rooms in Asia, the United States, and western Europe, with approximately 800 PCs, of which 300 systems were PCs with Intel vPro technology.

• Brian Brougham

This is 1/10 System Defense policy tests I worked on. This test has four systems: three servers & one AMT 3.0 client. I run pings from each server to the vPro and from the vPro (via RDP session) to each server. Then I block all IP except from one server. I lose connectivity including the RDP session but can still manage the system to remove the policy.