Rob Haverty
Technical Evangelist
Microsoft® Windows® User Interface (UI) Automation is the new accessibility framework for Microsoft Windows, which will ultimately replace MSAA, and is intended to address the needs of assistive technology products and automated testing frameworks by providing programmatic access to information about the user interface. UI Automation will be fully supported on Microsoft Windows XP and the next version of the Windows desktop operating system code-named “Longhorn”. Support will continue for MSAA in the Longhorn timeframe as well. UI Automation will also be made available for cross platform development via a royalty-free license.
In 2003, Microsoft Corporation commissioned Forrester Research, Inc., to conduct a study to measure the potential market of people in the United States who are most likely to benefit from the use of accessible technology for computers (see http://www.microsoft.com/enable/research/). Accessible technology enables individuals to adjust their computers to meet their visual, hearing, dexterity, cognitive, and speech needs. It includes both accessibility options built into products as well as specialty hardware and software products (assistive technology products - see http://www.microsoft.com/enable/at/types.aspx) that help individuals interact with a computer. Overall results show that 57% (74.2 million) of computer users in the United States are likely or very likely to benefit from the use of accessible technology due to experiencing mild to severe difficulties or impairments (see http://www.microsoft.com/enable/research/computerusers.aspx).
With the current technology, assistive technology vendors (ATVs) are required to use many different approaches to obtain and present information about the UI to the end user, thus spending an inordinate amount of time and resources on providing this basic information. With such a large percentage of users needing accessible information it is becoming increasingly important to make it easier for an ATV to programmatically obtain information about the UI.
The new accessibility model for Windows, UI Automation, is designed to provide a single reliable source of UI information to assistive technology products and automated test scripts. It provides programmatic access that allows automated tests to interact with the UI and allows assistive technology products to provide information about the user interface to their end users. UI Automation also provides means for manipulating the UI.
UI Automation has two main audiences: UI Automation providers and UI Automation clients. UI Automation providers are applications such as Microsoft Word, or third-party applications. UI Automation clients are assistive technology applications, such as screen readers, screen enlargers, or others. Automated test scripts can use UI Automation for automated testing and are also considered clients in the UI Automation framework.
This article includes information on the namespaces that the UI Automation framework uses as an organizational system - a way of presenting program components that are exposed to other programs and applications - as well as information on the following UI Automation features:
Please note that the description of UI Automation presented here is preliminary and is subject to change.
The following table lists the namespaces used in the UI Automation framework, as well as the audience that uses each namespace.
| Namespace | Audience used by |
|
System.Windows.Automation - see [1] |
Clients for finding automation elements, registering for events and working with control patterns. |
|
System.Windows.Automation.Provider - see [2] |
Providers for implementing UI Automation on “Avalon” controls or applications. |
Standard Windows programming has always exposed the relationship between elements in the user interface in a parent/child relational structure. UI Automation clients view the UI elements on the desktop (a UI element would be, for example the “OK” button) as a set of automation elements which are arranged in a tree structure. Automation elements are referenced through a common object (the AutomationElement - see [3]) to enable consistent information, interaction, and a navigation model. UI Automation unifies disparate UI Frameworks such as Avalon, Trident, and Win32 so that code can be written against one API rather than several.
Within the automation tree there is a root automation element which represents the current desktop and whose children represent application windows on the desktop. Each of these child elements can contain automation elements representing the UI elements comprising their UI, such as menus, buttons, toolbars, and others. Each piece of UI can contain automation elements representing their content, such as menu items, or list items. Even a button, which does not contain any items, may have children automation elements which represent the basic UI components that comprise the button, such as text and rectangles.
It is important to note that the automation tree is not a fixed structure. For performance reasons it is built on demand starting with an automation element which the client specifies.
The automation tree can be filtered to create customized views of the tree which contains only those automation elements that are relevant for a particular client. This approach allows clients to customize the structure presented through UI Automation to their particular needs. Default views are provided by the UI Automation framework, but clients can also define custom views.
The raw view of the automation tree is the full tree of elements for which the desktop is the root. The raw view closely follows the native programmatic structure of an application and therefore is the most detailed view that is available. It is also the base on which the other views of the tree are built. Because this view depends on the underlying UI framework, the raw view of an Avalon button will have a different view than a Win32 button.
The Control View of the automation tree simplifies the assistive technology product’s task of describing the UI to the end user and helping that end user interact with the application because it closely maps to the UI structure perceived by an end user.
The control view includes all elements from the raw view that an end user would understand as interactive or contributing to the logical structure of the control in the UI. Examples of elements that contribute to the logical structure of the UI but are not interactive themselves are item containers such as list view headers, toolbars, menus, and the status bar. Non-interactive elements used simply for layout or decorative purposes will not be seen in the control view. An example is a panel that was used only to layout the controls in a dialog but does not itself contain any information. Non-interactive elements that will be seen in the control view are graphics with information and static text in a dialog.
The content view of the automation tree contains elements that convey the true information in a user interface. For example, the values in a drop-down ComboBox will appear in the content view because they represent the information being used by an end user. In the content view, a ComboBox and ListBox are both represented as a collection of items where one or more items can be selected. The fact that one is always open and one can expand and collapse is irrelevant in the content view because it is designed to show the data, or content, that is being presented to the user.
The UI Automation framework also allows a client to create a custom view of the automation tree by specifying the desired match conditions and scoping information. This also allows clients to build their own interaction models for the application using just the data that they need.
The following example compares the control view and content view of the automation tree for the same application:
|
Automation Tree (Control View) The Control view of WordPad shown from the Desktop has the following structure:
|
Automation Tree (Content View) The Content view of WordPad shown from the Desktop has the following structure:
|
UI Automation uses control patterns to express the functionality contained in each control. UI Automation differentiates between what a user would call the control and what can be programmatically done with the control by using control patterns to express only functionality, separate from the type or name of that control.
Providers implement control pattern interfaces on UI elements. Control pattern interfaces are found in the System.Windows.Automation.Provider namespace [2] and have names that include the suffix “Provider” (for example, IScrollProvider [4] and IInvokeProvider [5]).
Clients access methods and properties of control pattern classes and use them to access information about a UI element, or to manipulate the UI. These control patterns classes are found in the Systems.Windows.Automation namespace [1] and have names that include the suffix “Pattern” (for example, InvokePattern [6] and SelectionPattern [7]).
Control patterns may define the structure, methods, properties, and events supported by a control:
Control patterns relate to UI as interfaces relate to COM objects. In COM, you can query an object to ask what interfaces it supports, and then use those interfaces to access functionality. In UI Automation, clients can ask a control which patterns it supports and then interact with the control through the properties, methods, events, and structure of the supported control patterns. For example, providers implement IScrollProvider [4] for a multi-line edit box. When a client detects that a UI element supports ScrollPattern [8], it can use the properties, methods, and events from that class to gather scroll-specific information or programmatically scroll its content to a new position.
Controls can support zero or more control patterns. For example:
To define the full set of functionality for a control, providers implement multiple control patterns. The following table shows more examples of standard controls and the control patterns they support.
| Control Type | Relevant Control Patterns |
| Button | Invoke or Toggle |
| CheckBox | Toggle |
| ComboBox | ExpandCollapse, Selection |
| Edit | Value, Text |
| List | Selection |
| ListItem | SelectionItem |
| Tree | Selection |
| TreeItem | SelectionItem, ExpandCollapse |
The following table lists some of the key control patterns and their classes and interfaces.
| Control Pattern |
Client-Side Class - details available via [1] |
Provider-Side Interfaces - details available via [2] |
| Dock | DockPattern | IDockProvider |
| ExpandCollapse | ExpandCollapsePattern | IExpandCollapseProvider |
| Grid | GridPattern | IGridProvider |
| GridItem | GridItemPattern | IGridItemProvider |
| Invoke | InvokePattern | IInvokeProvider |
| MultipleView | MultipleViewPattern | IMultipleViewProvider |
| RangeValue | RangeValuePattern | IRangeValueProvider |
| Scroll | ScrollPattern | IScrollProvider |
| ScrollItem | ScrollItemPattern | IScrollItemProvider |
| Selection | SelectionPattern | ISelectionProvider |
| SelectionItem | SelectionItemPattern | ISelectionItemProvider |
| able | TablePattern | ITableProvider |
| TableItem | TableItemPattern | ITableItemProvider |
| Text | TextPattern | ITextProvider |
| Toggle | TogglePattern | IToggleProvider |
| Transform | TransformPattern | ITransformProvider |
| Value | ValuePattern | IValueProvider |
| Window | WindowPattern | IWindowProvider |
UI Automation properties are a set of standard properties that expose information that is important to assistive technologies. Frequently, this information is exposed differently for each UI framework.
The following table shows how one standard UI Automation property maps to multiple property names in other UI frameworks.
| UI Automation Control Type | UI Framework | Framework Property | UI Automation Property |
| Button | Avalon | Content | NameProperty |
| Button | Win32 | Caption | NameProperty |
| Image | Trident/HTML | ALT | NameProperty |
By implementing UI Automation, providers map unique UI framework properties to standard UI Automation properties. When this done, it allows clients to query for property information using one API call for a UI Automation property.
UI Automation offers an event mechanism similar to WinEvents in the current Windows platform. However, unlike WinEvents, UI Automation’s events are not based on a broadcast mechanism. Clients register for specific event notifications and can request that specific UI Automation properties and control pattern information be passed into their event handler. This provides a much more powerful mechanism than WinEvents because clients make fewer calls to retrieve the information they require, which results in fewer cross-process calls, and therefore better performance. UI Automation provides event notifications for logical structure changes, control pattern changes, focus changes, property changes, and multimedia events.
UI Automation is a key part of the new accessibility model for Windows, gathering information about and interacting with the UI. Adoption of this technology will improve product quality for Windows applications and reduce the time to market for assistive technology products. Additionally, by implementing UI Automation, ATVs reduce the resources invested in obtaining UI information allowing them to improve and expand on the products that they offer.