Interactive Sonification of Geo-referenced Data

Haixia Zhao

Institution:

Department of Computer Science & Human Computer Interaction Laboratory,
University of Maryland, College Park

Email:

haixia@cs.umd.edu

Abstract

This paper describes an investigation of using interactive sonification (non-speech sound) to present geo-referenced statistical data to vision-impaired users for problem solving and decision making. By working with vision-impaired users, the work will identify effective interaction and sound designs for geo-referenced data, and derive principles that can guide general interactive data sonification designs for auditory information seeking.

Introduction

For people with vision impairment, audio is an important alternative or supplementary information channel. The current support for vision-impaired users to access geo-referenced statistical data (e.g., the population distribution or election results of US states) relies on screen readers to linearly speak the data presented as tabular records. Such linear textual presentation makes it hard for blind users to locate a specific data item and understand data trends, especially in the geographical context. Sonification is the use of non-speech audio to convey information [4]. Effective data sonification can help vision-impaired users to explore data collections for problem solving and decision making. As a result, it promotes equal working opportunities.

In my thesis research to improve vision-impaired users’ access to geo-referenced data, I first propose an Action by Component taxonomy to guide interactive sonification designs that are helpful to exploratory tasks. Guided by the taxonomy, I then systematically explore the design space for geo-referenced data. Through user studies with both vision impaired and blindfolded sighted users, guidelines are being derived and insights are obtained regarding people’s abilities to perceive complex information through interactive sound. Third, a customizable tool is being developed both for vision-impaired users to explore geo-referenced data collections and for researchers to investigate new sonification designs.

Related work

Ramloll et al. [6] found that using non-speech sound significantly improved vision-impaired users’ comprehension of 2-D numerical tables. Research in [2, 3] showed that users can interpret a quick sonified overview of bivariate scatterplots and 2-D line graphs with one or two data series. Alty and Rigas [1] found blindfolded sighted users can recognize simple 2-D graphical shapes presented by musical pitches tracing the outlines. Meijer [5] sonified images with time-multiplexed sound. Several data sonification toolkits have also been developed, such as Sandbox [7]. My work is distinct from previous ones in the emphasis on supporting task-oriented user interactions with the data instead of passive listening.

Taxonomy of interactive sonification

The Action by Component taxonomy includes a set of Auditory Information Seeking Actions (AISA) to interact with the data and a set of Design Components involved in the interaction process. An exploratory data analysis task can be accomplished through a sequence of AISAs. For example, a gist gives a quick grasp of the overall data trends and patterns. Users can navigate the data collection to closely examine portions of interest, and may need to situate self during the navigation. Searching and filtering are used to seek data items by query criteria. Select is to collect data items for later revisit. Details-on-demand gives detailed information about data items.

Each AISA is an interaction loop of the user issuing a command and the system giving an auditory feedback. A visual interface allows users to directly manipulate any part of the data display. However, in auditory displays, information is presented over time. The system needs to help users continuously explore, construct and maintain a mental representation of the data. Interactive sonification designs need to consider the following components. An Abstract Object is a representation form of the data items and their relations, such as a scatterplot or a map. A Navigation Structure in an abstract object defines the paths by which users can move around in the data. Users specify their interaction intentions via Input Devices and receive Auditory Feedback about the data items of interest.

Design space for geo-referenced data

Guided by the taxonomy, the design space for geo-referenced data is being systematically explored. The choice of Abstract Object needs to reflect the data relations that are most helpful to the task. Maps and tables are currently being explored. Navigating maps with irregularly shaped and sized regions imposes special challenges. Good navigation designs should maximize user orientation to help constructing their mental representations of the data space. Regarding Input Device and Auditory Feedback, a range of choices is being investigated, in order to reduce the dependency on special devices and to provide a thorough understanding of human perceptual abilities under varieties of situations. The investigation expands from standard input devices (e.g. keyboard) and MIDI sound to special devices (e.g. tablet) and advanced virtual spatial sound techniques. Proper sequencing of multiple data items is a challenge, since there is no natural mapping from geo-referenced data to the time dimension. For a large number of geographical regions, data aggregation may be necessary before the data is mapped to sound.

The design options are being systematically evaluated through user studies, in the forms of both controlled experiments and observational studies. Three blind people work with us regularly and we will reach more blind people through the National Federation of the Blind in Baltimore and online communities.

Customizable tool

Based on the design space, a customizable tool iSonic (Interactive sonification) is being developed for both researchers to investigate new sonification designs and for vision-impaired users to explore geo-referenced data collections. By providing synchronized visual and auditory displays, iSonic will allow low vision users to use both their vision and hearing, and improve the collaboration between vision-impaired users and sighted users.

Status

Several studies with both blindfolded sighted users and blind users have already been done to compare different design options in the design space and to examine users’ abilities to recognize the geographical distribution patterns of a 5-category data set [8, 9, 10]. The studies compared two abstract objects (map and table), five navigation methods using either a keyboard (e.g., Figure 1c, 1d) or a tablet, and three auditory feedback encodings. For example, in one of the interfaces, spatial sounds were tied to the map to create the effect of a virtual map surrounding the user at the center (Figure 1a). For each region, a string pitch is used to indicate its geo-referenced data. Using a keyboard, users can start an automatic spatial sweep (Figure 1b) from the west to the east to listen to a 25-second gist of all the regions, navigate the map to explore individual regions (Figure 1c), and request spoken details of interesting regions. In another keyboard-based interface, users can use a 3 x 3 numeric keypad to explore nine fixed map ranges (Figure 1d). The exploration can be done recursively. In a tablet-based interface, users can drag their fingers or press spots on a smooth surface touch sensitive tablet to explore. Studies have shown that subjects were able to perceive patterns on both familiar and unknown maps. Some designs (e.g., absolute navigation via a tablet or a keyboard (Figure 1(d))) were better than the others (e.g., relative navigation via a keyboard (Figure 1(c))) for pattern recognition and learning new map geography.

Acknowledgments

I thank my advisor Ben Shneiderma and co-advisor Catherine Plaisant. The work is supported by NSF grants EIA 0129978 & ITR/AITS 0205271 and US Census Bureau.

References

1. Alty, J. L., Rigas, D. Communicating graphical information to blind users using music: the role of context. Proc. ACM SIGCHI 1998
2. Brown, L., Brewster, S.A. Drawing by ear: interpreting sonified line graphs. Proc. ICAD 2003
3. Flowers, J.H., Buhman, D.C., and Turnage, K.D. Cross-modal equivalence of visual and auditory scatterplots for exploring bivariate data samples. Human Factors, 39, 3 (1997), pp 340-350
4. Kramer, G., Walker, B., Bonebright, T., Cook, P., Flowers, J.H., Miner, N., Neuhoff, J., et al. Sonification report: status of the field and research agenda (1997). Available at: http://www.icad.org/websiteV2.0/References/nsf.html
5. Meijer, P.B.L. An experimental system for auditory Image representations. IEEE Trans. Biomedical Engineering, 39, 2 (1992), 112-121
6. Ramloll, R., Yu, W., Riedel, B., and Brewster, S.A. Using non-speech sounds to improve access to 2D tabular numerical information for visually impaired users. Proc. BCS IHM-HCI 2001
7. Walker, B. N. and Cothran, J. T. Sonification Sandbox: a graphical toolkit for auditory graphs. Proc. ICAD 2003
8. Zhao, H., Plaisant, C., Shneiderman, B., Duraiswami, R. Sonification of geo-referenced data for auditory information seeking: design principle and pilot study. Proc. ICAD 2004
9. Zhao, H., Smith, B.K., Norman, K., Plaisant, C.,Shneiderman, B. Interactive sonification of choropleth maps: design and evaluation. IEEE Multimedia, Apr-June (2005)
10. Zhao, H., Plaisant, C., Shneiderman, B. “I hear the pattern” - Interactive sonification of geographical data patterns, Proc. ACM SIGCHI Extended Abstracts, 2005

Project web-site

Available at: http://www.cs.umd.edu/hcil/audiomap