Distal object perception through haptic user interfaces for individuals who are blind

Kanav Kahol, Sethuraman Panchanathan

Center for Cognitive Ubiquitous Computing, Arizona State University
PO BOX 878309, Tempe, AZ 85281, USA.

E-mail: kanav@asu.edu

Introduction and problem description

This paper proposes design, development and testing of an assistive device for distal environment perception for individuals who are blind through a haptic user interface (touch-based interface).

Distal perception can be defined as the ability and the act of perceiving environment beyond the physical reach of hands and legs: a region known as kinesphere [3]. It is a challenging activity for individuals who are blind. A number of assistive devices for distal perception have been developed over the years that communicate information to blind individuals through audio signals that either interfere with their existing perceptual capabilities, or give redundant auditory cues that the individual can already discern from the environment with out any assistive device [2]. Haptic sensory feedback is less intrusive and can be complementary to an individual's perceptual capabilities. The problem of distal environment perception through haptic user interface can be divided into three subproblems.

1. What to perceive? This depends on the final modality through which the information is being presented. In the case of the haptic user interfaces the modality of communication is haptics. In order to ensure optimal performance of the system, it is important to extract haptically salient features from the distal environment. Hence instead of extracting visual textures from distal environment, the system would now require to estimate haptic textures. A systematic approach to this problem would require psychological investigations into what features are haptically salient?
2. How to perceive? This question pertains to sensing methodology for distal environment. While various types of sensing methodology such as RFID, Infra-red are available, visual sensors are the most economical and arguably the most efficient way of sensing distal environment. Computer vision algorithms do not perform well in real environments with a large number of objects and classes. The key then, is to find features that are invariant to various types of noise and limiting the number of features and their quantization levels.
3. How to present the detected distal environment features through a haptic user interface? The detected features need to be systematically presented to the user. The presentation scheme, should present distal environmental objects in a veridical manner and simulate a user's interaction with the real object. [1] Rendering objects in a realistic manner where virtual objects feel like their real counterparts is by far the most desirable form of rendering. However, presently available haptic interfaces do not mimic the scale of the human haptic system and are limited in the quantity and quality of feedback they can provide [2]. This severely limits the usability and efficiency of the haptic interfaces. The key challenge then is to identify and invent new techniques for presentation of haptic features that work in conjunction with the realistic rendering technique.

Related work

Some attempts have been made to design haptic-based assistive devices specifically for individuals who are blind that convert visual data into tactile data. The Tactile Vision Sensory Substitution System (TVSS) (1970) was one of the first attempts to develop haptic assistive devices that convey spatial environmental information [1]. It converted an image captured by video camera into a low-resolution tactile image, and displayed that image using a matrix of 400 vibrotactile elements (20 rows and 20 columns of one millimeter diameter solenoids). However, TVSS have not been widely accepted by individuals who are blind. We surmise that the poor performance of sensory substitution devices can be attributed to the fact that humans perceive the world at the perceptual level rather then the sensory level. We perceive the world in terms of surfaces and objects. However, the TVSS essentially communicates pixel-level information through the tactile modality imposing a steep learning curve on the user to tactilely interpret pixel-level data.

In addition, several virtual reality based systems have been proposed for haptic perception of virtual objects. Such systems however suffer from two limitations [2]:

1. They provide a single point contact based interface. This is an extremely limiting method of haptic perception. Glove based interfaces can provide multipoint contact sensation but the algorithms are not capable of providing real-time feedback; and,
2. They present data in an exocentric reference frame that further limits haptic perception. Research has shown that humans tend to perceive objects in egocentric reference frames by using both the hands. The present single-point interfaces for which much of the technology has been developed, cannot present data in egocentric reference frames and force users to sense objects with one finger without holding it. This is a very limiting mode of haptic sensing [3].

Goal of the research

The primary goal of my research is to design, develop and test a haptic user interface for distal environment perception. Three parallel research efforts are being conducted to achieve this goal.

The first research effort is designed to systematically explore the haptic space and the saliency of features that are extracted from haptic exploration: the act of moving hands over objects to perceive haptic features. In this proposal, analysis of exploratory actions will be the basis of computational algorithms that can automatically perceive the haptic features of a stimulus. In past experiments, we have shown that haptic exploratory procedures can be automatically recognized and further these exploratory procedures and their parameters provide cues into the type of feature being perceived and its quantization level [2]. For example, lateral movement of index finger is used to perceive texture, and high speed of the movement with varying direction indicates a rough texture. These results allow capture of exploratory procedures when a user perceives a real object. The exploratory procedures can be used to parameterize the object perceptually. Secondly we will perform multidimensional scaling on a database of objects for understanding haptic space and its important dimensions. This is a common psychological technique used to explore perceptual spaces and reveals the basis of judging similarity between objects [3].

The second research effort is designed to convey information about objects haptically to the user. In this thesis, we intend to use three complementary approaches for haptic presentation of object. Overall object shape, size, material and texture information are presented in our system through tactile cues each of which is associated with haptic features and their quantization levels. This allows presentation of the concept of an object to the user in a fast and efficient manner. For example, consider the following statement and visualize an object with these features: The object is a glass made of plastic, is of medium size and its texture is smooth. Moreover the objects base is smaller than its rim and its vertical surface from rim to the base has no curvature. A majority of readers would concur with the view that this statement if accurate provides enough information to invoke a mental image of the object. Cueing uses simple vibratory pulses to invoke a mental image of a virtual object. The surface of the object is rendered realistically as a raised surface in an augmented reality environment. We propose to use a combination of gloves to hold the surface and Phantom joystick for interaction with the virtual surface, as it will allow perception in an egocentric reference frame. The computational complexity of this simulation is limited as compared to the complexity of rendering the entire object realistically. Haptic visualization will be used to convey visual features such as color.

The third research effort will consist of design of algorithms that extract perceptual parameterization of objects from visual data. It is important to note that our system does not rely on object recognition rather just estimating how it feels perceptually. For example instead of recognizing texture, the system simply needs to determine if the object is rough or smooth. This limits the number of classes that vision algorithms have to classify stimuli in, and hence perform well in real environments. This has been confirmed in initial experiments.

Current status of research and results

Our experiments on hand movement analysis revealed that it is possible to analyze hand movements automatically and predict the feature user is trying to perceive and the quantization level of the feature with very high accuracy. This system enables an unobtrusive analysis of a user's haptic perception process and develops contextual models of his/her style of perception. Further the multidimensional scaling revealed that shape, size, texture/hardness and curvature of vertical regions are the important dimensions based on which humans analyze haptic objects. These dimensions were used in our system to index the objects.

We designed an initial prototype of the tactile cueing system on which a 10 users, trained for half an hour with 5 objects. The cueing methodology sends shape, size, texture, curvature and material cues to different fingers of the hand in a serial order that mimics haptic perception of features in real world. In the test phase, the users were able to recognize a set of 107 objects through simple cues with 100% accuracy within 2-7 seconds. High accuracy was achieved in object recognition tasks using cues. Other tests have been conducted to study the effect of noise on cueing accuracy. Results show that cueing is robust to noise and users can be engaged in other tasks while still recognizing cues and objects.

A haptic environment has been coded that allows user's to hold and manipulate a virtual surface through a glove and feel the surface through a Phantom interfaces. Initial results show that this methodology allows for better recognition and perception. We have developed a system that presents color by converting color channels (R,G,B) into texture that a user feels. Initial results have shown that blind individuals including congenitally blind individuals can recognize color with 90-100% recognition accuracy after one learning phase. These results are especially encouraging as our system allows learning of color and their similarities through texture rather than simply stating the color name. We propose to test this system extensively.

We have also designed an initial prototype of computer vision algorithms that can determine perceptual categories of an object. This system achieves a recognition accuracy of 100% under real environment condition.

Open issues and future research

The system that automatically determines haptic features of the object by tracking hand movements is still under development. A strategy needs to be formulated for assembling the entire system and usability testing. Another open question is that of determining if the proposed task division between the three rendering methodology is optimal and whether contextual models need to be designed for optimally distributing tasks between the three rendering methodologies. It is expected, that once the system is developed, it will allow individuals who are blind, to systematically explore distal environment and obtain information about distal objects within a time period of 1 minute.

References

1. Hatwell, Y., Streri, A., Gentaz, E., Touching for knowing: cognitive psychology of haptic manual perception. Amsterdam; Philadelphia: John Benjamins Pub. 2003
2. Kahol, K., Tripathi, P., Panchanathan, S., and Goldberg, M., Formalizing cognitive and motor strategy of haptic exploratory movements of individuals who are blind. In: Proceedings IEEE HAVE 2004 25-30
3. Kahol,K. http://www.public.asu.edu/~kkahol 2005