DESIGNING MULTISENSORY EXPERIENCES FOR USERS WITH DIFFERENT READING ABILITIES VISITING A MUSEUM

Leandro Soares Guedes, Università della Svizzera italiana, Switzerland leandro.soares.guedes@usi.ch

Abstract

My work explores how technology can support different forms of reading and sense-making of text and multimedia content before, during, and after a museum visit. This paper will present AIMuseum, our pilot study, and how I plan my research. My main contribution is planned to be on the design, implementation, and evaluation of tools to support reading while catering for different abilities.

Introduction

Text here My Ph.D. thesis is part of the project BEST - Beyond Screen Readers and Alt Text - Designing Multisensory Alternative to Text for Different Reading Abilities. At project BEST, we have two sub-projects: EPPics (Enhanced Personalised Picture Stories) and Muses (Multisensory Storytelling).

My research explores how technology can support different forms of reading and sense-making of text and multimedia content before, during, and after a museum visit, making visiting a museum a more rewarding and memorable experience. Reading is a complex process that includes seeing the character (accessibility via visibility), being able to distinctly identify each character from the other (recognition via legibility), and finally making sense of the characters as a group (interpretation via readability). This work aims to provide useful and usable assistive tools for people with different reading abilities for a rewarding reading experience.

We believe that adequately designed multisensory representations can serve those experiencing a variety of reading difficulties, either temporarily or permanently. We plan our main contribution to the design, implementation, and evaluation of tools to support reading while catering to different abilities.

Our target visitors are people with different abilities who could not take advantage of the usual supports provided in museums, mainly adults with mild cognitive disabilities. These include audio guides, labels, and written signs, and multilingual textual explanation cards. We want to understand how technology can make people enjoy a visit even if they (for different reasons) could not read, make sense of available instructions and clues, and move around comfortably in the museum space.

We will also involve experts in special needs education to devise the most appropriate forms of communication for each visitor to match their abilities, level of previous and desire for additional knowledge, the overall purpose of visit, and specific interests.

We use assistive technology to help people maintaining and improving their functioning and independence. Screen readers belong to these tools as they support blind, visually impaired, or learning disabled in tasks involving reading. By interpreting what is displayed on a device, screen readers can provide users with a new representation accessible to them, often as in text-to-speech. We are also exploring Augmented Alternative Communication (AAC) potential for devising dialogues of the right complexity to stimulate and individually engage visitors.

Likewise, Alt Text is descriptive text introduced in 1995 with HTML 2.0, alternative descriptive writing to describe some visual content that, for some reason, permanent or temporal, would not be accessible to users. It has soon become used and read aloud by screen readers.

Related works and Discussion

Research nowadays is moving from designing for people with disabilities to recognizing people with disabilities as equal partners in design and research [8] [13]. As this change occurs, there is a need to reconsider existing approaches to how we engage, interact with, and refer to research participants. Disability and mild cognitive impairment have been studied in Health [10] [17] [18] and Technology related works [14] [12] [9].

Morris et at. [16] revisited the concept of alternative text, descriptive text associated with images in HTML and other types of documents, to provide a rich reading experience to blind and visually impaired users. Their study's primary outcome is a taxonomy of 5 categories (interactivity, stability, representation, structure, and personalization) to define the design space for producing prototypes of innovative descriptions of visual content in a multisensory way.

Multisensory storytelling is used in preliminary work to support learning for people with intellectual disabilities was made by Matos et al. [15]. Their results showed an overall improvement regarding the participant's memory when using the very multisensory contents. Children with special needs specifically were also part of works, from Virtual Reality [11] to Multisensory Participatory Design [19].

Reading in its many dimensions is very important for people with intellectual disabilities. Sighing words by using pictures identification is a common, basic approach [2]. Working on legibility via phonetics can support the ability to identify words [6]. Considering readability and comprehension, Bruner [3] argues how being able to understand stories describing real-life situations helps people deal with events in their lives.

When considering readers experiencing cognitive difficulties, Chen et al. [4] propose TriAccess, a system providing physical, sensory, and cognitive support to readers with special needs and a promising follow-up system [5] still to be tested appropriately.

Reading together with members of the same family can be a social opportunity too. ALLT [1] is an e-reading system to provide support to readers with different abilities such as adults with age-related impaired eyesight and children still developing and mastering reading skills.

Snelgrove et al. [20] explore collaborative read aloud to involve partially sighted users. Likewise, Augmentative and alternative communication [7] is used to study the Design Space of AAC Awareness Displays [21].

Stage and Status of the Research

As a pilot study, to understand accessibility concepts and to help including people on museum visits, I developed AIMuseum, an Accessible and Inclusive application for enhancing museums and exhibitions with Augmented Reality and Screen Readers. This work was published [22] as a full paper at ICCHP 2020 - 17th International Conference on Computers Helping People with Special Needs.

As part of the first year of Ph.D. studies, I collected and organized related works, evaluated AIMuseum, and participated in courses and conferences.

For the next three years of work, I plan to meet two different groups (children and adults with mild-cognitive disabilities) of local users. Co-design new solutions with them, develop new tools, evaluate them, and share the outcomes to this research community.

Development and Methodology

AIMuseum aims to facilitate access and interaction with cultural environments for people with different abilities, combining the use of technologies with local museums, artworks, and exhibitions.

On Fig. 1 AIMuseum is being executed in a smartphone, we point to a QR-code associated with the picture, and the application generates the picture and image description on the screen, running the screen reader to read it automatically. AIMuseum's QR codes can have different sizes and be positioned in various exhibitions to help people with different abilities and caregivers.

The work was evaluated with 38 volunteer users, ranging from 16 to 41 years old. Five declared having one type of disability: three participants with low vision, and two participants with reading disorders. We tested with disabled and non-disabled users to provide and evaluate inclusion and interaction. Unfortunately, due to the COVID-19 outbreak, we were not able to have more evaluation sessions. Users interacted with the application and answered a questionnaire. The results showed a positive experience and improved the users' interest in the artworks and their additional information.

AIMuseum is just a very initial step towards developing tools for supporting reading before, during, and after the visit, to suit individual needs and preferences. By considering a non-formal context, different combinations of age (from children to adults), and abilities, we aim to produce and test a series of prototypes to support various forms of reading with or without the help of their caregivers and educators. Our users will take an active role in critical collaborative design.

Each person has its challenges, and the system should respond to the different needs of the users. By following a developmental evaluation approach, we plan to run a series of experiments to test several factors: the impact of different read-aloud voices, e.g., male and female, different speech complexity, and contextualization.

We already know from previous observations with a similar group of museum visitors that describing items in the museum with a narrative that references their memories and past experiences can improve their involvement and the overall quality of their visit. Besides, we have witnessed high engagement and enjoyment of the tour by providing our visitors with material to prepare beforehand. We want to further explore these findings and work towards the design of tools to support more rewarding experiences.

Final considerations

We believe that this thesis can contribute to accessibility for people with different abilities. We plan to increase our multisensory experience to fulfill different needs, considering each person's different abilities. For future versions, we plan to co-design and increase the number of people in our evaluation. Our collaboration with a local museum and a special school provides us with access to co-designers to help us in our quest.

Acknowledgments

I would like to thank Dr. Monica Landoni for advising me, SNSF for funding the BEST project, IFMS, and USI for Institutional support.

References

1. Abbas Attarwala, Cosmin Munteanu, and Ronald Baecker. 2013. An Accessible, Large-print, Listening and Talking e-Book to Support Families Reading Together. In Proceedings of the 15th International Conference on Human-computer Interaction with Mobile Devices and Services (Munich, Germany) (MobileHCI ’13). ACM, New York, NY, USA, 440–443. https://doi.org/10.1145/2493190.2494658
2. Diane M. Browder, Shawnee Y. Wakeman, Fred Spooner, Lynn Ahlgrim-Delzell, and Bob Algozzinexya. 2006. Research on Reading Instruction for Individuals with Significant Cognitive Disabilities. Exceptional Children 72 ,4 (2006), 392–408. https://doi.org/10.1177/001440290607200401 arXiv: https://doi.org/10.1177/001440290607200401
3. Jerome Bruner. 1991. The narrative construction of reality. Critical inquiry 18, 1 (1991), 1–21.
4. Ming-Chung Chen, Chun-Han Chiang, and Chien-Chuan Ko. 2008. The Effectiveness of TriAccess Reading System on Comprehending Nature Science Text for Students with Learning Disabilities. In Computers Helping People with Special Needs, Klaus Miesenberger, Joachim Klaus, Wolfgang Zagler, and Arthur Karshmer (Eds.). Springer Berlin Heidelberg, Berlin, Heidelberg, 747–754.
5. Ming-Chung Chen, Chien-Chuan Ko, and Chi-Ren Huang. 2009. Supportive Reading System for Students with Special Educational Needs in Taiwan. In Proceedings of the 3rd International Convention on Rehabilitation Engineering & Assistive Technology (Singapore) (i-CREATe ’09). ACM, New York, NY, USA, Article 33, 4 pages. https://doi.org/10.1145/1592700.1592738
6. Karen Erickson, Gretchen Hanser, Penelope Hatch, and Eric Sanders. 2009. Research-based practices for creating access to the general curriculum in reading and literacy for students with significant intellectual disabilities. Monograph prepared for the Council for Chief State School Officers (CCSSO) Assessing Special Education Students (ASES) State Collaborative on Assessment and Student Standards (SCASS) (2009).
7. International Society for Augmentative and Alternative Communication. 2020. What is AAC? https://www.isaac-online.org/
8. Christopher Frauenberger, Judith Good, and Alyssa Alcorn. 2012. Challenges, Opportunities and Future Perspectives in Including Children with Disabilities in the Design of Interactive Technology. In Proceedings of the 11th International Conference on Interaction Design and Children (Bremen, Germany) (IDC ’12). ACM, New York, NY, USA, 367–370. https://doi.org/10.1145/2307096.2307171
9. Mark G Friedman and Diane Nelson Bryen. 2007. Web accessibility design recommendations for people with cognitive disabilities. Technology and Disability 19, 4 (2007), 205–212.
10. Serge Gauthier, Barry Reisberg, Michael Zaudig, Ronald C Petersen, Karen. Ritchie, Karl Broich, Sylvie Belleville, Henry Brodaty, David Bennett, Howard Chertkow, et al. 2006. Mild cognitive impairment. The lancet 367, 9518 (2006), 1262–1270.
11. Mirko Gelsomini, Franca Garzotto, Vito Matarazzo, Nicolò Messina, and Daniele Occhiuto. 2017. Creating Social Stories As Wearable Hyper-Immersive Virtual Reality Experiences for Children with Neurodevelopmental Disorders. In Proceedings of the 2017 Conference on Interaction Design and Children (Stanford, California, USA) (IDC ’17). ACM, New York, NY, USA, 431–437. https://doi.org/10.1145/3078072.3084305
12. Kirsty Henderson, Chris Gibson, and Forbes Gibb. 2013. The impact of tablet computers on students with disabilities in a higher education setting. Technology and Disability 25, 2 (2013), 61–76.
13. Jennifer Mankoff, Gillian R. Hayes, and Devva Kasnitz. 2010. Disability Studies As a Source of Critical Inquiry for the Field of Assistive Technology. In Proceedings of the 12th International ACM SIGACCESS Conference on Computers and Accessibility (Orlando, Florida, USA) (ASSETS ’10). ACM, New York, NY, USA, 3–10. https://doi.org/10.1145/1878803.1878807
14. Cristina Manresa-Yee, Ann Morrison, Joan Jordi Muntaner, and Maria Francesca Roig-Maimó. 2017. Multi-sensory Environmental Stimulation for Users with Multiple Disabilities. Springer International Publishing, Cham, 165–182. https://doi.org/10.1007/978-3-319-49879-9_9
15. Andreia Matos, Tânia Rocha, Luciana Cabral, and Maximino Bessa. 2015. Multi-sensory Storytelling to Support Learning for People with Intellectual Disability: An Exploratory Didactic Study. Procedia Computer Science 67 (2015), 12 – 18. https://doi.org/10.1016/j.procs.2015.09.244 Proceedings of the 6th International Conference on Software Development and Technologies for Enhancing Accessibility and Fighting Info-exclusion.
16. Meredith Ringel Morris, Jazette Johnson, Cynthia L. Bennett, and Edward Cutrell. 2018. Rich Representations of Visual Content for Screen Reader Users. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (Montreal QC, Canada) (CHI ’18). ACM, New York, NY, USA, Article 59, 11 pages. https://doi.org/10.1145/3173574.3173633
17. Ronald C Petersen, Rachelle Doody, Alexander Kurz, Richard C Mohs, John C Morris, Peter V Rabins, Karen Ritchie, Martin Rossor, Leon Thal, and Bengt Winblad. 2001. Current concepts in mild cognitive impairment. Archives of neurology 58, 12 (2001), 1985–1992.
18. Ronald C Petersen, Rosebud O Roberts, David S Knopman, Bradley F Boeve, Yonas E Geda, Robert J Ivnik, Glenn E Smith, and Clifford R Jack.2009. Mild cognitive impairment: ten years later. Archives of neurology 66, 12 (2009), 1447–1455.
19. Nigel Robb, Michael Leahy, Connie Sung, and Lizbeth Goodman. 2017. Multisensory Participatory Design for Children with Special Educational Needs and Disabilities. In Proceedings of the 2017 Conference on Interaction Design and Children (Stanford, California, USA) (IDC ’17). ACM, New York, NY, USA, 490–496. https://doi.org/10.1145/3078072.3084314
20. W. Xavier Snelgrove and Ronald M. Baecker. 2010. A System for the Collaborative Reading of Digital Books with the Partially Sighted: Project Proposal. In Proceedings of the Third Workshop on Research Advances in Large Digital Book Repositories and Complementary Media (Toronto, ON, Canada) (BooksOnline ’10). ACM, New York, NY, USA, 47–50. https://doi.org/10.1145/1871854.1871870
21. Kiley Sobel, Alexander Fiannaca, Jon Campbell, Harish Kulkarni, Ann Paradiso, Ed Cutrell, and Meredith Ringel Morris. 2017. Exploring the Design Space of AAC Awareness Displays. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (Denver, Colorado, USA) (CHI ’17). ACM, New York, NY, USA, 2890–2903. https://doi.org/10.1145/3025453.3025610erence here.
22. Leandro Soares Guedes, Luiz André Marques, and Gabriellen Vitório. 2020. Enhancing Interaction and Accessibility in Museums and Exhibitions with Augmented Reality and Screen Readers. In Computers Helping People with Special Needs, Klaus Miesenberger, Roberto Manduchi, Mario Covarrubias Rodriguez, and Petr Peňáz (Eds.). Springer International Publishing, Cham, 157–163.

About the Authors