THE USE OF AUTOMATIC TEXT SIMPLIFICATION TO PROVIDE READING ASSISTANCE TO DEAF AND HARD-OF-HEARING INDIVIDUALS IN COMPUTING FIELDS

Oliver Alonzo, Rochester Institute of Technology oa7652@rit.edu

Abstract

Automatic Text Simplification (ATS) aims to rewrite text in a way that reduces its linguistic complexity while preserving its original meaning. While some prior research has explored using ATS to provide reading assistance to different user groups, relatively little work has investigated its use for Deaf and Hard-of-hearing (DHH) adults or readers in a particular domain. In this project, we investigate the use of ATS-based reading assistance tools for DHH individuals in the computing and information technology (IT) fields, motivated by prior work suggesting that computing professions often require reading about new technologies in order to stay current in the profession. Employing a variety of research methods, we investigate questions including the needs and interests of DHH individuals in the computing and IT fields for ATS-based reading assistance tools and their preferences for different interface parameters of these tools. We also investigate how to evaluate these technologies with this particular user group and how they may benefit from using these tools. This summary presents the motivation for this work, positions it in the context of the related literature, and outlines the proposed solution, our current progress and the project's contributions.

Motivation

Automatic Text Simplification (ATS) uses natural language processing (NLP) to rewrite text with the goal of reducing its linguistic complexity while preserving as much of its original meaning as possible [24]. Researchers in the area of ATS have explored different approaches, which include: lexical approaches, which aim to replace individual complex words with simpler synonyms, e.g. [16]; syntactic approaches, which aim to rewrite longer chunks of text such as phrases or sentences and reduce their syntactic complexity, e.g. [7]; or approaches that incorporate both syntactic and lexical changes, e.g. [17, 28]. While the research on improving the underlying tecnologies remains prevalent as this is an emerging experimental field, a parallel line of research in human computer interaction (HCI) and computing accessibility also investigates the potential applications of these technologies to provide reading assistance to different user groups, including people with disabilities.

Research has observed great diversity in the literacy skills of DHH adults. While many DHH adults can be strong readers, some studies have found, for instance, 4th grade reading levels among DHH high school graduates [25] and other studies suggest that over 17% of DHH adults can be classified as having "low literacy" [9]. Low literacy can pose particular challenges to individuals in the computing and IT fields, which often requires professionals to learn new technology skills, often through text-based content. Thus, our research focuses particularly on the use of ATS to provide reading assistance to Deaf and Hard-of-hearing (DHH) adults in the computing and information field.

Related Work

Prior accessibility research on the use of ATS as reading assistance tools has focused on different user groups, including people with dyslexia (e.g. [22]), people with aphasia (e.g. [8]), people with autism (e.g. [27]), low-literacy readers (e.g. [26]), and DHH adults [13, 14] or children (e.g. [12]). Most of this prior research focuses on evaluating whether there are benefits obtained from using different ATS technologies and approaches with these user groups (e.g. [20, 23]). However, prior work in the context of DHH adults have only evaluated syntactic approaches to text simplification [13]. Considering that prior education research has suggested that DHH readers often face difficulty at both the lexical- and syntactic-level [15], we explore whether there are benefits obtained from other approaches to ATS (e.g. lexical approaches) when using them to provide reading assistance to DHH readers.

Recent work in NLP has advocated for incorporating human judgements of complexity into the complexity scoring systems of ATS, as well as focusing on particular text domains [16] (i.e. texts that focus on a particular topic such as medical texts [7]). However, while some accessibility work has investigated the incorporation of direct feedback from users during the simplification process [5], to the best of our knowledge no prior work has gathered judgements on a wide lexicon from users with a particular literacy profile, nor from a particular domain. Thus, in our work we gather judgements on a large lexicon of both a general (as described in [16]) and a computing vocabulary from DHH individuals to support the use of these judgements in the complexity scoring systems of ATS systems.

Some research has also focused on evaluating the text simplification needs of particular user groups [27], as well as the linguistic needs (as defined by what aspects of the text increase their complexity) of particular groups [19, 21]. While the latter has suggested that different populations possess different literacy needs and linguistic profiles, considerations of how a particular literacy profile may be better served by different ATS approaches are usually not reported. Further, the linguistic and text simplification needs of DHH adults in the context of ATS have not been explored. Thus, in our work, we explore the simplification needs and requirements, and preferences for different ATS approaches of DHH adults in the computing and IT field.

Finally, many of the experimental studies in prior work have made particular choices about how the users interact with their prototypes, including decisions about the platform (e.g. on a website [23] vs. a browser extension [5]), the level of user initiative for requesting simplifications (e.g. automatically replacing text [20] vs. allowing the user to request simplifications [5, 26]), the placement of the simpler text (fully replacing the text [20] vs. providing the simplified text in a tooltip while preserving the original text [10, 20]), and the visibility that a replacement has taken place (e.g. providing a background highlight to convey that a change has happened [5] vs. providing no visible indications [6, 13, 22]. However, while prior work has speculated that the choices of interaction parameters may be more important than even the choice of simplification strategy [21], little work has focused on users' preferences for different settings of these parameters and how those may affect their perspectives on the systems usability, acceptability, as well as the benefits obtained. Thus, in our work we incorporate participatory approaches for driving the decisions of the different interface parameters, as well as summative evaluations to investigate the influence of those decisions in users' perspectives of the systems.

Proposed Solution

Our goal is to investigate user-centered solutions for using ATS as reading-assistance tools for DHH individuals in the computing and IT field. As outlined in the previous section, this incorporates investigating their needs and preferences, evaluating whether there are benefits obtained from different approaches to ATS, gathering their judgements on the complexity of general and computing lexicons, as well as investigating their preferences for different interface parameters which would allow us to create a prototype to support a summative evaluation of the technology. Each of these steps incorporates its own challenges, and there are different methodologies that we employ for these different stages of our project.

  1. Investigating needs and preferences. We conduct a survey and follow-up interviews to investigate DHH adults' reading experiences, how they currently overcome complicated text, as well as their perspectives and interests on the use of ATS as reading assistance tools. We also conduct qualitative research to investigate the social accessibility of these tools, which may inform other interface parameters that will be important to consider in addition to the ones outlined above.
  2. Investigating interface parameters. We employ participatory methods to investigate users' preferences for the tuning of the different interface parameters, including the platform, user initiative, placement and visibility of the replacement text, as outlined above, as well as the permanence of the replacement text, displays of the system's confidence of the appropriateness of a replacement, and other parameters that may emerge as important from our qualitative user studies.
  3. Evaluating ATS. Typically, prior work has evaluated ATS and their benefits for users by conducting experimental studies, employing a variety of both objective and subjective judgements such as comprehension questions or judgements of readability or understandability. However, considering that prior work has identified challenges in evaluating linguistic technologies with DHH adults given their diversity in literacy [4, 11], we first need to conduct methodological research on how to evaluate these technologies with DHH adults to ensure we can trust the metrics employed.
  4. Gathering word complexity judgements. We follow the methodology of [16] to gather judgements from DHH adults on a 15,000 word general vocabulary lexicon, as well as on a computing lexicon of similar length obtained from computing sources online.
  5. Summative evaluation. Based on the results from all the previous items, we develop a prototype to conduct experimental and/or observational studies with DHH adults in the computing and IT fields to evaluate its usability and social accessibility, as well as the benefits in comprehension when reading or engaging in self-directed learning tasks obtained from the use ATS-based reading assistance.

Current Progress

I am finishing the third year of my PhD program, and have completed all required course work for my program. Over the past three years, I have also been able to conduct research advancing many of the items outlined in the previous section through projects that have been disseminated at the ASSETS and CHI conferences, which are described below. This section also outlines on-going and the plans for future work, which will inform the direction of my dissertation proposal.

Completed Work

We have completed a survey and interview study focusing on the reading experiences and interests of DHH adults with experience in the computing and IT field [1]. In that study, we found that participants read often online for work and academic purposes, and expressed strong interest in ATS-based reading assistance tools. The results from that study also highlighted concerns from participants around the accuracy and social accessibility of these tools, as well as interface parameters that would be important to consider such as how much text is replaced at once when a user requests a simplification.

We also conducted a methodological study on how to evaluate the complexity of simplified texts among DHH adults with different levels of literacy [3]. In that study, we found that comprehension questions were effective to distinguish text complexity only among DHH adults with lower literacy skills, and only when they were written with low linguistic complexity. Only subjective judgements of readability were effective with DHH adults at all literacy levels in our study.

Finally, we conducted an experimental study in which we evaluated whether there are benefits obtained by DHH adults from lexical simplification approaches, as well as participants' perspectives on the autonomy of ATS tools (as defined by the level of user initiative for requesting simplifications, as well as the placement and visibility of the replacement text that the interface provides to the user) [2]. In that study, we found that while our comprehension questions employed in that study did not yield significant differences, participants' subjective judgements suggested they perceived benefits from using lexical simplification. Participants also reported being more likely to use systems that provide them with the autonomy to request simplifications on demand, as well as visibility as for what text has been replaced.

Current Work

Based on the results from [1], we are conducting further qualitative research focusing specifically on the social accessibility of ATS-based reading assistance tools. More specifically, we are conducting an interview study with DHH adults with experience in the computing industry to learn about their values and aspirations at work, and how the use of ATS-based reading assistance tools may support or conflict those values and aspirations in the social context of the work environment.

Considering that ATS is still an emerging technology and that errors may be introduced in the simplification process, we are also currently conducting further methodological work in which we investigate how to evaluate two other aspects of the quality of the output of ATS, which are often the subject of ATS evaluation: meaning preservation and fluency (or grammatical correctness). We are following an approach similar to our previous study [3] to ensure the integrity of the metrics identified as effective in that previous study when semantic and grammatical errors are introduced.

Finally, our data collection of judgements from DHH adults on the general and computing lexicons is also on-going.

Future Work

In the future, we plan to conduct research employing participatory methods for the design of a working prototype based on users' preferences for the different design parameters. We are planning to follow a similar approach to that of [18]. To support this, we are working on a highly customizable prototype that allows tuning the different interface parameters identified above. However, we are still on the planning stages for this project and it would be great to receive feedback on how to best design this study during the Doctoral Consortium, especially on how to best incorporate participatory design approaches to investigate the preferred settings for different interface parameters.

Based on all the work outlined above, we hope to conduct our summative evaluation in the next year by deploying our working prototype in experimental and/or observational studies. While our prototype will be informed by our qualitative and participatory research, these summative studies will also be informed by our methodological research on how to best evaluate ATS technologies with DHH adults. In these studies, we hope to explore aspects such as the usability and social accessibility of ATS-based reading assistance tools, as well as their potential benefits in reading comprehension and self-directed learning tasks. This is also an area of future work that could benefit from feedback during the Doctoral Consortium, especially considering the potential challenges of conducting observational studies in the current context of the COVID-19 pandemic where the future of in-person human-subject studies still remains unknown

Contributions

While the main contribution of this research project will be a user-centered working prototype for ATS-based reading assistance tools that can be used for future evaluations of ATS technologies in accessibility research with DHH adults in the computing and information field, there are also other contributions that emerge from the many sub-projects in this work. These include:

  1. Requirements and interests of DHH individuals in the computing field, suggesting that they frequently read on electronic devices to learn about new topics for their work, and expressed strong interest in such tools, which motivate the need for further technical and design work on ATS-based reading assistance tools for this group.
  2. Datasets of judgements of complexity from DHH adults on general and computing lexicons, which can be used to tune ATS technologies and investigate how incorporating data from specific user groups or specific domains may affect their effectiveness.
  3. Frameworks for conducting methodological research on how to evaluate different aspects of the quality of ATS output, including complexity, grammatical correctness and meaning preservation, which can be used with other user groups to ensure the proper metrics are employed in user evaluations.
  4. A framework for investigating the preferences of different interface parameters with other user groups or those working in other domains for developing user-centered prototypes.

Doctoral Consortium

The ASSETS Doctoral consortium would be a great opportunity to receive feedback on our future studies, as outlined in the future work section above. I am planning to present my dissertation proposal in December 2021, so this would be the perfect timing to discuss our plans and receive feedback, and incorporate that feedback into my dissertation proposal. I would also love to have the opportunity to widen my network within the ASSETS community, and share my experiences and contributions with other students and senior members of the community participating in the Doctoral Consortium. Finally, as someone who comes from a developing country, where careers in scientific research are not common, I am always looking for opportunities to interact with researchers at different levels of seniority across academia and industry to learn more about their paths and experiences as a way to inform my decisions for my own career choices. So, having the opportunity to attend this Doctoral Consortium and the general ASSETS conference would be instrumental as the time for making those career choices approaches.

Acknowledgments

I am thankful for the mentorship and support of my advisor, Dr. Matt Huenerfauth, both for my dissertation work and for my growth as a researcher. I am also grateful for the contributions and support from my co-authors, as well as the mentorship from my colleagues and faculty members at the Center for Accessibility and Inclusion Research (CAIR) at RIT. This material is based upon work supported by the National Science Foundation.

References

  1. Oliver Alonzo, Lisa Elliot, Becca Dingman, and Matt Huenerfauth. 2020. Reading Experiences and Interest in Reading-Assistance Tools Among Deaf and Hard-of- Hearing Computing Professionals. In The 22nd International ACM SIGACCESS Conference on Computers and Accessibility (Virtual Event, Greece) (ASSETS '20). Association for Computing Machinery, New York, NY, USA, 13. https://doi.org/ 10.1145/3373625.3416992
  2. Oliver Alonzo, Matthew Seita, Abraham Glasser, and Matt Huenerfauth. 2020. Automatic Text Simplification Tools for Deaf and Hard of Hearing Adults: Benefits of Lexical Simplification and Providing Users with Autonomy. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (Honolulu, HI, USA) (CHI '20). Association for Computing Machinery, New York, NY, USA, 1-13. https://doi.org/10.1145/3313831.3376563
  3. Oliver Alonzo, Jessica Trussell, Becca Dingman, and Matt Huenerfauth. 2021. Comparison of Methods for Evaluating Complexity of Simplified Texts among Deaf and Hard-of-Hearing Adults at Different Literacy Levels. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (CHI '21). Association for Computing Machinery, New York, NY, USA. https://doi.org/10.1145/3411764. 3445038
  4. Larwan Berke, Sushant Kafle, and Matt Huenerfauth. 2018. Methods for Evaluation of Imperfect Captioning Tools by Deaf or Hard-of-Hearing Users at Different Reading Literacy Levels. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (Montreal QC, Canada) (CHI '18). ACM, New York, NY, USA, Article 91, 12 pages. https://doi.org/10.1145/3173574.3173665
  5. Joachim Bingel, Gustavo Paetzold, and Anders Søgaard. 2018. Lexi: A tool for adaptive, personalized text simplification. In Proceedings of the 27th International Conference on Computational Linguistics. Association for Computational Linguistics, Santa Fe, New Mexico, USA, 245-258. https://www.aclweb.org/anthology/ C18- 1021
  6. Stefan Bott, Horacio Saggion, and David Figueroa. 2012. A Hybrid System for Spanish Text Simplification. In Proceedings of the Third Workshop on Speech and Language Processing for Assistive Technologies (Montreal, Canada) (SLPAT '12). Association for Computational Linguistics, Stroudsburg, PA, USA, 75-84. http://dl.acm.org/citation.cfm?id=2392855.2392865
  7. Ashwin Devaraj, Iain J Marshall, Byron C Wallace, and Junyi Jessy Li. 2021. Paragraph-level Simplification of Medical Texts. arXiv preprint arXiv:2104.05767 (2021).
  8. Siobhan Devlin and Gary Unthank. 2006. Helping Aphasic People Process Online Information. In Proceedings of the 8th International ACM SIGACCESS Conference on Computers and Accessibility (Portland, Oregon, USA) (Assets '06). ACM, New York, NY, USA, 225-226. https://doi.org/10.1145/1168987.1169027
  9. Organization for Economic Co-operation and Development. 2012. Survey of Adult Skills (PIAAC). http://skills.oecd.org/Survey_of_Adult_Skills_US.pdf
  10. Dhananjai Hariharan, Sedeeq Al-khazraji, and Matt Huenerfauth. 2018. Evaluation of an English Word Look-Up Tool for Web-Browsing with Sign Language Video for Deaf Readers. In International Conference on Universal Access in Human- Computer Interaction. Springer, 205-215.
  11. Matt Huenerfauth and Hernisa Kacorri. 2015. Best practices for conducting evaluations of sign language animation. In 30th Annual International Technology and Persons with Disabilities Conference Scientific/Research Proceedings. California State University, Northridge.
  12. Kentaro Inui, Atsushi Fujita, Tetsuro Takahashi, Ryu Iida, and Tomoya Iwakura. 2003. Text Simplification for Reading Assistance: A Project Note. In Proceedings of the Second International Workshop on Paraphrasing - Volume 16 (Sapporo, Japan) (PARAPHRASE '03). Association for Computational Linguistics, Stroudsburg, PA, USA,9-16. https://doi.org/10.3115/1118984.1118986
  13. Poorna Kushalnagar, Scott Smith, Melinda Hopper, Claire Ryan, Micah Rinkevich, and Raja Kushalnagar. 2018. Making cancer health text on the Internet easier to read for deaf people who use American Sign Language. Journal of Cancer Education 33, 1 (2018), 134-140.
  14. Slavina Lozanova, Ivelina Stoyanova, Svetlozara Leseva, Svetla Koeva, and Boian Savtchev. 2013. Text modification for Bulgarian sign language users. In Proceedings of the Second Workshop on Predicting and Improving Text Readability for Target Reader Populations. 39-48.
  15. John L Luckner and C Michele Handley. 2008. A summary of the reading com- prehension research undertaken with students who are deaf or hard of hearing. American Annals of the Deaf 153, 1 (2008), 6-36.
  16. Mounica Maddela and Wei Xu. 2018. A Word-Complexity Lexicon and A Neural Readability Ranking Model for Lexical Simplification. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Brussels, Belgium, 3749-3760. https://doi.org/10. 18653/v1/D18- 1410
  17. Angrosh Annayappan Mandya, Tadashi Nomoto, and Advaith Siddharthan.2014. Lexico-syntactic text simplification and compression with typed dependencies. In 25th International Conference on Computational Linguistics.
  18. Hugo Nicolau, André Rodrigues, André Santos, Tiago Guerreiro, Kyle Montague, and João Guerreiro. 2019. The Design Space of Nonvisual Word Completion. In The 21st International ACM SIGACCESS Conference on Computers and Accessibility (Pittsburgh, PA, USA) (ASSETS '19). Association for Computing Machinery, New York, NY, USA, 249-261. https://doi.org/10.1145/3308561.3353786
  19. Gustavo Paetzold and Lucia Specia. 2016. Understanding the Lexical Simplification Needs of Non-Native Speakers of English. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers. The COLING 2016 Organizing Committee, Osaka, Japan, 717-727. https://www.aclweb.org/anthology/C16- 1069
  20. Luz Rello, Ricardo Baeza-Yates, Stefan Bott, and Horacio Saggion. 2013. Simplify or Help?: Text Simplification Strategies for People with Dyslexia. In Proceedings of the 10th International Cross-Disciplinary Conference on Web Accessibility (Rio de Janeiro, Brazil) (W4A '13). ACM, New York, NY, USA, Article 15, 10 pages. https://doi.org/10.1145/2461121.2461126
  21. Luz Rello, Ricardo Baeza-Yates, Laura Dempere-Marco, and Horacio Saggion. 2013. Frequent words improve readability and short words improve understandability for people with dyslexia. In IFIP Conference on Human-Computer Interaction (INTERACT 2013). Springer, 203-219.
  22. Luz Rello, Roberto Carlini, Ricardo Baeza-Yates, and Jefrey P. Bigham. 2015. A Plug-in to Aid Online Reading in Spanish. In Proceedings of the 12th Web for All Conference (Florence, Italy) (W4A '15). Association for Computing Machinery, New York, NY, USA, Article 7, 4 pages. https://doi.org/10.1145/2745555.2746661
  23. Horacio Saggion, Sanja Štajner, Stefan Bott, Simon Mille, Luz Rello, and Biljana Drndarevic. 2015. Making It Simplext: Implementation and Evaluation of a Text Simplification System for Spanish. ACM Trans. Access. Comput. 6, 4, Article 14 (May 2015), 36 pages. https://doi.org/10.1145/2738046
  24. Matthew Shardlow. 2014. Out in the Open: Finding and Categorising Errors in the Lexical Simplification Pipeline. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC'14). European Language Resources Association (ELRA), Reykjavik, Iceland, 1583-1590. http://www.lrec- conf.org/proceedings/lrec2014/pdf/479_Paper.pdf
  25. C. B. Traxler. 2000. The Stanford Achievement Test, 9th Edition: National Norming and Performance Standards for Deaf and Hard-of-Hearing Students. Journal of Deaf Studies and Deaf Education 5, 4 (Jan 2000), 337-348. https://doi.org/10.1093/ deafed/5.4.337
  26. Willian Massami Watanabe, Arnaldo Candido Junior, Vinícius Rodriguez Uzêda, Renata Pontin de Mattos Fortes, Thiago Alexandre Salgueiro Pardo, and Sandra Maria Aluísio. 2009. Facilita: Reading Assistance for Low-literacy Readers. In Proceedings of the 27th ACM International Conference on Design of Communication (Bloomington, Indiana, USA) (SIGDOC '09). ACM, New York, NY, USA, 29-36. https://doi.org/10.1145/1621995.1622002
  27. Victoria Yaneva, Constantin Orasan, Le An Ha, and Natalia Ponomareva. 2019. A Survey of the Perceived Text Adaptation Needs of Adults with Autism. In Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2019). INCOMA Ltd., Varna, Bulgaria, 1356-1363. https://doi.org/10.26615/978- 954- 452- 056- 4_155
  28. Xingxing Zhang and Mirella Lapata. 2017. Sentence Simplification with Deep Reinforcement Learning. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Copenhagen, Denmark, 584-594. https://doi.org/10.18653/v1/D17- 1062

About the Authors

Oliver Alonzo is a PhD Student in Computing and Information Sciences at the Rochester Institute of Technology. His research focuses on computing accessibility and human computer interaction (HCI), exploring the user-centric design and evaluation of linguistic tools for Deaf and Hard-of-hearing adults. For more details, visit his website oliveralonzo.com.