An Epidemiology-Inspired, Large-Scale Analysis of Mobiule App Accessibility

Anne Spencer Ross, DUB Group, Paul G. Allen School of Computer Science and Engineering, University of Washington, Seattle, USA, ansross@cs.washington.edu

Abstract

My research explores large-scale analysis of mobile application accessibility, with a goal of expanding the types of accessibility barriers for which applications are tested and the ways in which results are presented. I am basing my research in a novel epidemiology-inspired framework that we developed for structuring the examination of mobile application accessibility. It frames accessibility as a product of diverse factors within the rich ecosystem in which applications are created, maintained, and used. The framework proposes large-scale analysis as a method for understanding the state of accessibility. My completed work applies this framework by conducting a large-scale analysis of an existing dataset of Android applications, assessing them for label-based accessibility barriers on image-based buttons. My future work will extend this research to analyze current applications at scale and expand the accessibility barriers that are tested for.

1. Introduction

Mobile applications (apps) are increasingly important in daily life (e.g. finance, transportation). Apps must be accessible to all people regardless of abilities or their use of assistive technologies. Large-scale analyses can provide insights into the state of app accessibility, can help identify significant problems, and can guide enhancement efforts.

My work expands existing approaches to app accessibility to encompass large-scale analysis and the rich ecosystem in which apps exist. The key thesis is that insights gained from these ecosystem-framed analyses can drive innovations in enhancing app accessibility. I discuss prior work and propose future work toward exploring this thesis.

2. Background

Prior work assessing app accessibility on a small scale has revealed many barriers. These included elements missing labels [3,13,14], low color contrast [3,19], and un-focusable elements [3,14]. Most app testing techniques included manual analysis, which limits scalability. Furthermore, the breadth of errors tested was mostly limited to modified web-based accessibility guidelines, with some authors noting the lack of mobile-specific guides [3,13,14,19]. I am working on tools for scalable app accessibility analysis. I also aim to extend tests to include more nuanced and diverse accessibility barriers appropriate to mobile apps.

Accessibility has been analyzed on a large scale in the web. Prior work found that accessibility barriers still exist in many websites [12]. Longitudinal, large-scale work found that accessibility tended to increase on the web over a 14-year span [11], but that this increase was more likely due to changes in coding practices than in increased accessibility regulation [16]. These works exemplify the types of insights that can be gained from large-scale and multi-factor analysis.

3. Research Questions

Key questions for this work are:

  1. What are impactful accessibility barriers in apps that represent a diverse set of abilities?
  2. How can apps be tested for these barriers at scale?
  3. How can insights from analysis techniques be applied to create impact toward enhancing app accessibility?

4. Framing and Performing Large-Scale Analysis

Thus far, I have created a guiding conceptual framework and performed an initial large-scale analysis of labeling errors in image buttons. These projects provide a foundational understanding of app accessibility. The results from these completed steps motivate and guide my future work.

4.1. Epidemiology-Inspired Framework

Current techniques analyzing app accessibility largely consider apps individually and focus on the role of the app developer. Although this perspective is essential, efforts for enhancing accessibility could benefit from considering the richer context in which apps exist. To structure a broadened exploration of app accessibility, the framework we presented at ASSET 2017 [17] frames app accessibility as the product of a rich ecosystem of influential factors. Factors include code reuse, development tools, and education. Inspired by epidemiology, accessibility barriers are modeled as diseases in an app population. We proposed large-scale and longitudinal analyses as methods to better understand the state of accessibility and create data-driven treatments.

To create the framework, I performed interviews with people who use mobile screen readers. I explored the types of apps they would like to use and what accessibility barriers were most problematic [20]. I combined these insights with published accessibility guides for apps [2,9,15], prior research in app accessibility, and app accessibility forums to create a foundational understanding of what barriers exist.

4.2. First Large-Scale App Accessibility Analysis

Applying our framework, we performed a large-scale analysis of label-based accessibility barriers, to be presented at ASSETS 2018 [18]. We tested three types of image-based buttons in 5,753 apps for missing labels, duplicate labels, and uninformative labels. The first two tests were from the Accessibility Testing Framework for Android [10]. I defined uninformative label errors through manual analysis of poor labels (e.g. “image” and “desc”).

We found missing labels to be the most prevalent error, and floating action buttons to have the highest proportion of errors. These results motivate further work on enhancing app accessibility for even basic compliance like labeling images. The work also shows the value of large-scale app analysis.

5. Proposed Remaining Research

My first iteration of large-scale app accessibility analysis was based on a limited set of criteria applied to a pre-collected and pre-processed set of apps. Guided by the epidemiology framework, I aim to more robustly assess app accessibility. I want to further apply those insights to enhance access to apps such as through developer and end-user reports.

5.1. Method

The next steps for this work include (1) creating infrastructure to collect app data and (2) defining and implementing a set of accessibility tests for apps.

5.2. Implementing Collection Tools

Gathering meaningful data from apps is challenging. We used an openly released dataset [4] for our first analysis. The collection method for that data is a starting point for our own collection infrastructure. I aim to additionally capture data that reflect the importance of elements and screens. The ability to determine if a barrier prevents access to a key feature of an app can serve as one metric of severity. Our current plan combines automated traversal and crowdsourcing, informed by prior work [5]. Improvements to the data collection methods may include using better screen comparison metrics and different crowd prompts.

5.3. Testing App Accessibility

A key parallel component for this work is enhancing app testing tools. Current tools are limited in the types of barriers for which they test, often focused on easier heuristic-based tests such as the size of an element or if it is completely missing a label. Tools are also limited in application, usually needing manual intervention [1,7] or integration into app source code [8]. Our tools will broaden the errors tested for and the scalability of testing apps.

Additional tests may include whether elements are focusable with assistive technologies, whether linear navigation orders are meaningful, and whether labels are meaningful. We are exploring using crowdsourcing and pixel-level analysis [6].

I am currently determining the set of accessibility barriers on which I will focus. I plan to use multiple methods including stakeholder involvement (including developers and end-users), prior work, guidelines, and public forums. I will iteratively work with the community of stakeholders to ensure the chosen tests are the most impactful.

6. Involvement in ASSETS Doctoral Consortium

I have completed three years of my PhD program and plan to graduate in the Spring of 2021. The ASSETS DC provides a strong community of experts in accessibility to advise on the key questions in my work. Participating in the DC would help me determine appropriate accessibility tests for apps and research methods to expand and validate those tests. This help would support me in increasing my work’s impact in the accessibility field and broader community.

The DC comes at an ideal time in my research. I have created a framework to guide my work and performed foundational work in testing for well-known app accessibility barriers at a large scale. This foundational work will allow me to elicit actionable advising. I have begun implementation of a testing infrastructure. However, the DC will occur while I am choosing the specific accessibility barriers on which I will focus. The feedback I receive at the DC would be incorporated into these key decisions.

Through my experiences thus far, I have strengthened my ability to discuss and perform research. These skills will allow me to understand, engage with, and contribute to discussions around my and other participants’ research.

7. Conclusion

The epidemiology-inspired framework we developed for this work broadens how app accessibility is considered. It provides structure for pursuing a richer understanding of the state of app accessibility and the impact of diverse factors. App accessibility analyses help capture the state of app accessibility. Having metrics and data to establish the severity of barriers is one important component to motivate work on enhancements. Once motivated, future work can be guided by the results of large-scale analysis to focus on the most prominent and impactful projects. For example, my future work includes creating developer feedback and end-user information reports.

The combination of the framework, analysis tools, and large-scale analysis results are impactful for the accessibility field. This work motivates future work in app accessibility. It can also translate to other accessibility analyses.

Acknowledgments

I thank all of my collaborators, including Xiaoyi Zhang and Anat Caspi and my advisors James Fogarty and Jacob O. Wobbrock. This work is supported in part by the National Science Foundation under the Graduate Research Fellowship Program and by awards IIS-1702751 and IIS- 1053868; by the Wilma Bradley Endowed Fellowship in Computer Science & Engineering; by a Google Faculty award; and by the Mani Charitable Foundation.

References

  1. Apple Accessibility Scanner.
  2. Apple Inc. (2012). Accessibility Programming Guide for iOS.
  3. Lucas Pedroso Carvalho, Bruno Piovesan Melchiori Peruzza, Flávia Santos, Lucas Pereira Ferreira, and André Pimenta Freire. (2016). Accessible Smart Cities?: Inspecting the Accessibility of Brazilian Municipalities’ Mobile Applications. Proceedings of the 15th Brazilian Symposium on Human Factors in Computer Systems - IHC ’16. http://doi.org/10.1145/3033701.3033718
  4. “Data Driven Design Group.” Rico: A Mobile App Dataset of Building Data-Driven Design Applications. http://interactionmining.org/rico
  5. Biplab Deka, Zifeng Huang, Chad Franzen, Joshua Hibschman, Daniel Afergan, Yang Li, Jeffrey Nichols, and Ranjitha Kumar. (2017). Rico: A Mobile App Dataset for Building Data-Driven Design Applications. Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology - UIST ’17, 845–854. http://doi.org/10.1145/3126594.3126651
  6. Morgan E. Dixon, Gierad Laput, and James A. Fogarty. (2014). Pixel-based methods for widget state and style in a runtime implementation of sliding widgets. Proceedings of the 32nd annual ACM conference on Human factors in computing systems - CHI ’14, 2231–2240. http://doi.org/10.1145/2556288.2556979
  7. Google. (2016). Accessibility Scanner.
  8. Google. (2015). Accessibility Test Framework for Android.
  9. Google. Android Accessibility Developer Guidelines.
  10. Google Inc. Accessibility Test Framework for Android.
  11. Vicki L. Hanson and John T. Richards. (2013). Progress on Website Accessibility? ACM Transactions on the Web, 7(1), 1–30. http://doi.org/10.1145/2435215.2435217
  12. Shaun K. Kane, Jessie A. Shulman, Timothy J. Shockley, and Richard E. Ladner. (2007). A Web Accessibility Report Card for Top International University Web Sites. Proceedings of the 2007 international cross-disciplinary conference on Web accessibility (W4A) - W4A ’07, 148. http://doi.org/10.1145/1243441.1243472
  13. Lauren R. Milne, Cynthia L. Bennett, and Richard E. Ladner. (2014). The Accessibility of Mobile Health Sensors for Blind Users. International Technology and Persons with Disabilities Conference Scientific/Research Proceedings (CSUN 2014), 166–175. http://doi.org/10211.3/133384
  14. Kyudong Park, Taedong Goh, Hyo-Jeong So, Hyo-Jeong Association for Computing Machinery., HCI Society of Korea, and Hanbit Media (Firm). (2014). Toward accessible mobile application design: developing mobile application accessibility guidelines for people with visual impairment. Proceedings of HCI Korea - HCIK ’15, 478. https://dl.acm.org/citation.cfm?id=2729491
  15. Kim Patch, Jeanne Spellman, and Kathy Wahlhim. (2015). Mobile Accessibility: How WCAG 2.0 and Other W3C/WAI Guidelines Apply to Mobile. https://www.w3.org/TR/mobile-accessibility-mapping/
  16. John T. Richards, Kyle Montague, and Vicki L. Hanson. (2012). Web Accessibility as a Side Effect. Proc. ASSETS 2012, 79. http://doi.org/10.1145/2384916.2384931
  17. Anne Spencer Ross, Xiaoyi Zhang, James Fogarty, and Jacob O. Wobbrock. (2017). Epidemiology as a Framework for Large-Scale Mobile Application Accessibility Assessment. Proceedings of the 19th International ACM SIGACCESS Conference on Computers and Accessibility - ASSETS ’17, 2–11. http://doi.org/10.1145/3132525.3132547
  18. Anne Spencer Ross, Xiaoyi Zhang, Jacob O. Wobbrock, and James Fogarty. (2018). Examining Image-Based Button Labeling for Accessibility in Android Apps Through Large-Scale Analysis. To be at ACM SIGACCESS Conference on Computers and Accessibility (ASSETS 2018).
  19. Leandro Coelho Serra, Lucas Pedroso Carvalho, Lucas Pereira Ferreira, Jorge Belimar Silva Vaz, and André Pimenta Freire. (2015). Accessibility Evaluation of E-Government Mobile Applications in Brazil. Procedia Computer Science, 67, 348–357. http://doi.org/10.1016/J.PROCS.2015.09.279
  20. Xiaoyi Zhang, Anne Spencer Ross, Anat Caspi, James Fogarty, and Jacob O. Wobbrock. (2017). Interaction Proxies for Runtime Repair and Enhancement of Mobile Application Accessibility. Proceedings of the ACM Conference on Human Factors in Computing Systems (CHI 2017), 6024–6037.

About the Author

Anne is a PhD student in the Paul G. Allen School of Computer Science and Engineering at The University of Washington. She is passionate about enabling diverse groups of people to access, interact with, and communicate information through technology. She is currently working with Dr. James Fogarty and Dr. Jacob O. Wobbrock on making Android applications more accessible to people with disabilities. She received her B.S. in computer science from Colorado State University.