Designing Accessible, Explainable AI (XAI) Experiences

Christine T. Wolf, IBM Research – Almaden,
Kathryn E. Ringland, Northwestern University,


Explainable Artificial Intelligence (XAI) has taken off in recent years, a field that develops techniques to render complex AI and machine learning (ML) models comprehensible to humans. Despite the growth of XAI techniques, we know little about the challenges of leveraging such explainability capabilities in situated settings of use. In this article, we discuss some particular issues around the intersection between accessibility and XAI. We outline two primary concerns: one, accessibility at the interface; and two, tailoring explanations to individuals’ diverse and changing explainability needs. We illustrate these issues by discussing two application areas for AI/Ml systems (aging-in-place and mental health support) and discuss how issues arise at the nexus between explainability and accessibility.

1. Introduction

In recent years, great strides have been made in computational processing capabilities in the fields of statistical machine learning (ML), due in large part to advances in black-box modelling techniques. While such models typically perform with impressively high levels of accuracy (often surpassing human capabilities), their immense complexity and opaque constitution renders them incomprehensible to humans. This presents a problem in a range of application settings, where a number of public policies demand that humans be able to understand and reason about a model’s predictive behaviors. For example, the use of ML in regulated industries (e.g., banking, criminal justice) requires the ability to audit such models. More broadly, some jurisdictions such as the European Union are establishing a data subject’s “right to explanation” as part of large-scale technology regulation (e.g., GDPR) [18]. In addition to compliance with regulation, there are also broader societal reasons why we ought to advocate for human comprehensibility of predictive models’ behaviors – namely to further the goals of inclusive design with digital experiences that make informed, empowered participation possible for all those interacting in data-intensive ecosystems. Enabling participation is important both in everyday encounters with AI/ML systems (e.g., informed participation when interacting with AI systems that make recommendations, provide you with some type of advice, or otherwise shape your daily life). It is also important in attracting and making possible successful careers for individuals of all abilities in technical roles in science, technology, engineering, and math (STEM) fields [22].

Explainable Artificial Intelligence (XAI) is a growing facet of technical AI development that aims to develop techniques which render AI/ML models comprehensible to humans. While there has been rapid growth of XAI techniques recently, considerable work is needed to bridge these technical approaches with settings of actual use where they might prove valuable [21]. In this article, we discuss some particular issues around the intersection between accessibility and XAI. We outline two primary concerns: one is accessibility at the interface; and another is tailoring explanations to individuals’ diverse and changing explainability needs. Next we briefly define some XAI concepts, then discuss two case studies which focus on application domains: aging-in-place and mental health management.

Guiding Concepts: Explainable AI (XAI)

Although concern for the interpretability and comprehension of complex, predictive models is not new, the immensely complex models produced through black-box modelling techniques create new difficulties in how best to design human-readable representations of such models. To render these immensely complex models comprehensible to humans, a common strategy is to probe a trained AI model with many dummy or test data points, using their outputs to produce a simplified, approximation of the model or understanding of the model’s decision-boundaries [e.g., 6, 10, 16]. The simpler approximations and boundaries these perturbation approaches produce are then said to “explain” the model to users. Other common strategies involve analyzing internal components of the model (such as attentions), to provide understanding into its mechanics and behaviour [e.g., 8, 20].

There are a number of ways to approach the XAI problem from a technical standpoint, and indeed an array of different engineering configurations can be found in XAI literature reviews such as Guidotti et al. [4] and Abadi and Berrada [2]. XAI is a fast-growing, vibrant area of research, a review of which is outside the scope of this article. Instead, we briefly introduce XAI concepts at a high-level to guide our discussion. Broadly such techniques share an aim to provide humans with either of two types of explanations: global or local. We define each below.

A global explanation is one that provides an explanation at the model level. This type of explanation is based on representations of the model’s overall “internal state” (describing all classes). Alternately, a global explanation might be divided up into class-specific explanations (e.g., a description of the internal state for each model’s inner classes). As Adadi and Berrada [2] have noted: “global model interpretability is hard to achieve in practice, especially for models that exceed a handful of parameters. Analogically to human, who focus effort on only part of the model in order to comprehend the whole of it, local interpretability can be more readily applicable.” (p. 52148). A local explanation is one that provides an explanation at the input level. This type of explanation strives to describe the specific behavior associated with a particular input – that is, why did the model assign a certain label (dog) to this data input (image of a dog)?

2. Accessibility at the XAI Interface

We discuss two main themes at the intersection between accessibility and XAI. The first is the issue of accessibility at the interface. This involves interaction design and the configuration of information on interfaces that enables their meaningful engagement via alternative modes of interaction (e.g., screen readers or voice dialog). Consider, for example, the XAI technique of visual explanation [5]. Visual explanations are used in image classification tasks, where an AI/ML model classifies images into specific label classes (e.g., photo of a cardinal or a hawk).

 Images of birds with their definitions and the explanation for each.

Images with headers and definitions and explanations. First image 'This is a Marsh Wren because… Definition: this bird is brown and white in color with a skinny brown break and brown eye rings. Explanation: this is a small brown bird with a long tail and a white eyebrow.' Next to this is an image of this bird. Second image, 'This is a Downy Woodpecker because…Definition: this bird has a white breast black wings and a red spot on its head. Explanation: this is a black and white bird with a red spot on its crown.' Third image, “This is a Shiny Cowbird because… Definition: this bird is black with a long tail and has a very short break. Explanation: this is a black bird with a long tail feather and a pointy black beak.'
Figure 1. Images of birds with their definitions and the explanation for each from [5].

Visual explanations generate text that provides both a description of the image, as well as an explanation of what features in that image caused it to be assigned to the label class (see Figure 1 above). An example visual explanation from the figure below provides the following for “This is a Marsh Wren because…”

Definition: “this bird is brown and white in color with a skinny brown beak and brown eye rings.”

Explanation: “this is a small brown bird with a long tail and a white eyebrow.”

Figure 2 provides a more detailed schematic of how visual explanations are generated and points out specific pixel clusters in the image that feature prominently in the model’s internal processing, causing the image to be assigned to the particular class. These features are then used to generate the textual explanation for the class label assignment.

Visual explanations increase accessibility in some ways, but challenges remain. Visual explanations provide humans with access to a description of the black-box model’s inner-workings, providing a global, text-based explanation of the model’s internal state (i.e., the key features it uses to define a label class) and a rationale for its local behavior (i.e., why it applied a specific label to an image). The text-based nature of a visual explanation supports accessibility to a wider range of users (e.g., those who process information more easily via multi-modalities; those who use screen readers/text-to-voice translators). The fact that they are text-based also opens up space for further innovations in the form of conversational/dialog systems, which can enable dynamic queries to further ensure a user’s understanding and comprehension of the model’s state and behavior.

Figure 2. Image from [5:3] describes how visual explanations are generated, with important features fed into black-box classification models which are then fed into black-box text generation models.

A landscape orientation image shows a photo of a cardinal on the left side, with blue and yellow boxes highlighting relevant pixels (important features) that are involved in the black-box classification model’s internal processing. From these features the visual explanation method derives the class label (i.e., 'This is a cardinal because…') as well as the image-specific features which cause it to be labeled at a cardinal. These image-specific features are then fed into black-box text generation models, depicted by a schematic flowchart on the right side of the image, which generate the text string that appends to the class label: '…it has a bright red…'
Figure 2. Image from [5] describes how visual explanations are generated, with important features (i.e., relevant pixels in the image) fed into black-box classification models which are then fed into black-box text generation models.

Yet, the hybrid modality of image+text continues to create challenges in maintaining cohesive context for blind and low-visibility (low-viz) users. How would a blind or low-viz ML developer de-bug a model mis-labelling photos of a scarlet tanager (another North American red bird) as a cardinal? As we see in Figure 2, an important feature for this image of a cardinal is the bird’s crest (indicated by a blue box around the pixel clusters that comprise the cardinal’s prominent pointed head feathers). But how would a blind or low-viz developer be able to connect the specific highlighted pixels with the broader textual explanation provided by this XAI technique without seeing the blue box highlight this visual feature? We raise this point to highlight the need for further technical development around making XAI accessible. While XAI techniques like visual explanations do indeed increase accessibility of opaque, black-box models, challenges persist on how to enable configurations at the interface that bring different pieces of information together in ways that maintain context and accessibility across modalities.

3. Tailoring Explanations to Diverse Users

Our second main theme around accessibility and XAI involves adaptation or tailoring of explanations to the dynamic needs of people in a given context. In the following section, we set out two cases inspired by recent trends in AI/ML domain applications. The first case involves using AI/ML systems to support “aging in place.” The second case involves using AI/ML systems to support those managing chronic mental health conditions such as depression.

Through these two cases, we highlight two central concerns integral to designing accessible XAI experiences. First, are the diverse and dynamic explainability requirements of people in AI/ML ecosystems. Second, is the need to explain not only the global and local explanations of the model, but also the model’s underlying alignment with the experience and trajectory of domain phenomena (i.e., chronic health conditions).

Case 1: Supporting “Aging in Place” for Older Adults

The ability for older people to “age in place” (that is, live in community-based housing rather than medicalized, assisted living facilities) as long as safely possible supports better health outcomes [14]. The natural progression of aging, however, leads to a gradual decline in physical and cognitive abilities. These declines over time cause a diminished capacity to appropriately care for one’s self. It can be difficult to know when is the “right time” for an older person to transition from community-based to a medicalized living arrangement. This can be particularly difficult as often the adult children or familial/community caregivers for older people are unable to provide consistent monitoring of the older person’s capabilities (e.g., they live far away or are only able to visit along certain intervals). Assisting in these activities of monitoring (and providing data to support decision-making) is a domain application of AI/ML systems that has garnered much speculation and anticipation in public discourse [9, 17]. The overall premise of AI/ML systems in this context involves the tracking and smart monitoring of an older person’s activities of daily living (ADLs), alerting to noticeable changes in daily patterns [15].

As Wolf [21] has examined, the actual deployment of such AI/ML systems in this domain will likely raise a number of issues around explainability. A key point raised in that paper was the recognition that there are a range of stakeholders for whom explanations will need to be provided (including, the older person; their adult children/community caregivers; their healthcare providers and social workers; care facility administration, to name a few). This expanded the range of considerations when selecting XAI techniques and designing XAI experiences. In this article, we draw attention to an additional consideration: that the range of people in these aging-in-place ecosystems have differing levels of technical aptitude and attention, and also that those abilities can change situationally. Thus, making an explanation truly accessible (i.e., comprehensible) for a particular person at a given time in a given context requires careful articulation.

By differing levels of technical aptitude and attention – we mean that people involved in the aging-in-place ecosystem will have different abilities to comprehend the technical details of an XAI system but also that attention plays an important role in these knowledge practices. The older adult who worked as a computer engineer will have different explainability needs than one who worked as a farmer, and even still different than the older adult who worked as a schoolteacher (i.e., an individual’s technical aptitude). Similarly, an adult child who themselves has a chronic health condition or disability, will have different attentional abilities when also managing their parent’s aging than an adult child who does not (i.e., attentional considerations). Further, an adult child who is managing a parent’s aging jointly with siblings will have different needs than one who is the sole caregiver (i.e., collaborative attention and decision-making). These factors influence what types of explanations these various actors will need as they try to make use of and integrate an AI/ML monitoring system into the care of the older adult in their life.

By those levels can change over time – we mean that comprehension is situationally dynamic. Consider for example, the adult child or caregiver who, over time, becomes quite adept at understanding the AI/ML systems employed to monitor their parents’ everyday activities in their living space. We might consider this a type of experientially-gained, increased competence, which an XAI system ought to adapt to over time (providing information at a depth that continues to be meaningful and useful to this user). But similarly, the older person’s comprehension may diminish over time as their cognitive capabilities change (thus, requiring the XAI system to adapt explanations to ensure comprehension continues to be achieved for this user). Such changes in the older person’s status will no doubt impact their adult child’s ability to focus and pay close attention to the technical details of the AI/ML system, thus changing the adult child’s explainability needs at that time (i.e., at periods of more or less attentional capacity).

These scenarios highlight how caring for older adults is a situated practice that takes place within a sociotechnical system – people, practices, machines, and information must all work together to achieve shared goals (e.g., monitoring the continued appropriate of the older adult aging-in-place). As others have noted, the distributed cognition that takes place in such complex care ecosystems requires ongoing configuration and adjustment [23]. By raising these issues in the context of XAI design and development, we point to the diverse and dynamic explainability requirements of people who encounter AI/ML ecosystems. Accessibility in these ecosystems requires that meaningful participation and comprehension is made possible for a range of people with differing levels of technical aptitude and attention; these abilities are not static, but instead change situationally, requiring careful alignment with XAI system capabilities.

Case 2: Supporting People with Depression

Often, people with mood-related health conditions such as depression can benefit from the help of structured support and care, such as counseling or therapy. However, despite a continued and growing need for such services, the availability of clinical resources (such as licensed mental healthcare providers) are often limited. The use of emergent technologies are seen as a potential solution in this area, a growing field called “digital mental health” [11–13].

Intelligent Agents to Provide Therapeutic Care

One proposed application of AI/ML in digital mental health is the development of intelligent agents, such as chatbots, that are able to provide lightweight forms of therapy or counseling [3]. The use of conversational agents could extend care by providing a real-time, first-line of defense for those trying to manage milder symptoms of depression. For example, a chatbot might suggest therapeutic activities for an individual to complete in response to specific symptoms. If the person indicates to the chatbot they are feeling lonely or unmotivated, the chatbot could then provide suggestions based on past information (e.g., suggesting they message their best friend or spend time in their favorite local coffee shop). Conversational systems like this could also be integrated with other sensing/capture systems to help predict moods. Together these tools can not only help (re)shape a person’s experiences to improve their mood, but can also help provide data-driven insights into mental health experiences (e.g., reveal triggers, physical manifestations of mood, and predictions about near-future moods).

While the use of AI/ML systems in this way raises a number of questions around ethical and inclusive design, we focus on comprehension and sensemaking around the underlying machine intelligence. The chatbot use case raises traditional concerns over global and local explanation. When the chatbot makes a recommendation, the person ought to be able to uncover at least a local explanation of why the model classified their text input in a certain way and provided a particular response to that input.

But this use case also raises broader concerns over the alignment of statistical modelling and therapeutic experiences. Statistical modelling is premised on modelling historical data and outcomes – past is prologue. But health conditions, particularly those that deal with mental health such as depression, are not always linearly progressive. The lived experiences of a condition is situationally emergent – just because today “wasn’t good” doesn’t mean tomorrow will predictively follow that trajectory. How is such dynamism reflected in the modelling of historical data and how do such (mis)alignments get explained to the human?

We also note the complexities of therapeutic practice and questions around if, how, and when explanations of machine behaviors are most appropriate. Central to the therapeutic relationship between a therapist and client is the dynamic and situated shaping of the encounter based on both the client’s history but also on therapeutic needs that emerge in that moment [1, 7, 19]. Do (and if so, when do) therapists explain their actions to clients? The answer is often “it depends,” which raises important questions of accountability and transparency in therapeutic contexts. The need for an explanation is situational – i.e., therapists often engage in strategic or therapeutic obscuring (not telling a client something in the moment to provoke self-discovery or a psychological breakthrough, but later an explanation can be provided if desired) [1]. This dynamism complicates how we might design the explainability functions of AI/ML digital mental health tools – the need for explanation (and what type of explanation and when) cannot be prescriptively programmed into a system, but instead requires situated action, inter-action, and mutual meaning-making in practice.

Summary and Conclusions

In this article, we have set out two high-level issues at the intersection between accessibility and XAI. The first theme was accessibility at the XAI interface, where we discussed how XAI techniques such as visual explanation [5] have made needed headway in rendering explanations of computer vision processing accessible. Yet, continued work is needed to more fully think through how multi-modal XAI interfaces (e.g., image and textual information) can be made more fully accessible to blind and low-viz users.

The second theme focused on tailoring explanations to diverse users in ways that honors situational, emergent contexts. In this section, we discussed two cases (aging-in-place and digital mental health), to highlight two central concerns integral to designing accessible XAI experiences. First, are the diverse and dynamic explainability requirements of people in AI/ML ecosystems. Second, is the need to explain not only the global and local explanations of the model, but also the model’s underlying alignment with the experience, trajectory, and spirit of underlying domain phenomena (i.e., mental health therapies).

XAI is a vibrant and active area of research that has made many advancements recently towards rendering AI models more comprehensible to humans. In providing some provocations at the intersection of XAI+accessibility, this article aims to spark new insights and ideas in working towards more informative and accessible interactions between people and artificially-intelligent machines.


Thank you to Mark S. Baldwin for helpful feedback and discussions on earlier drafts. This work is supported in part by the National Institute of Mental Health (T32MH115882). All opinions expressed herein are our own and do not reflect any institutional endorsement.


  1. Ackerman, S.J. and Hilsenroth, M.J. 2003. A review of therapist characteristics and techniques positively impacting the therapeutic alliance. Clinical Psychology Review. 23, 1 (Feb. 2003), 1–33. DOI:
  2. Adadi, A. and Berrada, M. 2018. Peeking Inside the Black-Box: A Survey on Explainable Artificial Intelligence (XAI). IEEE Access. 6, (2018), 52138–52160. DOI:
  3. Cameron, G. et al. 2017. Towards a chatbot for digital counselling. (2017), 1–7.
  4. Guidotti, R. et al. 2018. A Survey of Methods for Explaining Black Box Models. ACM Comput. Surv. 51, 5 (Aug. 2018), 93:1–93:42. DOI:
  5. Hendricks, L.A. et al. 2016. Generating Visual Explanations. Computer Vision – ECCV 2016 (2016), 3–19.
  6. Hind, M. 2019. Explaining Explainable AI. XRDS. 25, 3 (Apr. 2019), 16–19. DOI:
  7. Houts, P.S. et al. 1969. Patient-therapist interdependence: Cognitive and Behavioral. Journal of Consulting and Clinical Psychology. 33, 1 (1969), 40–45.
  8. Jain, S. and Wallace, B.C. 2019. Attention is not Explanation. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers) (2019), 3543–3556.
  9. Markoff, J. 2017. Opinion: Artificial intelligence could improve how we age. Washington Post.
  10. Miller, T. 2019. “But Why?” Understanding Explainable Artificial Intelligence. XRDS. 25, 3 (Apr. 2019), 20–25. DOI:
  11. Mohr, D.C. et al. 2018. A Solution-Focused Research Approach to Achieve an Implementable Revolution in Digital Mental HealthSolution-Focused Approach to an Digital Mental Health RevolutionSolution-Focused Approach to an Digital Mental Health Revolution. JAMA Psychiatry. 75, 2 (Feb. 2018), 113–114. DOI:
  12. Mohr, D.C. et al. 2017. IntelliCare: An Eclectic, Skills-Based App Suite for the Treatment of Depression and Anxiety. Journal of Medical Internet Research. 19, 1 (Jan. 2017). DOI:
  13. Mohr, D.C. et al. 2017. Three Problems With Current Digital Mental Health Research . . . and Three Things We Can Do About Them. Psychiatric Services. 68, 5 (May 2017), 427–429. DOI:
  14. Mynatt, E.D. et al. 2000. Increasing the Opportunities for Aging in Place. Proceedings on the 2000 Conference on Universal Usability (New York, NY, USA, 2000), 65–71.
  15. Pollack, M.E. 2005. Intelligent Technology for an Aging Population: The Use of AI to Assist Elders with Cognitive Impairment. AI Magazine. 26, 2 (Jun. 2005), 9–9. DOI:
  16. Ribeiro, M.T. et al. 2016. “Why Should I Trust You?”: Explaining the Predictions of Any Classifier. Proceedings of the 22Nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (New York, NY, USA, 2016), 1135–1144.
  17. Rieland, R. 2017. How Will Artificial Intelligence Help the Aging? Smithsonian.
  18. Selbst, A. and Powles, J. 2018. Meaningful Information and the Right to Explanation. Proceedings of Machine Learning Research (2018), 1.
  19. Simon, G.M. 2012. The role of the therapist: what effective therapists do. Journal of Marital and Family Therapy. 38, 1 (2012), 8–12.
  20. Wiegreffe, S. and Pinter, Y. 2019. Attention is not not Explanation. CoRR. abs/1908.04626, (2019).
  21. Wolf, C.T. 2019. Explainability Scenarios: Towards Scenario-based XAI Design. Proceedings of the 24th International Conference on Intelligent User Interfaces (New York, NY, USA, 2019), 252–257.
  22. Wolf, C.T. 2019. Professional Identity and Information Use: On Becoming a Machine Learning Developer. To Appear in Lecture Notes on Computer Science (2019).
  23. Wu, M. et al. 2008. Collaborating to remember: a distributed cognition account of families coping with memory impairments. (Apr. 2008), 825–834.

About the Authors

Christine T. Wolf is a Research Staff Member at IBM Research – Almaden. Her research leverages ethnographic and participatory-design techniques to study, design, build, and implement data-driven technologies for organizational work practices. Currently, AI sensemaking and explainability are a central focus of her work, driven by a desire to create informative, engaging, and accessible experiences in the “future of work.” She holds a PhD in Information and Computer Sciences from the University of California, Irvine, an MS in Information from the University of Michigan, Ann Arbor, and a JD from Southern Methodist University.

Kathryn Ringland, PhD in Informatics from the University of California, Irvine, is a NIH Ruth L. Kirschstein National Research Service Award Postdoctoral Fellow at Northwestern University. Her research interests include studying and designing games, social media, and assistive technology for people with disabilities. Her research contributes to the larger understanding of how people with disabilities experience and interact through social media and games. She also studies how disability is impacted and influenced by people’s other identities and the communities they live in. She is the co-founder of the Kaina Institute for Equitable Research, a non-profit that studies how technology can support people in marginalized communities.