TOWARDS LESS COMPLEX E-GOVERNMENT SERVICES FOR PEOPLE WITH LOW VISION: AN EVALUATION OF FIVE SPANISH PUBLIC WEB SERVICES
Aritz Sala, EGOKITUZ, University of the Basque Country (UPV/EHU), Informatika Fakultatea, Donostia, Spain asala020@ikasle.ehu.eusMyriam Arrue, EGOKITUZ, University of the Basque Country (UPV/EHU), Informatika Fakultatea, Donostia, Spain myriam.arrue@ehu.eus
J. Eduardo Pérez, EGOKITUZ, University of the Basque Country (UPV/EHU), Informatika Fakultatea, Donostia, Spain juaneduardo.perez@ehu.eus
Sandra M. Espín-Tello, EGOKITUZ, University of the Basque Country (UPV/EHU), Informatika Fakultatea, Donostia, Spain sandramartina.espin@ehu.eus
Abstract
Online public services may alleviate many of the inconveniences citizens have to face when accessing public services in the real world. People with low vision are one of the collectives who could benefit from e-services the most, as they can be accessed anytime, anywhere and with the help of their usual assistive technology. Many public e-services are provided by means of accessing, completing and submitting an online-form. However, this process is not always accessible for people with low vision and completing all the steps often becomes a complex task. It is necessary to identify the most difficult tasks and steps in public services in order to improve the user experience in e-government services. This paper presents a user study of five public Spanish e-services with ten participants, five of them with low vision. The performance of participants during the process of fulfilment of the online forms has been analysed. The results indicated the main barriers they found in each step of the process and highlighted the best approaches for implementing more accessible e-services.
Introduction
The use of information and communication technologies (ICTs) in public administration, commonly referred to as e-government, is increasing around the world [7]. The Web is used for delivering government information and services to the citizens more easily and quickly, at a lower cost [4, 5], and by means of online-forms as the predominant style of interaction [6, 10]
People with low vision (PwLV) may be one of the collectives who could benefit from e-services the most, but taking into account that Web experiences are fundamentally based on visual elements, it is not surprising that PwLV are prone to experience barriers when accessing e-government services.
Our main research objective is to study the complexity of public online-forms for PwLV in order to define guidelines for developing enhanced public e-services as well as virtual assistance methods for helping users in the interaction with the e-services. For this, we focused on the complexity analysis of two main aspects of e-services: the required process to fulfil the online-forms and the accessibility of the interaction components in online-forms. As a first step, we carried out a preliminary study of the complexity of five Spanish public e-services. In this paper, we briefly present the results of this study and we draw up the following steps of our research.
Related work
The Web Content Accessibility Guidelines (WCAG) [11] are the most well-known and applied guidelines for accessibility evaluation of websites. Its latest version (2.1) [12], structures the guidelines into four principles: perceivable, operable, understandable, and robust. The document includes success criteria for checking the fulfilment of each one of these principles. However, evaluation of forms based on these guidelines may be tedious as the success criteria applicable to forms are spread across these principles along with others related to another type of web content.
Some efforts have been focalized to order and group the success criteria included in WCAG by user profiles. For example, the Barrier Walkthrough Method proposed by Brajnik [2] groups the success criteria applicable to forms by user categories, along with another type of criteria. Most recently, the World Wide Web Consortium (W3C) has launched the Accessibility Requirements for People with Low Vision [13], which describes the needs of PwLV when accessing electronic content. This is a working draft containing general guidelines for information presentation (font, text size, style, etc.) but it does not refer to the common interaction elements in online-forms (i.e. input fields, field labels, action buttons, feedback, etc.).
Related to measuring the complexity of online-forms, Elliman et al. [3] developed a set of Bespoke Online Form Selection (BOFS) metrics oriented to analyse the complexity of forms for older adults. These metrics were evaluated through a study that involved 80 participants [6]. However, the evaluation was carried out based on qualitative data obtained by applying thinking aloud techniques and did not analyse real user interaction data.
Description of public e-services
Accessing public services on the web often requires filling in an online-form. The fulfilment of the online-forms usually follows a process including several steps. One of our objectives is to identify the different interaction events in each step in order to identify the most complex ones for PwLV. In the preliminary study carried out, we selected five public Spanish e-services for the study. All of them were related to the task "make an appointment" in five different websites: DNI, the public website of the Ministry of Home Affairs; SPE, the public website of the Public Service of State Employment; SGS, the public website of Social Security; DON, the public website of the Donostia-San Sebastian Town Hall; and OSA, the public website of the public health service in the Basque Country.
These public e-services were selected due to all of them offering the same functionality but with different fulfilment processes and online-forms (regarding the number of pages, data to be inserted, interaction elements in the form, etc.). This gave us the opportunity to evaluate different approaches for completing the same type of task.
The common steps in the fulfilment process of these e-services include the following: providing personal information ("User Identification"), selecting the service ("Service Selection"), selecting the location ("Location Selection"), selecting the date and time ("Date Selection"), answering security questions ("Security Question"). These steps are implemented in the e-services but using different approaches. As an example, Figure 1 shows different approaches for the "User Identification" step implemented in the selected e-services. It can be appreciated that each e-service includes a different number and type of interaction elements in this step: DON and OSA e-services include two interaction elements whereas the others include more (DNI includes six interaction elements, SGS more than ten and SPE four). Regarding the type of elements, DON includes INPUT TEXT elements, OSA includes INPUT TEXT and an optional CALENDAR, DNI includes several INPUT TEXT elements and a CAPTCHA, SGS and SPE are the ones showing more types of elements (INPUT TEXT, CHECKBOX, SELECT, RADIO, and CAPTCHA).

Evaluation of e-services
The evaluation of the selected e-services was carried out by measuring their complexity based on BOFS metrics [3], performing conformance to accessibility guidelines evaluation, and by user testing. Table 2 shows the summary of the values obtained for each e-service in the evaluation of complexity and conformance to accessibility guidelines.
According to the obtained values for BOFS metrics, the ranking of evaluated e-services is, from less to more complex, as follows: SPE (58), DON (75), OSA (187), DNI (390) and SGS (459). More information about the analysis of e-services according to BOFS metrics can be found in [8].
All the selected public e-services included information about accessibility in a dedicated web page (through a link labelled "Accessibility"). WCAG guidelines were mentioned in every case and the conformance level to these guidelines was specified: DNI and SPE Double-A of WCAG 1.0, SGS and OSA level AA of WCAG 2.0, DON level AAA of WCAG 2.0.
USER IDENTIFICATION | ||||||||||
---|---|---|---|---|---|---|---|---|---|---|
BUT | INP | RAD | MAP | SEL | HYP | CAL | Total | Required | Num. Pages | |
SPE | 2 | 5 | - | - | - | - | - | 7 | 6 | 2 |
DON | 1 | 2 | - | - | - | - | 1 | 4 | 4 | 1 |
OSA | 1 | 2 | - | - | - | - | 1 | 4 | 4 | 1 |
DNI | 2 | 5 | - | - | - | 2 | - | 9 | 6 | 1 |
SGS | 1 | 4 | - | - | 1(3) | - | - | 5+1(3) | 3+1(3) | 1 |
SERVICE SELECTION | ||||||||||
SPE | 1 | - | 1(9) | - | - | - | - | 1+1(9) | 1+1(9) | 1 |
DON | - | - | - | - | - | - | - | - | - | - |
OSA | 7 | - | 2(2) | - | - | - | - | 7+2(2) | 3+2(2) | 1 |
DNI | - | - | - | - | - | 3 | - | 3 | 1 | 1 |
SGS | 2 | - | 1(10) | - | - | - | - | 2+1(10) | 1+1(10) | 1 |
LOCATION SELECTION | ||||||||||
SPE | - |
- | - | - | - | - | - | - | - | - |
DON | - | - | - | - | - | - | - | - | - | - |
OSA | - | - | - | - | - | - | - | - | - | - |
DNI | - | - | - | 1(50) | - | 28 | - | 1(50)/28 | 2/3 | 2 |
SGS | 2 | - | 1(8) | - | - | - | - | 2+1(8) | 2 | 1 |
DATE SELECTION | ||||||||||
SPE | - | - | - | - | - | - | - | - | - | - |
DON | 6 | 1 | var | - | - | - | 1 | 8+var | 3 | 2 |
OSA | 8 | 1 | 1(2)+var | - | - | - | 1 | 9+1(2)+var | 3+1(2) | 2 |
DNI | - | - | - | - | - | var | - | var | 1 | 1 |
SGS | 4 | - | var | - | - | - | - | 4+var | 4 | 2 |
SECURITY QUESTION | ||||||||||
SPE | 1 | 1 | - | . | - | 2 | - | 4 | 2 | 1 |
DON | - | - | - | - | - | - | - | - | - | - |
OSA | - | - | - | - | - | - | - | - | - | - |
DNI | 1 | 1 | - | - | - | 1 | - | 3 | 2 | 1 |
SGS | 1 | 1 | - | - | - | - | - | 2 | 2 | 1 |
|
BOFS |
WCAG |
BWM |
|||
---|---|---|---|---|---|---|
Ranking |
Conformance |
Total barriers detected |
Minor |
Significant |
Critical |
|
SPE |
58 |
AA (1.0) |
3 |
2 |
1 |
– |
DON |
75 |
AAA (2.0) |
2 |
2 |
– |
– |
OSA |
187 |
AA (2.0) |
5 |
1 |
4 |
– |
DNI |
390 |
AA (1.0) |
5 |
– |
4 |
1 |
SGS |
459 |
AA (2.0) |
4 |
2 |
1 |
1 |
The presence and severity of accessibility barriers in the online-forms of each e-service were evaluated based on the Barrier Walkthrough Method [2] for PwLV. The online-forms with more barriers detected were DNI and OSA, with a total of five in both of them (four significant and one critical in DNI, and four significant and one minor in OSA), followed by SGS with four (one critical, one significant and two minor), SPE with three (one significant and two minor), and DON with two minor barriers. Insufficient visual contrast was a common barrier in all reviewed online-forms. In two of them (DNI and SSG) the severity of the barrier was critical, in one (OSA) was significant, and in the other two (SPE and DON) was minor. Inflexible page layout was a significant barrier in three online-forms (OSA, DNI, and SPE), and minor in one (SGS). Missing layout clues were identified as a significant barrier in two online-forms (DNI and OSA), and as a minor in another two (SPE and DON). Significant barriers were detected concerning rich images badly positioned, as well as image maps in DNI, widely formatted forms in OSA, and images used as titles in SGS. We also detected minor barriers about too long lines of text in two online-forms (OSA and SGS).
Participants without visual impairment |
Participants with low vision |
||||||||
---|---|---|---|---|---|---|---|---|---|
ID |
Gender |
Age |
Visual disability |
Assistive technology |
ID |
Gender |
Age |
Visual disability |
Assistive technology |
P1 |
M |
59 |
– |
– |
P6 |
M |
57 |
Legally blind |
Windows Magnifier |
P2 |
M |
28 |
– |
– |
P7 |
M |
23 |
Legally blind |
Browser Zoom |
P3 |
F |
35 |
– |
– |
P8 |
F |
28 |
35% visual capacity |
Browser Zoom |
P4 |
M |
33 |
– |
– |
P9 |
M |
19 |
Legally blind |
Browser Zoom |
P5 |
F |
24 |
– |
– |
P10 |
F |
31 |
20% visual capacity |
Browser Zoom |
User testing
Five participants of the user tests that were carried out belonged to the group of PwLV (P6-P10). P6, P7 and P9 were legally blind; and P8 and P10 had less than 40% of visual capacity (35% and 20% respectively). All of them performed the experiment in one session, using their usual assistive technology that was the browser zoom for all except for P6 who used the Windows Magnifier. The other five participants (P1-P5) had no visual disability. The mean age of participants was 33.8 years (SD 13.2), and four of them were females (P3, P5, P8 and P10). Table 3 shows the detailed information of the participants.
All participants reported having high experience with computers and Web browsing, and all PwLV reported having more than seven years of experience with their assistive technology. All the experimental sessions with participants were carried out in a laboratory at the Computer Science Faculty of the University of the Basque Country. The same computer was utilized in all sessions with an additional widescreen LCD monitor (aspect ratio 16:10) with a diagonal size of 24 inches and the display resolution set to 1920x1200 pixels. Before starting the session, PwLV were encouraged to adjust the assistive technology to meet their preferences. All sessions were video-recorded and the interaction data (page load, start and end of tasks, links visited, scroll, clicks, etc.) was collected with the RemoTest platform [1, 9].
During the experimental session, participants were asked to search for the online-form for "make an appointment" on the website and fulfilling it in each e-service. The time for completing each task was limited to ten minutes. Participants could ask for help in any step.
Results
Table 4 shows the results achieved by each user group (LV-users with low vision and CG-users without low vision) for the different steps of the five e-services. The average time corresponds to the time needed to complete a step on average for the users in a group. The failed attempts measurement counts the average number of erroneous attempts when completing a step for the users in a group. The average clicks measurement counts the average number of mouse clicks along a specific step for the users in a group. In addition, the percentage of participants in each group who reached and completed the step is shown in the table.
USER IDENTIFICATION |
||||||||
---|---|---|---|---|---|---|---|---|
|
AVERAGE TIME (sec.) |
FAILED ATTEMPTS |
AVERAGE CLICKS |
COMPLETED (%) |
||||
CG |
LV |
CG |
LV |
CG |
LV |
CG |
LV |
|
SPE |
51.0 |
51.4 |
0 |
0.4 |
5.8 |
5.4 |
100 |
100 |
DON |
72.8 |
99.6 |
0 |
0.2 |
0 |
0.2 |
100 |
100 |
OSA |
24.0 |
71.6 |
0.6 |
0 |
0.6 |
0 |
100 |
100 |
DNI |
57.1 |
95.8 |
0.2 |
0 |
6.4 |
9.0 |
100 |
80 |
SGS |
39.9 |
129.2 |
0 |
0.2 |
2.2 |
4.0 |
100 |
100 |
SERVICE SELECTION | ||||||||
SPE |
16.0 |
26.7 |
0 |
0.4 |
1.0 |
1.1 |
100 |
100 |
DON |
– |
– |
– |
– |
– |
– |
- |
- |
OSA |
20.3 |
33.4 |
0.2 |
0.4 |
5.0 |
11.5 |
100 |
100 |
DNI |
8.1 |
10.4 |
0 |
0 |
1.0 |
1.3 |
100 |
100 |
SGS |
29.2 |
34.1 |
0 |
0 |
2.8 |
5.3 |
100 |
100 |
LOCATION SELECTION | ||||||||
SPE |
– |
– |
– |
– |
– |
– |
- |
- |
DON |
– |
– |
– |
– |
– |
– |
- |
- |
OSA |
– |
– |
– |
– |
– |
– |
- |
- |
DNI |
4.0 |
16.3 |
0 |
0.5 |
2.2 |
2.8 |
100 |
100 |
SGS |
13.5 |
13.8 |
0 |
0.3 |
2.5 |
3.7 |
100 |
100 |
DATE SELECTION | ||||||||
SPE |
– |
– |
– |
– |
– |
– |
– |
– |
DON |
29.7 |
31.9 |
0 |
0 |
3.4 |
1.2 |
100 |
100 |
OSA |
18.9 |
35.5 |
0 |
0 |
10.0 |
6.8 |
100 |
100 |
DNI |
18.2 |
10.0 |
0 |
0 |
5.2 |
4.0 |
100 |
100 |
SGS |
14.3 |
22.0 |
0 |
0 |
4.0 |
3.0 |
100 |
50 |
SECURITY QUESTION | ||||||||
SPE |
16.3 |
26.6 |
0.2 |
0.8 |
1.4 |
2.3 |
100 |
100 |
DON |
– |
– |
– |
– |
– |
– |
- |
- |
OSA |
– |
– |
– |
– |
– |
– |
- |
- |
DNI |
9.9 |
16.9 |
0.2 |
2.0 |
3.0 |
1.7 |
100 |
100 |
SGS |
25.1 |
28.8 |
0.8 |
0 |
1.3 |
1.3 |
100 |
80 |
average time for completing a step differed among the five steps identifies. The "User Identification" step obtained the largest average time values of all the e-services tested for both user groups. It is not surprising as this step is the one requiring more input from the participants. Comparing the average time needed by both groups for this step, participants with low vision spent more time in all e-services. The average time for completing the "User Identification" step increased in more than 35% for the participants with low vision in comparison with the other user group in four of five e-services tested (SGS: 223.8%, OSA: 198.3%, DNI: 67.8%, and DON: 36.8%). The SPE e-service obtained similar values for both groups (51 seconds for participants without low vision and 51.4 seconds for the other group). The e-service obtaining the best average time for participants without low vision was OSA (24 seconds) whereas for participants with low vision was SPE (51.4 seconds). The e-service with the worst average time value was DON (72.8 seconds) for participants without low vision and SGS (129.2 seconds) for participants with low vision.
For the "Service Selection" step the group of participants with low vision obtained larger average times than the other user group in all e-services tested, increasing from 16.8% in SGS to 66.9% in SPE. The ranking from lower higher times is the same for both groups: DNI, SPE, OSA and SGS. All participants completed this step except one with low vision in the DNI e-service. Regarding the number of clicks along the "Service Selection" step, participants with low vision made more than the double of clicks in the OSA e-service (5 clicks for participants without low vision and 11.5 clicks for participants with low vision).
The "Location Selection" step is implemented only in two of the five e-services tested (DNI and SGS). The average time for completing this step in the SGS e-service was similar. In contrast, the average completion time of the DNI e-service increased by 307.5% in participants with low vision (16.3 seconds) in comparison to the other user group (4 seconds).
The "Date Selection" step in the DNI e-service was one of the few times that participants with low vision outperformed the other user group, decreasing the average completion time by 45.1% (10 seconds versus 18.2). Only half of the participants with low vision could complete this step in SGS e-service.
The "Security Question" is the one that obtained the highest values of failed attempts in all the e-services tested for participants without low vision (0.2 for SPE, 0.2 for DNI and 0.8 for SGS). Regarding participants with low vision, this step also obtained the highest values of failed attempts of all steps studied (2 for DNI and 0.8 for SPE). However, the SGS e-service did not obtain any failed attempt by the participants with low vision, whereas the other user group obtained 0.8. Something similar occurred in "User Identification" step as participants with low vision got none failed attempts, and the other group of users got a higher value for the OSA (0.6) and the DNI e-services (0.2). In other cases, the participants with low vision obtained equal or slightly higher values for failed attempts.
Discussion
The "User Identification" step is not the one displaying the highest number of elements to the users (average total elements number is 6 whereas other steps such as "Service Selection" has an average total number of 8.5) but it obtained the largest average time values for all e-services and user groups. In some cases, we detected that specifically one item of the process which is related to the introduction of date of birth is causing trouble for participants with low vision. DON and OSA e-services provide a calendar assistant for that, it launches automatically only in DON. This has hindered the process: all participants (with and without low vision) were unable to insert the data so they deactivate the assistant and typed it directly into the input text element. This increased the time spent for both user groups. The assistant in OSA service was optional and it was used with success by one participant without low vision and two participants with low vision. However, the worst average time for participants with low vision was in SGS e-service. This is the only service and step displaying a SELECT element to users which caused many problems for participants with low vision.
Both groups of users perform better in DNI e-service for "Service Selection". It could be directly related to the number of elements displayed to the users as in this service only 3 hyperlinks are displayed in order to select the service required by the user. The other services contain radio buttons with a different number of options and buttons that users have to click after selecting their option. In some cases, there are several buttons, for example, OSA e-service shows 5 different buttons (back, home, next, etc.). The combination of radio button elements with buttons could be disorientating for participants, this could be the cause of the increase in click numbers in OSA e-service for participants with low vision.
The "Location Selection" step included an image map element in DNI e-service. This map shows all the provinces in Spain and the user is required to select one of them. This interaction element increased the time for completing the step for participants with low vision. In SGS e-service the location was selected by the use of radio buttons. In this case, those interaction elements worked well for both user groups, as the text for each option was shorter than the texts for radio buttons in "Service Selection" step.
The "Date Selection" step was implemented based on different approaches: a calendar assistant was displayed in DON and OSA whereas radio buttons and hyperlinks were used in other e-services. The calendar assistant seems to increase the average time for participants with low vision. This type of interaction element needs more analysis in order to identify the specific problems and design new accessible approaches. The number of clicks for accomplishing this step is lower in the case of participants with low vision. This issue needs more analysis as it can be due to the variable options shown to the user in order to select a date.
The "Security Question" step was implemented based on two different approaches: a trivia question used in SGS and a CAPTCHA used in DNI and SPE. This step was the one with the most failed attempts for both user groups. Between both approaches, the CAPTCHA was the one with more failed attempts. The trivia question in SGS e-service was successful for participants with low vision. This group of users seem to be comfortable when reading plain text and directly typing the answer.
Conclusions
Results from the user study pointed out many aspects to take into account when developing public e-services in order to facilitate the steps of fulfilling the online forms. All the e-services tested claim to be accessible according to some degree of the conformance levels of WCAG. However, not only the accessibility of the elements in the forms should be considered in e-government services but also it is necessary to pay attention to the process required to fulfil them. In addition, the number of questions and elements displayed to users may not be a direct complexity indicator for the e-service as suggested by BOFS metrics.
Future work will be focused on performing more analysis of the collected data. All the findings will be confirmed by involving more participants in the user testing and including more public e-services into the study. This will lead us to define appropriate guidelines and mechanisms for ensuring an improved user experience for PwLV.
Acknowledgments
The authors are members of the ADIAN research team, supported by the Basque Government, Department of Education, Universities and Research under grant IT980-16. The authors also thank all the individuals who participated in the study and the associations Begiris and Begisare, which helped in the recruitment.
References
- M. Arrue, X. Valencia, J. E. Pérez, L. Moreno and J. Abascal. 2019. Inclusive Web Empirical Studies in Remote and In-Situ Settings: A User Evaluation of the RemoTest Platform. In International Journal of Human–Computer Interaction, vol. 35, no. 7, pp. 568–583.
- G. Brajnik. 2011. Barrier Walkthrough Method. https://users.dimi.uniud.it/~giorgio.brajnik/projects/bw/bw.html (Access 24 September 2020).
- T. Elliman, A. Money, L. Lines, and S. Fernando. 2008. Online form complexity assessment for developing assistive technologies. In Proceedings of the 14th Americas Conference on Information Systems (AMCIS 2008).
- Å. Grönlund and T.A. Horan. 2005. Introducing e-gov: history, definitions, and issues. In Communications of the association for information systems, vol. 15, no. 1, pp. 713–729.
- J. Guida and M. Crow. 2009. E-government and e-governance. In ICT4D: Information and Communication Technology for Development. T. Unwin, Eds. Cambridge University Press, pp. 283– 320.
- A. G. Money, L. Lines, S. Fernando, and A. D. Elliman. 2011. e- Government online forms: design guidelines for older adults in Europe. In Universal Access in the Information Society, vol. 10, no. 1, pp. 1–16.
- Organisation for Economic Co-operation and Development (OECD). 2003. The case for e- government: Excerpts from the OECD report "The E-government Imperative". In OECD Journal on Budgeting, vol. 3, no. 1, pp. 61–96.
- A. Sala, M. Arrue, J. E. Pérez, and S. M. Espín-Tello. 2020. Measuring complexity of e-government services for people with low vision. In Proceedings of the 17th International Web for All Conference (W4A’20). Article 21.
- X. Valencia, J. E. Pérez, U. Muñoz, M. Arrue and J. Abascal. 2015. Assisted interaction data analysis of web-based user studies. In 15th IFIP TC 13 International Conference on Human-Computer Interaction (INTERACT 2015). Part of the Lecture Notes in Computer Science book series, vol. 9296, pp. 1–19.
- M. Winckler, R. Bernhaupt, and F. Pontico. 2010. Challenges for the development of user interface pattern languages: A case study on the e-government domain. In IADIS International Journal on WWW/Internet, vol. 8, no. 2, pp. 59–84.
- World Wide Web Consortium (W3C). 2008. Web content accessibility guidelines (WCAG) 2.0. https://www.w3.org/TR/WCAG20/ (Access 24 September 2020).
- World Wide Web Consortium (W3C). 2018. Web content accessibility guidelines (WCAG) 2.1. https://www.w3.org/TR/WCAG21/ (Access 24 September 2020).
- World Wide Web Consortium (W3C). 2016. Accessibility requirements for people with low vision. http://www.w3.org/TR/low-vision-needs/ (Access 24 September 2020).