A personal information management approach for people with low vision or blindness

Silas S. Brown and Peter Robinson

University of Cambridge Computer Laboratory
15 J.J. Thompson Avenue, Cambridge. CB3 0FD, UK

E-mail: {Silas.Brown, Peter.Robinson}@cl.cam.ac.uk

Abstract

A method and tool is presented that extends a word-processor so as to assist with organising personal notes and other information. It is usable by people who are blind or who have low vision, as well as by others, on a small display. The basic principle is to allow the user to annotate parts of the text with labels and then group together any parts of the text that have the same label. The tool was prototyped using the 4DML transformation system, was experimentally evaluated, and refinements are reported on.

Introduction

People write notes about their plans, contacts and other information to make up for limitation in their own memories. Designers and writers are also likely to have notes about their creative ideas. Many who are proficient with computers choose to enter their notes into a computer because of the ease with which they can be stored, copied, changed and organised. People with disabilities that impede their handwriting may also choose to make notes electronically.

Reviewing large amounts of notes can be unwieldy if they are not well-organised. However, the taking of notes is often done in haste, which does not facilitate good organisation, and the later re-organisation of notes takes time and effort even on a computer.

Both the reviewing and the re-organising of notes are made much easier if they can be spread out across a large display space. However, most computers limit the amount of display space, and hence less information can be displayed at any one time. This is particularly relevant when large print, speech or Braille must be used, either in order to overcome a disability or in order to overcome a situational restriction, and it is also the case when restrictive, smaller than usual displays are being used. Conventional time-management software does not address that problem; moving notes around can be a lengthy task even with the aid of user-interface macros, because of the excessive time that must be spent navigating around the notes, and perhaps also navigating around the software functionality.

This paper investigates an alternative approach to organisation that can be used both in ideal circumstances, when adequate display space is available and a user is not limited by disabilities or other special requirements, and in restrictive circumstances, when display space is unusually small or the user requires very large print or other media that limits the amount of information that can be simultaneously presented.

Related work

Portable electronic organisers aimed at totally blind people have been produced and sold commercially for some time. The earliest was probably Robotron Sensory Tools�Eureka A4, which has since been superseded by the Aria personal digital assistant; both have Braille-based input devices and speech output [8]. Blazie Engineering (a division of Freedom Scientific) produce a similar product called Braille 'n Speak. Another product of Blazie, BrailleLite (Figure 1), also features a refreshable Braille display for output, as does the BrailleNote organiser which is sold by Pulse Data. These products are expensive (the typical price for a BrailleLite is currently $6000 and it is notoriously fragile); also they are not usable by people who have not learned to type in Braille (which includes many blind people as well as the much larger numbers of people with low vision), and they do little to alleviate the navigation problem.

Figure 1: BrailleLite, a Braille-based note-taking device (photo: Hans Schou)

Other product designers have chosen to use voice for input and allow the user to organise voice recordings, producing what is effectively a portable dictation machine with added organisation features. Examples of this are the Parrot Voice Mate, which resembles a mobile telephone in appearance [5], and the "I.M.P." marketed by Voice Diary Inc. [9]. These devices are capable of recording arbitrary sound, and then navigating around it under the control of special-purpose controls on the device. Stifelman et al [7] extended this idea by using speech recognition for the navigation. However, all of these devices have their limitations: Re-organising an existing collection of notes requires excessive work (in the case of Stifelman et al's project, it does not appear to be possible at all); the devices cannot be used for input when silence is required (although headphone output is possible), and because the information is stored as a sound signal rather than in textual form, it is difficult for the device to exchange information with a word-processor or other text-based repository. These devices can be useful in some situations for keeping track of small quantities of notes, but their limitations often prevent their use with larger amounts of information.

Navigation on restrictive displays

Organising notes on a restrictive display that is unable to show significant amounts of the notes at a time, either because of low vision or because of restrictive hardware or both, is a different process from organising notes when everything is visible, because in the former case there is much more navigation involved.

There are several factors that make conventional personal information management software more difficult to use for people with low vision. Watts-Perotti and Woods [10] show that, in any application, when the amount of data that has to be displayed is significantly greater than what will fit on the display, users have to navigate around the data and can "get lost". The use of screen magnification, speech, Braille, or any other output method that cannot display as much information simultaneously as the software designer expected, can accentuate this problem, as can the use of unusually large font sizes that the designers did not anticipate. Large text implies that less of it can be fitted into a given area. If the layout is not flexible then it will require such things as laborious horizontal scrolling; if it is flexible then it may still require extensive navigation, much of which may be unnecessary because the layout is really designed for smaller font sizes.

Even if the layout is flexible, some screen-layout algorithms can fail to produce a readable layout when given the extreme constraints that very-large print presents; they have a maximum font-size-to-available-area ratio beyond which they break down. This can be demonstrated by viewing a website with frames and tables on a low-resolution screen at the largest font setting. Because of this layout problem, the programmers of Microsoft Windows XP decided to make most dialog boxes exempt from the font size requirements imposed by the accessibility-related "high contrast mode" and this is one reason why Windows applications can be inaccessible without adaptations.

An observation from a writer

It is instructive to consider the thoughts of modern literary writers who wish to organise their unfinished book manuscripts and who are familiar with computers. They have a similar problem as the users who wish to organise notes on restrictive displays - the amount of information that is being organised is much larger than can be displayed at one time, and hence navigation is needed. In the case of authors, this is not necessarily because of low vision or unusually restrictive display hardware, but because the amount of text they are dealing with (an entire book) is so large. However, the problem is similar in many respects.

In 1990, science-fiction writer Robert J. Sawyer wrote an essay entitled "A Writer's Word Processor" [6] in which he defended the use of WordStar for DOS in creative composition. His observations about how he and other authors reorganise their written ideas during the writing process is useful for anyone implementing a notes organisation system, especially if it must cope with navigation around data that is much larger than the display. Sawyer makes a distinction between the "typewritten page" metaphor of most word-processors and the "long-hand manuscript page" metaphor that many writers prefer to work with. After observing that "typing is a top-down, linear process, not at all conducive to the intuitive, leaping-here-and-there kind of thought human beings are good at," Sawyer goes on to explain that the strength of the long-hand page metaphor is its support for many concurrent editing operations to be in intermediate states at any one time:

However, WordStar requires extensive learning before it can be used to its full functionality, and it is not available on every kind of device used by people with low vision. Moreover, the presence of too many incomplete editing operations can cause a degree of cognitive overload if they are not clearly labelled; Henney alludes to this in his article on the software design pattern of "stable intermediate forms" [4] and it is especially true if the user is multitasking between editing and some other task, as will be the case if the user is utilising "dead time" (short periods of waiting time during travelling and other activities) for organisation. Hence an ideal organisation system will not only support multiple incomplete operations that do not interfere with each other but will also ensure that all intermediate states are valid and usable.

Organising notes from a word-processor

We have designed and implemented a new organisation tool that can be used from within a simple word-processor. Most word-processing and similar software allows the print size to be increased, and many programs allow the colours, style of text, and other typesetting parameters to be adjusted. This is helpful to both partially-sighted and dyslexic people [3]. Moreover, increasing the print size on a word-processor, as opposed to a specialised personal information management system, is less likely to cause layout problems, because the layout of a word-processor can be made to be very simple and able to cope with any font size that will fit on the display. Word-processors are also available on the numerous commercially-available computers that are small enough to fit in one's pocket, hence allowing the taking and reviewing of notes almost anywhere. Most of these can be linked to a desktop computer allowing information to be transferred when necessary. These mobile devices can be used by people with moderate print disabilities provided that the input method is suitable (a good-quality physical keyboard is generally best), the software can be made to display in large print (or at least to put the essential parts of the display in large print), and the device can be given adequate physical protection if accidents are expected.

A common method of organising notes is to group them into a small number of categories, each of which may have further divisions into categories; usually the broad grouping is done first and the finer grouping later. Conventionally this is done by moving or copying the text from one place to another, which can be time-consuming due to the overhead of navigation. An alternative is to leave the text where it is and to add labels to it as brief markup; once the labelling at each level has been completed, software can be invoked to automatically regroup the text according to the labels.

Our system uses a word-processor to edit the labelled text. Any of the symbols available in the word-processor can be reserved for use as an indicator that the next word is a label; that label will specify the category of all text that follows it, up to the next label. For example, in the following input, the symbol …  (which has been made available using one key-press) has been reserved to indicate that the next word is a label:

A function is available to take the word-processor's text and group together any parts of it that have the same label. Thus the system will reorganise the above example into the following output:

This re-organisation can be triggered by using one key-press to invoke an appropriate macro. Since the output is also a valid input, the user can perform the organisation task incrementally and the re-organisation function can be invoked at any time on a partially-finished document. From the user's point of view, the system is simply a word-processor with an additional "put it into order" function. After this function has been invoked, the user can find a particular category by performing a single search operation.

This re-organisation system is called "notesorg" (short for Notes Organiser), to reflect the fact that category names are often written as lower-case abbreviations so as to facilitate typing them repeatedly.

Use of 4DML

The re-organisation function was prototyped using the 4DML transformation system. The workings and benefits of 4DML have previously been published elsewhere [2, 1]; we now show that 4DML can be applied to this problem. By adding a suitable header to the wordprocessor's text, it can be made to be valid input in 4DML's Matrix Markup Language (MML), with the reserved "labelling" character as one of the separators:

The 4DML transformation utility will then parse the input, marking it up with elements as appropriate and grouping them by category. An appropriate 4DML model will cause it to format the output as desired, outputting one category at a time.

The advantage of using the 4DML approach is that different models can be experimented with to show the notes in different forms. For example, the notes can be output with HTML markup, with links between the different sections for browsing, or they can be output with TeX markup for typesetting. Moreover, sometimes notes are categorised on several dimensions such as project, priority, time-scale, location, or needed resources; in this case the model can be adjusted to group by any of these categories, transposing the structure as necessary, although this is rarely needed in practice.

The 4DML prototype was ported to run on a mobile device (Figure 2) and this method was effectively used to organise a large amount of personal notes in large print on that device, with the text editor displaying 4 lines of 20-25 characters at a time. The same functionality was also available on a desktop PC, and it was made available over the World Wide Web by using a simple form with a large text box and a submit button that causes the re-organisation to be effected (however this last mode of access can cause privacy concerns if the server is untrusted and the notes are confidential).

Figure 2: Working with notesorg on a mobile device's wordprocessor (actual size 14cm x 12cm)

Because 4DML ran quite slowly on the mobile device, the prototype was later re-implemented in lower level code. However, it had still been useful to employ 4DML in the design stage when different approaches were being experimented with, since this could be achieved without excessive coding overhead.

Experimental evaluation and refinement

A group email discussion was held with 12 visually-impaired people, 10 of whom were totally blind and 2 had some sight. Before being told of the existence of our system, all participants were asked to explain what strategies they already had for keeping their notes organised. All agreed that organising notes was a problem. Reported strategies included avoiding the use of notes and relying on one's own memory, taking on fewer tasks so that fewer notes were needed, and employing a sighted helper to do the work. Two older individuals maintained that the best solution was to emboss Braille manually into a ring-binder which allows its pages to be re-ordered, but they were concerned about the fact that an increasing number of young blind people are not being adequately trained in Braille reading and writing and will not be able to do this in future.

Those that had tried to use time-management software on non-visual electronic organisers such as BrailleLite described it as "too complicated" and the lack of the ability to synchronise with a desktop computer was also identified as a problem. A recurring problem was that circumstances often precluded the use of the software as intended. For example, the user wishes to quickly enter some notes which are not related to any appointment, but the software is ready to accept timed appointments, and switching modes is too complex given that the notes must be taken quickly, so the notes are entered as timed appointments, hence introducing a daily task of moving them around, which is a lengthy process when the user has little or no sight and the designer did not anticipate such misuse.

The participants were then asked to try our new system. For a realistic evaluation it is necessary to allow users to organise their own notes, but this makes it difficult to measure the system's effectiveness quantitatively due to privacy concerns. However, feedback was very positive and some users began to use it regularly.

One user was bothered by the new system's inability to issue alarms at specific times and so still uses his PDA's time management software for timed, alarmed appointments, but the fact that he uses the new system for everything else means the use of the previous time management software is significantly simpler due to the reduced amount of non-timed information that must be navigated through in the process. Sometimes he combines the two systems by creating a category that is for his attention at a specific time in the future and then setting an alarm to remind him to look at that category.

The most useful result of the evaluation is the refinements that were suggested as a result of long-term use. The following refinements were implemented:

• Generating a list of categories at the top of the output for a brief overview (in order to fit more onto a small screen, they were all listed in one paragraph);
• Accepting an alternative form of markup for re-categorising only one line without affecting subsequent text, so that an item can be re-categorised after being found in a search, and new notes can be recorded more quickly while in the process of organising other notes (we implemented this by accepting two successive instances of the special labelling character as indicating that the label applies only to the current line);
• Using a special category to specify, at any time, partial instructions about which categories to place first and in which order (these instructions are preserved across the organisation so as to support incremental organisation), to save having to navigate to the top of the file and create the categories in the correct order with empty items then navigate back;
• Specifying that any categories which are numbers (1, 2, etc) are always moved to the top and placed in numerical order, to support the rapid indication of high-priority tasks;
• Specifying that a category called '0' will automatically be renumbered to '1' and will cause all other numerical categories to increase by 1, to support situations where the user thought he or she was dealing with the highest priority tasks but then found something higher;
• Supporting these numerical conventions within the context of named categories also, so for example categories called "morning1", "morning2" etc. will be relocated to occur immediately prior to any category called "morning"; if there is a "morning0" then the above renumbering also applies. However, in the special case when there is no "morning1" or there is a discontinuity in the numbering, the higher numbers will be located after the unnumbered category, to support particularly low-priority information.

Regular users considered that the version with these refinements was an improvement on the first version. The refinements are "orthogonal" in the sense that it is not necessary for a user to learn them before starting to use the system. In each case, the aim was to provide simple functionality that will not add undue cognitive load to the organisation task. Any erroneous markup is simply left alone and this does not stop the rest of the file from being processed.

Additionally, it was found that users who generated more categories than could be listed in one or two pages on the small screen or a few seconds of speech at the user's preferred speech rate, would tend to want to reduce the number of categories in their "working set", for example by saving the ones that will not be used for some time to a separate file for re-insertion at a later date, by merging some categories together, by rendering some obsolete and deleting them, or by filing longer-term notes in another system. This is probably due to the nature of short-term memory when dealing with the categories. However, the frequency with which this happened was reported as being much less than the frequency with which the users became frustrated with their previous time-management software (for those who had been using such software) and the method of resolving the problem was simpler.

Conclusion

We have described a tool to assist with the organisation of personal notes from within a word-processor. The method is usable with restrictive displays and can be used by people with low vision, blindness, or temporary situationally-induced impairments. Comparatively little training is required.

Further work could include the development of tools that use variations on this method to assist with such activities as managing the code in a large software project, managing the files on a disk, managing one's email or other messages, or even managing non-textual information such as audio, images or video. It can already be used as an aid to organising essays and papers. Additionally, perhaps fully-sighted people will eventually wish to re-organise their notes on ever smaller devices, such as mobile phones and wristwatches, especially if the disorganised text has already been input elsewhere.

Regrettably, the "Psion" range of pocket computers that were used in this experiment is no longer being produced. The newer pocket devices have poorer keyboards or are pen-based, which is difficult for those who cannot write quickly. They also tend to utilise displays with reduced contrast and fonts that cannot always be made larger, and they have shorter battery life and are more expensive and fragile. It is hoped that these issues will be addressed by manufacturers in the future.

References

1. Silas S. Brown. Conversion of notations. Technical Report UCAM-CL-TR-591, University of Cambridge, Computer Laboratory, June 2004. http://www.cl.cam.ac.uk/TechReports/UCAM-CL-TR-591.pdf.

2. Silas S. Brown and Peter Robinson. Transformation frameworks and their relevance in universal design. Universal Access in the Information Society, 3(3-4):209-223, 2004.

3. Peter Gregor and Alan F. Newell. An emperical investigation of ways in which some of the problems encountered by some dyslexics may be alleviated using computer techniques. In Proceedings of the Fourth International ACM Conference on Assistive Technologies ASSETS 2000, pages 85-91, Nov 2000.

4. Kevlin Henney. Stable intermediate forms: A foundation pattern for derisking the process of change. In Proceedings of the Ninth European Conference on Pattern Languages of Programs, 2004.

5. Parrot S.A. Company. Voice Mate. http://www.voice-assistant.com/.

6. Robert J. Sawyer. A Writer's Word Processor, 1990, 1996. http://www.sfwriter.com/wordstar.htm.

7. Lisa J. Stifelman, Barry Arons, Chris Schmandt, and Eric A. Hulteen. Voicenotes: A speech interface for a hand-held voice notetaker. In Proceedings of ACM INTERCHI93 Conference on Human Factors in Computing Systems, Voices and Faces, pages 179-186, 1993.

8. Robotron Sensory Tools. Eureka A4. http://www.sensorytools.com/eureka.htm.

9. Voice Diary Inc. I.M.P. http://www.voicediary.com/.

10. Jennifer Watts-Perotti and David D. Woods. How experienced users avoid getting lost in large display networks. International Journal of Human-Computer Interaction, 11(4):269-300, 1999. ISSN 1044-7318.