11. SUMMARY OF FINDINGS
11.1 General observations of searches
During the Search Engine task, eleven of the sighted users chose to use the search facility only, two just followed links on the page and seven followed links and performed searches. Nine of the visually impaired users chose to use the search facility only, six just followed links on the page, four tried a combination of following links and searching and one user abandoned the task before trying either.
In both cases (sighted and visually impaired), users who opted to use the search facility were able to execute the task more quickly than those who followed hypertext links from the Home Page. This was particularly true for the visually impaired sample, as the link to the weather provided by the Search Engine was towards the bottom of the Home Page, which contained around 180 hypertext links followed by a second page containing around 200 hypertext links to specific cities. The sighted users were able to browse through the links faster than the visually impaired users because they could quickly scan a page visually and pin-point relevant links. However, presentation of the links was rather cumbersome and not particularly user friendly for either of the samples.
During the Directory task, seven of the sighted users chose to use the search facility only, ten just followed links and three chose a combination of following links and searching. Five visually impaired users chose to use the search facility, twelve chose to follow links on the page, two followed links and performed a search and one user abandoned the task without following any links or performing any searches.
Observations of the Directory interface revealed that those using a screen reader were able to complete this task more efficiently than some of the sighted users. This seemed to be because, if using a screen reader, options such as "Search", "A-Z" and "Countries" which were offered on the top menu were read out at the start of the task. When users chose to follow one of these options they generally got to the required information fairly directly (although not necessarily quickly as they were required to listen to long lists of countries or alphabetical listings). On the other hand, users who surveyed the page visually were often drawn to the middle screen which had a collection of subject categories presented in slightly larger text than the top menu bar. This route took users on a more complex journey to the information, requiring them to think a little more laterally about which link to follow and certainly taking a less direct route.
As well as offering a search option to follow, the Directory offered a search box directly from the Home Page. This also took users on a fairly direct route to the information but it was presented at the bottom of the page in small text and was not always obvious to the users as it required them to either scroll down to the bottom of the page to locate it, or wait for the screen reader to read out the whole page to them.
For the Online Shopping task, ten of the sighted users chose to use a combination of following links and searching, seven just followed links and three tried to follow links but were forced to abandon the task because of problems with the site. Two of the visually impaired users chose to follow links and to search and three just used the search facility. Fourteen of the visually impaired users followed links on the page (one user was forced to abandon the task before following any links or trying a search due to software compatibility problems).
Users from both samples were able to use the search facility on the Online Shopping site without too much of a problem. The results returned, however, were not always very comprehensive and sometimes completely confusing (at one stage the site was promoting a sale and returned other items as well as items defined by the search). Users (both sighted and visually impaired) at times seemed unsure that the items returned were a true reflection of what was actually available. The visually impaired users tended to stop more readily once they had found at least one relevant item, whereas the sighted users were more inclined to look further. The reason for this may be due to the fact that sighted users could see a lot of other potentially relevant items on screen whereas the visually impaired users were listening item by item and may not have been aware that ten other possible items were displayed. For the visually impaired users it might have been useful to hear how many items were displayed.
All users (sighted and visually impaired) had to perform an initial search for the OPAC task. Both samples had similar problems with this task, mainly relating to the layout of the screen and the terminology used, which was ambiguous at times.
Overall, users who performed a search looked at only the first page of returned results - usually just the first two or three hits - before pursuing one of these links further or reformulating the query. Users who were using magnification generally located the results without too much of a problem as long as the text had not become "pixelated" when enlarged. However those using speech technology (the majority) found that because a screen reader generally starts reading from the top of a page it took a while to reach the results and it was not always obvious if there were any results displayed, or whether the search had even been activated successfully. Users commented that they found the results "difficult to find" and "hard to read", also "not sure if results are there, the top of the page is the same as the last one".
11.1.1 Time spent searching
Overall, visually impaired users took longer to complete each task although the amount of time taken varied between tasks. Completion of the task was counted when the user indicated this to the observer. Users were told that it was up to them to decide when they had found the required information - or when they wanted to stop searching.
The Search Engine and OPAC tasks took visually impaired users between three and five times longer to complete than the sighted users. There were exceptions of course, one visually impaired user completing the Search Engine task in two minutes using the search facility. In this case he/she only had to listen to part of a sentence being read out by the screen reader to be able to understand and move on. (It should also be noted that the user's search terms were very specific and thus quickly retrieved relevant results). The visually impaired users were generally more efficient at completing the Directory and Online Shopping tasks, but it still took them longer to complete than the sighted users.
Further investigation is needed to examine the depth of searching undertaken by both samples in order to ascertain whether users stopped searching once they had found something reasonable or whether they tried other searches or features to check that they had found all the available information. For example, some users (from both samples) indicated they had completed a task even if they had searched no further than a hypertext-linked list of headings and descriptions, whereas others continued to follow one or more of the headings before they indicated they had completed the task. This could be seen as more of a user behaviour issue than a usability one, although for some users (specifically users in the visually impaired sample) it had taken them a considerable amount of time to just retrieve a list of headings. In these cases comments suggest that users may have been more inclined to stop searching at this point because "it just takes so long" and it is "a lot of action for something very simple". Previous studies of user behaviour with search engine tasks (including both visually impaired and sighted users) suggest that on average users search for around 16 minutes before stopping and they often rate success and satisfaction levels with the ease of retrieval rather than information provided in the results (Craven and Griffiths, 2002).
11.1.2 Surveying the page
Comparisons of surveying between the sighted users and visually impaired users (i.e. sighted users who looked at pages and visually impaired users who had text read out) show that there are similarities between the two groups. All users need to spend some time surveying a page or pages before deciding how to proceed, and the layout of the page affected their efficiency in surveying. However, the time spent on surveying differed between the two groups - for example a sighted person often only had to glance at a page, quickly scanning it for relevant information or links. Visually impaired users with screen magnification or who could look at a standard screen at close proximity could also scan pages to some extent, being able to control which part of the screen they surveyed. In contrast, visually impaired users with screen readers had to listen to text read out and this tended to take considerably longer. Although some of the visually impaired users were able to listen to text read out at high speed, listening to just the beginning of a sentence before tabbing to the next block of text, others had to spend much longer surveying before they were able to proceed. This suggests that there are a higher number of variables impacting on visually impaired users' efficiency in surveying a page. Variables include the type of assistive technology used, how compatible this is with the pages surveyed, levels of expertise in using assistive technologies and familiarity with the layout of the screen and terminology used. (The last two variables will also impact on sighted users' efficiency).
During the Search Engine and OPAC tasks the sighted users surveyed pages on fewer occasions than the visually impaired users. Reasons derived from the visually impaired users' comments suggest that all the Search Engine pages contained too much information to enable rapid surveying, and that terminology used on the OPAC site was unfamiliar to many of the visually impaired users, causing them to listen (or look at) pages several times before deciding how to proceed.
In contrast, the visually impaired users surveyed pages on fewer occasions than the sighted sample during the Directory and Online Shopping tasks. Comments from the visually impaired sample suggest the information provided on these sites enabled them to find their way around without having to spend as much time surveying. The Directory, for example, had less information on its home page and although subsequent pages sometimes contained a large number of links, users seemed able to navigate through them quite easily (if not quickly). Similarly although the Online shopping pages contained a lot of distracting graphical information, relevant links were read out in a fairly logical way.
11.1.3 Keystrokes
Keystrokes per task were logged according to serial steps taken (clicking on, clicking back) and parallel steps (including typing in, tabbing down and up, clicking on enter). In most cases visually impaired users took more steps (using keystrokes or mouse clicks) per task, and used a greater combination of keystrokes than the sighted users. Keystrokes logged for each task reveal that visually impaired users used a combination of around sixteen different keystrokes, whereas the sighted users consistently used the same six. Differences between keystrokes used by the visually impaired users compared to the sighted users mainly related to the assistive technologies used, which offered a variety of ways to navigate a page using keystrokes such as the up and down arrow, the tab key, the return key and specific control keys. Users interacting with standard PCs, however, tended not to use control keys, relying instead on scrolling up and down the page, typing in terms, selecting options and clicking on the return key.
11.1.4 Serial and Parallel Steps
Differences between the two samples were found according to steps taken during the four tasks. In total, visually impaired users took more steps (serial and parallel) than the sighted users. The Search Engine and OPAC tasks revealed the greatest difference where visually impaired users took almost twice as many steps overall as the sighted users. The Online Shopping task shows that visually impaired users took more parallel steps than the sighted users, but took fewer serial steps (i.e. movement within pages), indicating that the visually impaired users could locate a relevant page quite efficiently but had to spend more time navigating within a page in order to find the information.
The only task where visually impaired users took fewer steps overall (both serial and parallel) was the Directory task, indicating that for this task visually impaired users were able to locate the required information with a more efficient use of steps than the sighted users. As mentioned previously, a possible reason for this, derived from observations and comments, was because those using a screen reader (the majority of visually impaired users in the sample) had potentially relevant links such as "search", "A-Z" and "countries" read out to them from the top menu almost as soon as they started the task.
The findings show that whilst movement serially from page to page (or between web sites) varies to some extent between the sighted and visually impaired users, the main differences occur when moving around, or within a page (parallel). This suggests that the more choices provided on a page and the more complex the design of the interface, the longer a visually impaired person must spend trying out different options before finding the information they want or moving on to use another resource. This was particularly true for users who were listening to screen readers, as they were to some extent forced to navigate the page serially when it had actually been designed for parallel navigation (e.g. a page containing frames).
11.1.5 Search Terms
Similarities occurred between the two samples in the type of search terms used and in the type of search. Almost all of the users who searched applied simple search terms and did not make use of the advanced function. These findings are in line with previous research into user searching behaviour (Craven and Griffiths, 2002), indicating that in general all users whether they are blind, visually impaired or sighted want to be able to perform quick and simple searches.
Differences did occur in the reformulation of searches and search terms. Both samples reformulated searches to include new terms or to add to existing ones. However, the visually impaired users also reformulated their search because they had mis-spelled terms, which meant no results were retrieved. Sighted users also mis-spelled terms, but they could generally spot their mistake and correct it fairly quickly, whereas this was often chance or guesswork for the visually impaired users. Another reason that visually impaired users reformulated a search was when the results were displayed half way down the screen and the screen reader starting reading out from the top. Users assumed the results would be read out first, so were unsure if any results had been retrieved or whether their search had been activated and therefore re-submitted the search.
11.2 General questions
General questions included aspects of using the web and the chosen resources. Users from both samples mentioned using various methods to tell them that a page was loading or that it had loaded. The main methods mentioned by the sighted users were visual ones such as looking at the page loading on-screen or at the various clues provided on screen such as the browser icon moving around, the status bar or messages such as "Document Done". For visually impaired users these types of clue were only useful if the user had enough sight to be able to decipher icons or messages, or if they could use screen magnification. For those using a screen reader, the user had to rely on aural clues such as listening to noises from the hard drive, or waiting for the screen reader to start reading out from the top (which most automatically do once the page has loaded), or for "Document Done" to be read out. Most used a combination of clues rather than relying on one, and a degree of patience and chance seemed to be part of the process, suggesting that even before they have started to interact with a web page a number of barriers to their success already exist.
Before commencing a task, users were asked to comment on the interface of each resource, describing their first impression of the interface and what sort of information they thought it was providing. Generally, both samples commented favourably if the information provided on screen was presented in a clear and logical way (whether to look at in standard size, magnified or read out). One of the visually impaired users commented that when listening to a page read out it appeared to him/her as one long list, and therefore it was more helpful if the list followed a logical order and was not too overbearing in terms of the number of links per page. Clarity of background and text was mentioned by users from both samples but had more of a bearing on visually impaired users, especially those who were using screen magnification as the text from some sites became "pixelated" when enlarged and therefore difficult, if not impossible, to read ("smoothing" of text could help improve this to some extent, but only if the user was aware of this function).
Users from both samples were less happy with pages that provided them with too much information. The sighted users commented that pages with a large number of links, graphics and repeated menus sometimes appeared "cluttered" or "busy" to them, which was annoying. For visually impaired users such pages were time consuming and overbearing and could discourage the user from proceeding. If a page contained duplicate menus and/or repeated links it was not always clear to them if they would take the user to the same place (as well as having to listen to the link read out twice if using a screen reader). Interestingly, pages that were full of graphical links were not necessarily a problem for the visually impaired users as long as appropriate alternative text was provided clearly describing where the link would take them.
Pages that included logos were a problem if the logo was not properly explained, and pages that had ambiguous descriptions to links presented potential problems to users (from both samples). Similarly, large block of text were not favoured by either the visually impaired users or the sighted users and most said that they would not bother to read this type of information.
Users were required to use a stated resource for each task, the aim being to ensure valid and reliable comparisons between the users and the tasks. At the end of each task users were asked what resource they would have chosen to search for the information. For each task, a few users said that they would have chosen to use the stated resource. However the majority suggested other resources such as bookmarked or favourite sites, other search engines, or even non-web-based resources. The visually impaired users tended to have their own bookmarked or favourite sites. Generally the sites they mentioned were ones that they found accessible, which was an important factor for them. Of other search engines mentioned, the most popular with visually impaired users were Google and AlltheWeb. For the visually impaired users who said they would not have used the web at all for one or more of the tasks, alternatives mentioned were CD-Roms, ringing up the library, asking someone, listening to the radio, or (in the case of the Online Shopping task) just going to the shop.
11.3 Usability questions
Users were asked to describe their experience of undertaking each of the tasks, including ease of navigation, executing searches and following links, and overall satisfaction levels.
11.3.1 Navigating around the resource
Visually impaired users found navigation of the Search Engine more of a problem than the sighted users. In general the visually impaired users found it to be "unpleasant" and with "too much information", whilst the sighted users found it quite easy and straightforward to navigate (even if they did not like it). Overall, both samples seemed to either like it or loathe it, with only five of the users overall expressing neutral feelings relating to navigation of this site.
Navigation of the Online Shopping site was less problematic for visually impaired users than the Search Engine, with just under a half expressing positive feelings about navigation. However, the site still had its frustrations, the main problems being the amount of duplicated information, the rather inflexible search facility and presentation of the "results" (which proved confusing both for the visually impaired and sighted users).
Sighted users found navigating the OPAC site easier than the visually impaired users, describing it as "simple" and "intuitive" whereas comments from the visually impaired users described it as "quite confusing" and spoke of having to "use a lot of guesswork". One reason for this was that many of the sighted users were familiar with the layout of the site, whereas visually impaired users were less familiar, although some commented that it was an interface they could get used to if necessary.
The most popular site in terms of navigation was the Directory. Around three quarters of the users (from both samples) expressed either positive or neutral feelings about navigation of this site. The main problems expressed related to ambiguous or unclear descriptions (comments from both samples) and pixelated text when using screen magnification.
11.3.2 Locating hypertext links
In general, most of the sighted users had little or no problems locating hypertext links, which were in a "different colour", or "underlined", or when the cursor "turned into a hand", or they just assumed it was a link. In general the visually impaired users also had few problems locating the hypertext links (even if there were too many), because they were read out by the screen reader as a "link" (or could be seen in similar ways to the sighted users).
Some problems locating hypertext links did however arise. During the Search Engine task, location of links was described as confusing if plain text was presented amongst a long list of hypertext links, because to the user it seemed logical that this was also a link. This was particularly problematic if using a screen reader, as it was not always easy to decipher which bit of text the "link" prompt was referring to.
Problems with links were also experienced by the visually impaired users during the Online Shopping task, where some of the links were graphical ones and not obvious, while others were too close together, not separated by anything, or "all higgledy-piggledy".
In general, both samples found links on the OPAC and Directory sites easy to locate. The main problems experienced with both of these sites (again, for both samples) was that it was not always obvious what the terminology meant or where the link would take them, examples being what the A-Z link was referring to or what a link to "Reference Details" meant.
11.3.3 Locating the search box
If a search box was provided on the screen most of the sighted users were able to locate it, particularly if it was placed in a prominent place. Similarly, the visually impaired users were able to quickly locate search boxes if they were prominently placed (such as near the top of the page). Visually impaired users also preferred to be provided with a search box rather than a link to a separate search page, as this was just one more step for them to take.
During the Search Engine task, around half of the visually impaired users and almost all of the sighted users found it reasonably easy to locate the search box. However once located it was not always obvious to some of the visually impaired users where they had to enter search terms, or how to activate the search. This was a particular problem when using a screen reader and some users thought they had entered terms and activated the search, but in fact had not. This confusion continued when no results were returned, as it was unclear to the user whether there were no resources available or whether they had not properly activated the search.
The OPAC home page provided links to various search options such as Title Search, Keyword Search or Title/Author Search. On selecting a search link, users were taken to a separate search page. Once in the search page the search box could be clearly seen together with some extra options and an example of how to enter terms. The cursor was automatically set in the search box ready to type in terms. This was very useful for users who were able to look at the screen (standard size or magnified), but was sometimes confusing for users with screen readers as they were not always aware that the cursor was set ready for typing in terms. Although some of the visually impaired users were able to use a keystroke to place them directly into the search box, not all users seemed be aware of this option and in some cases "just guessed" how and where to enter terms.
The Online Shopping site provided a separate search page with simple search box, plus several drop-down combination boxes. This did not prove a particularly successful experience for either of the samples, for although locating the search box or combination boxes was not too problematic, once found it was not always obvious what to do next or how to enter terms. A help screen was provided, but only a few users activated this.
The Directory site provided a link to a separate search screen from the home page and also provided a small search box at the bottom of the home page. Not all the users (from both samples) were aware a search facility was provided, either because the search option was hidden amongst the other links, or because users had to scroll down the page to locate it or have to "go all the way down reading everything to find it".
11.3.4 Understanding the results
Once results from the Search Engine had been returned users from both samples found them reasonably easy to understand. However, the main problem for the visually impaired users was actually locating the results, as they were presented part of the way down the screen and, if using a screen reader, they would have to listen to the top part of the screen read out first. When this happened it was not always obvious whether they had retrieved any results at all. Another problem experienced was that if the search returned a large number of results it was not initially clear to the visually impaired user that only the first ten were displayed, the worry being that they may have to listen to hundred of results being read to them. Although in reality most users rarely went past the first three or four results, it was still off-putting to be faced with a large number of results.
Overall, visually impaired users seemed to find the Online Shopping results easier to understand than the sighted users. This was mainly the case for users with screen readers: reasons for this may have been that they were listening to the results read out to them item by item rather than looking at items on what was a rather cluttered and distracting page.
The visually impaired and sighted users generally found the Directory results easy to understand, although some users (from both samples) said that they only looked at or listened to the headings rather than the whole description, or just clicked on a link without thinking about it. Problems experienced with the Directory results were pixelated text (when using screen magnification) which was hard to read, and the amount of detail given which was "a bit too much at one stage".
As mentioned previously, many of the sighted users were familiar with the terminology used on this site and therefore nearly all the sighted users said they were able to understand the results. The visually impaired users expressed mixed feelings about the OPAC results, but almost half of them also said they could easily understand the results. The remaining visually impaired users (and a few of the sighted users) either found the results confusing or found them hard to read (i.e. some of the text appeared pixelated when magnified). An example of confusion was when the results returned informed the user that they had one result, but proceeded to read out a long list of titles (including the title searched for). Some users said they found the terminology confusing because they were unused to it but given time would be able to use the site more efficiently.
11.3.5 Getting lost
Many of the users experienced times when they were either not sure where they were during a task or were unsure how to continue. This did not necessarily cause users to give up on a task (unless it kept happening or if they could not get back on track after several attempts), the main strategy being to "go back" a stage or even to "start again". This strategy was used by both the sighted and visually impaired users.
During the Search Engine task visually impaired users had more problems telling where they were than the sighted users. Over half said that they had moments during the task when they felt partially or totally lost, compared to just a quarter of the sighted users.
Sighted users also seemed slightly better able to tell where they were during the Online Shopping task (although the only two users to feel completely lost were sighted). The visually impaired users expressed mixed feelings about this task, one commenting that he/she was "not lost, just didn't know where I was going" and another "more lost than any of the other (tasks)".
During the OPAC task most of the users from both samples knew where they were and did not express feelings of getting lost. However some were unsure how to find the information required and were not always sure how to proceed. Terminology rather than getting lost seemed to be the main stumbling block during this task which, and as one visually impaired user commented, "needed a bit of ingenuity" to complete.
The Directory was probably the most successful site in terms of users knowing where they were during the task. More sighted users expressed feelings of getting lost or feeling confused than the visually impaired users, of whom only around a quarter expressed any feelings of getting lost. The remaining visually impaired users said it was clear where they were up to and they simply back-tracked if they were unsure at any time.
11.3.6 Error messages
Any error messages that appeared during a task were generally ignored and backed out of or closed (by both samples). Pop-up windows were also often ignored and closed, although one of the visually impaired users commented that these can be good for information, even if they are sometimes hard to close.
11.3.7 Levels of satisfaction
Satisfaction levels relating to the Search Engine task were mixed. Sighted users generally seemed happier with this task than the visually impaired users. The visually impaired users tended to respond in either a neutral or negative way to the task, expressing feelings of frustration about the length of time it took them to find "something very simple". This feeling is reflected in the number of users who cited search engines with simpler interfaces such as Google or Alltheweb as their preferred method of searching.
Satisfaction of the Online Shopping task was again mixed, although the visually impaired users were generally more positive than the sighted users. Sighted users tended to express more neutral comments relating to satisfaction with this task, and both samples expressed feelings of dissatisfaction. A general feeling seemed to be that even if they were satisfied with the task it was not necessarily a good experience, or as one of visually impaired users who was satisfied with the task commented: "I won't be coming back to the site". However, perhaps this is more a reflection of attitudes to online shopping in general than of the site itself.
Sighted users were more satisfied with the OPAC task, although both samples were generally happier with this task compared to the Search Engine and some responses given were extremely positive. The majority of neutral or negative responses came from the visually impaired users and this seemed to relate more to a lack of familiarity with the site and the terminology used, which seemed to hamper users' success with this task.
The Directory task was again the most successful in terms of satisfaction levels with nearly three quarters of the visually impaired users giving positive responses (and over half of the sighted users). Neutral responses generally related to the fact that, although the required information was retrieved, it was "a bit hit and miss" or that the user was "guessing all the way through". Only two negative responses were made relating to this task, the main problem being that some of the information and links provided were in very long lists which meant that it could take a visually impaired user a lot longer to find the information.
[ Previous Section: Usability Questions ] [ Next Section: Conclusions ]