5. METHODOLOGY: NOVA EXPERIMENTAL FRAMEWORK
5.1 NoVA Project Aims
The formal aims of the NoVA project were, in the context of access to the digital library by blind and visually impaired people:
· To develop an experimental framework for exploration of serial
searching and retrieval in non-serial environments
· To undertake a series of experiments with serial searching and retrieval,
and subsequent use of digital content
· To map serial/non-serial approaches so as to develop understanding of
how serial searching, retrieval etc. can be optimised in non-serial environments
· To report on findings and to make recommendations for digital library
system design.
5.2 NoVA Experimental Framework
The NoVA experimental framework included, as indicated in Chapter 4, four different electronic resources selected for the usability testing phase. These were a search engine, a library online catalogue, an online directory of Internet resources and a commercial online shopping site. Resources were chosen to reflect the aims of the NoVA project: "to map serial/non-serial approaches so as to develop understanding of how serial searching, retrieval etc. can be optimised in non-serial environments". Therefore each resource chosen had, to greater and lesser degrees, elements of parallelism in their interfaces. At the same time it was important that a set of different types of site was chosen, reflective of the range of sites likely to be encountered in real-world environments. It will be noted that in selecting these resources a broad view of the digital library was taken.
Four information seeking tasks were devised and piloted using each of the selected resources. The tasks were devised so that, where possible, users could choose whether to use a search facility or to follow links from the home page. For example, users were asked to look for the UK weather forecast using a specific Search Engine. It was possible to find this by either typing in search terms and exploring relevant hits or by following a "weather forecasts" link on the Search Engine's Home page, which led to a summary of the weather forecast. Again, the aim was to reflect as far as possible the real-world choices which users are offered.
5.3 The Sample
In order to obtain specific data on the way people search electronic sources it was necessary to select a sample who were reasonably familiar with using the Internet and were comfortable using assistive technology. The sample was therefore a purposive one, i.e. handpicked for the research and included users of similar IT ability. The minimum requirements for the sample were that:
· Users could understand the basic concepts of the Internet and the
Web.
· Users could interact with a screen unaided using a keyboard, mouse or
assistive technology.
· Users had used at least one search engine to retrieve information.
The sample size comprised 40 users, split equally between sighted (the control group) and visually impaired users. Although official definitions of "sighted" and "visually impaired" were taken into account, users for the NoVA project were grouped according to their ability to read a standard (14" or 15") sized computer screen. Therefore, the "sighted" sample were users who could comfortably read a standard sized screen and the "visually impaired" sample were users who needed to use assistive technology or to be very close up to a standard sized screen in order to read or interpret it.
5.4 Tasks
Tasks were consistently set so that comparative analysis could take place between the sighted and visually impaired users. It was recognised that success in performing searches could be influenced by previous knowledge or experience, either of the technology or the subject matter of the task, or by individual interpretation and approach to a task. In an attempt to obtain a balanced picture the tasks set covered a fairly broad subject base such as weather forecasts, shopping for clothes, travel information etc., and using a "simulated work task situation" (Borlund and Ingwersen 1997 pp. 225-246) to try and dispel feelings among the users that they were being tested or timed in any way (although inevitably this still occurred to some extent). The tasks were:
· Search Engine: look for a current weather forecast for the United
Kingdom and then the most recent weather forecast for Manchester.
· Directory: look for any resources relating to the Solomon Islands.
· OPAC: find details of the British Journal of Visual Impairment and then
find out who published it and what its former title was.
· Online Shopping site: look for a selection of men's suits and then find
any priced between £100 and £200.
The users were allowed to choose how they undertook each task, although to ensure everyone started from the same place they were required to commence using the stated electronic resource. So for example, to look for the national weather forecast for the UK, users were not allowed to simply type the Met Office URL into the location box but had to start from the Search Engine home page. Information about the weather for the Search Engine task could be retrieved either from within the Search Engine site itself or from external sites retrieved via the search function.
Users were allowed to choose whether they used the search facility provided by the resource or to simply browse through the stated resource's site for relevant links. For example, the Search Engine, Directory and the Online Shopping site provided links from the Home Page to information required for the task, as well as offering a search option. The OPAC was the only resource used where it was necessary to initially type something into the search box, after which it was possible to browse through the site clicking on relevant links.
Users were not given a time limit to complete each task. At the beginning of the session they were told that they could stop the task at any time and were given examples such as "if you are satisfied that you have completed the task", "if you are not satisfied, but think you have found all the information there is", "if you are fed up with the task". Users were asked to indicate to the observer when they wished to stop the task. This was done to help ensure completion was determined by the user rather than the observer. Only on a very few occasions did the observer prompt a tactful termination of the task. This was done, as suggested by Tamler, when "the user is either at a loss or has reached the point of diminishing returns" (Tamler 1998). On these few occasions the observer generally said something like "do you want to stop, or are you happy to carry on?" to reassure the user that it was still up to them whether to continue or not. Once the user indicated that they wished to stop, the task was counted as "completed" whether or not they had found all, some or none of the information required. The reason for this was to try and simulate real life or "work task" information searching behaviour, where information required by a user may or may not be found from within a specific resource. "Completion" thus refers to the task and is not a judgement of the amount of information retrieved.
5.5 Pre- and Post-Task Interviews
The pre- and post-task interviews consisted of a set of semi-structured questions based on Ravden and Johnson's method of evaluating usability of human computer interfaces (Ravden and Johnson 1989 pp.27-44). Questions were set in accordance with the "person centred" approach recommended by Duckett and Pratt (2001), where the participant's role is one of a collaborator rather than a subject to be investigated and the researcher adopts a non-judgemental attitude towards each participant's experiences, opinions and feelings (Duckett and Pratt 2001 p.819).
Interviews began with a set of user profile questions to help draw up a profile of user characteristics. Questions included details such as age range, use of the Internet, location of access and a brief description of their visual impairment (if applicable).
The nature of a user's visual impairment has only been gathered to provide a clearer picture of the range of users involved in the study rather than an indication of the problems faced by different impairments. Likewise, age and experience have been used as an indication of demographic spread rather than to analyse behaviour according to a person's age or ability. The aim of this is to adhere to a social model of disability, which focuses on the "social, economic and political barriers that 'disable' people who have impairments" (Duckett and Pratt 2001 p.816) in contrast with the medical model, which focuses on the disability itself and how it can be overcome to fit into society.
The semi-structured interviews were also conducted to provide data on emotion, feelings and experience. These comprised general questions, such as how to tell a page is loading, initial comments about the interfaces and the type of information provided; and usability questions, such as their experience in finding resources required, correcting errors, knowing where to input information, online help facility. Interviews were conducted before (general) and after (usability) each task to help ensure the electronic resource and the task performed were still fresh in the user's mind before moving on to the next resource.
5.6 Recording tasks
Data from the information searching tasks was logged using a combination of Lotus ScreenCam (http://www.lotus.com/products/screencam.nsf), and sound recording and note taking. ScreenCam records on-screen activity plus verbal dialog, and thus "provides a more qualitative observational approach to transaction logging " as it "captures everything that the user sees and interacts with on the screen" (Griffiths et al. 2002). For the visually impaired sample, it was found that ScreenCam can interfere with the assistive technology they were using, and therefore observation, sound recording and note taking replaced the on-screen recording. All other elements of task recording remained.
The observer attempted not to prompt subjects or give any instructions while the subject was performing the task. This proved difficult at times, particularly when it was evident that the user was becoming distressed, as it was felt that undue stress was not a helpful way forward. Therefore in some cases the observer provided a "hint" to enable the user to continue. Nielsen suggests that this type of intervention is necessary in certain circumstances, as is prompting a user to ensure the transcription is accurate (Nielsen 1993 p.197).
Both methods used for logging data were tested to ensure valid and reliable data could be transcribed. As well as logging the information retrieval process, pre and post-task questions were asked in order to gather data of a more qualitative nature so that as well as logging what the users did, it was also possible to ascertain (to some degree) why they did it and how they felt about it.
5.7 Data Analysis
Analysis of data logged from the searches and the pre- and post-task interviews was entered into the Atlas.ti software analysis tool (http://www.atlasti.de/). This method of analysis was chosen because it enabled large amounts of data to be analysed qualitatively and to some extent quantitatively.
Data analysis for the NoVA project concentrated on the text retrieval and code and retrieve functions available in Atlas-ti. The "text retrieval" function was used to analyse the pre- and post-task interviews. Data from the interviews was coded according to instances of words and phrases, using general headings such as "GEN :Doc Done" or "GEN:Status Bar" for responses from users to the question "how do you know the page is loading or has finished loading". As well as logging the number of responses given it was also possible to log comments such as:
It says document done - assume that it has finished, although it doesn't always mean that (sighted user).
I look for the egg timer (visually impaired user).
Using the "code and retrieve" function, each step of the NoVA search process was coded to describe a particular action undertaken by the user. Movements from page to page (serial step) such as clicking on a link, or clicking on the back button were coded as CO (Click On) and BACK. Movements within the screen (parallel steps) included actions such as typing in search terms (coded as TI), or scrolling up and down the screen (coded as CU and CD). Each of the assigned codes is explained further under the appropriate heading of this report and a short description of each is given in Appendix One.
[ Previous Section: Digital Libraries ] [ Next Section: User Profile Characteristics ]