This site's design is only visible in a graphical browser that supports web standards, but its content is accessible to any browser or Internet device.

mmu | cerlim

VITAL > Final Report: Vital Methodology
or use the sitemap
CONTENTS

4. The Vital Methodology

4.1 Introduction

As stated in Chapter 1, one of the key aims of the VITAL project was to develop a set of methodologies to enable evaluation of the impact of ICTs in public libraries, and the assessment of their value to end users. The methods were designed to be straightforward to implement for public librarians who may not have extensive experience of carrying out research, yet robust enough to provide useful data which could, with careful interpretation, be compared between different library authorities. The three library authorities chosen to pilot the methods which were developed reflect the different circumstances of authorities across the UK: Birmingham, an inner-city authority with a multi-ethnic population; Cheshire, a more rural authority, with some large towns and a population reflecting a diversity of economic circumstances; and Cumbria, a large, rural county authority with some very small libraries serving a population dispersed over wide geographical distances.

These three library authorities piloted the methods developed in the initial stage of the project, and the results from those studies fed into the final methods, collected together in the Workbook which is the companion to this report. This section, therefore, does not contain detailed descriptions of the final methods, but reports on the process of the project and the testing of the methodologies by the library authorities. The results obtained from those test implementations are reported in Chapter 5.

4.2 Preparatory stages

During the first phase of the project the team examined methodologies being used elsewhere to obtain information about library value and impact. As noted earlier, most of the data collected systematically has been quantitative in nature, although there is a more recent trend towards the inclusion of measures of user satisfaction, which - while expressed quantitatively - represent a qualitative dimension. The Best Value process, for example, places a duty on local authorities to consult with their clients and to take into account their views on service performance.

It is also noticeable that in assessing the value of library services there has been something of a trend towards the recording of 'rich pictures' of use. These are individual accounts of the value and impact of interaction between a user (or sometimes group of users) and a library service. They are indicative and must be used carefully, for no one interaction can tell the whole story of a library's service performance. Nevertheless, when used as part of the story, and when their evidence is triangulated with other evidence, they form a valuable input. An early example of their use can be found in the report, Open &amp Distance Learning in Public Libraries (Brophy et al. 1996).

The conclusion reached in this stage of the work was that the methodologies should enable the collection of both quantitative and qualitative data, and that among the latter there should be encouragement to collect statements (whether positive or negative) from library users and non-users, in which they express their views, preferably in their own words. It has to be recognised that impacts are personal, especially when they concern learning which is itself an intensely personal experience. By collecting a representative range of such opinions and looking for common threads, the researchers aimed to build up an overall picture of user perspectives on value and impact. To set this evidence in context, each library would provide a background profile document describing its services, the availability of end-user ICT services, etc. The resulting portfolio of data would enable evidence to be cross-checked and conclusions drawn.

On the basis of these observations, the team developed a detailed evaluation plan and devised a first draft of evaluation instruments. Again, readers are referred to the Workbook for details of these.

4.3 Testing the methodologies

Each library authority seconded an existing member of staff to work half-time on the project for a period of six months, beginning in July 1999. The members of staff had a variety of previous research experience and varying levels of responsibility within their library, reflecting the different contexts within which the methodology proposed would be utilised. At the beginning of the test phase, the researchers were invited to an initial training day, at which they were given the initial draft 'workbook' of methods, consisting of some introductory material about conducting research, a proposed questionnaire for library users and another for library non-users, and an interview schedule. Discussion throughout the day gave the researchers and the VITAL team the opportunity to refine the questions and to raise any particular issues relating to the implementation of the methods. The questionnaires and interview schedule were further circulated within the project's Advisory Committee before being finalised and printed for distribution to the three authorities. As the researchers had varying experience of carrying out this type of project, the workbook was designed to provide a directed approach to the research. Guidance on how to conduct interviews, for example, was provided, as well as specific examples of quantitative data to be collected (for example, number of PC workstations available, number of PC hours booked per week) for inclusion in the background context to be provided. Again, this approach was designed to fit the reality of library staff across the country attempting to apply the VITAL methodologies at a future date.

Throughout this phase, the researchers were provided with ongoing support in the form of additional appendices to the workbook (for example, guidelines on data entry of questionnaire results), site visits, and regular email and telephone contact.

The methods developed for the test implementations consisted of the following:

4.3.1 Library users questionnaire

After the standard questions were agreed upon, some additions or slight alterations were included for each authority, dependent on the particular context. For example, it was agreed that questions relating to library OPACs would not form part of the standard questionnaire (since OPACs were not universally available), but one authority inserted a section relating to its newly-introduced computerised OPAC as it seemed an ideal opportunity for this evaluation to be undertaken. These variations were regarded as an essential part of the methodology: the data must primarily focus on the local situation rather than forcing services into an artificial, and therefore sometimes misleading, standard framework.

Five hundred questionnaires were distributed to each library authority, which then made its own decisions about the number and location of library sites to include in the research. This ranged from four sites in Birmingham and Cheshire to six sites in Cumbria, with each site receiving a number of questionnaires proportionate to its size and opening hours. The distribution methodology adopted was very similar to that recommended by the existing CIPFA PLUS surveys. [CIPFA (Chartered Institute of Public Finance and Accountancy) has developed the PLUS (Public Library Users Scheme), which distributes a standard questionnaire and methodology to member library authorities (67% of all authorities were members in 1998) to assess user levels of satisfaction with library services.] It was felt that this approach would be familiar to the libraries involved and to future users of the methodology - and has already proved to be successful. However, as this was a pilot, aimed at testing the validity of the methodology and the effectiveness of the questions, it was felt that 500 questionnaires would be a sufficient sample, and the library authorities were not required to issue the numbers usually required using CIPFA calculations (based on the number of library visits to a given site per week). Consistent with CIPFA implementation guidelines, authorities were encouraged to number the questionnaires, and distribute them within a range of time slots during the week which would facilitate as diverse a representation of users as possible.

4.3.2 Non-users questionnaire

Again, a set of standard questions was developed for surveying non-users within each authority. However, the methods selected for administering the survey varied between the authorities according to local circumstance and resources available. Again, this was a deliberate decision taken with the aim of securing replicability in a wider-scale roll-out. Each authority was encouraged to aim for a target of 300 non-user responses; this proved to be very challenging within the timescale of the testing phase (6 months). Cumbria was, in fact, able to undertake the survey on this scale, due to assistance from their Information and Intelligence Team within the County Council. A telephone survey, using the VITAL questions, and drawing on an existing database of known library non-users was undertaken on behalf of the library. For Birmingham and Cheshire, who did not have access to these resources, reaching the target number proved much more difficult. One of the recommended approaches from the VITAL team was the "street survey" approach, which was used in Cheshire with some success in the larger towns, such as Chester, but proved to be very time consuming in smaller areas. Thus, Cheshire finally used a combination of this approach and a method which involved targeting employees of the County Council via email and inviting them to complete the survey by telephone. Birmingham felt, for various reasons, including staff safety, that the "street survey" would be inappropriate, and opted instead for a targeted method of contacting groups who are currently under-represented amongst library users. A list of community groups catering for a range of people was drawn up and groups contacted and visited. The researcher was then able to conduct the questionnaires and engage in some profile-raising and promotion of library services. It was noted, however, that this method would not provide a sample which was necessarily representative of the population as a whole.

This part of the research proved to be the most difficult for those authorities without access to the type of database resources and assistance available in Cumbria (where, again, it was felt that the street survey approach would be inappropriate, due to the size of towns, and the vast distances involved in surveying across the County). It was felt, however, that this was an important part of the research, as it not only provided the opportunity to contact non-users and promote library services to them, but also to ascertain whether the introduction of ICTs would encourage some non-users to begin taking advantage of library services. Developments during the course of the VITAL project, such as the introduction of Best Value, indicate that this type of community consultation will become increasingly important. Furthermore, by linking VITAL-type questions into other consultative exercises, some of the difficulties experienced by the researchers would be removed.

4.3.3 Interviews

One of the aims of the VITAL project was to encourage the collection of qualitative data in library authorities, in order to supplement quantitative figures about ICT usage and non-usage with detailed 'rich pictures' of individual use, able to demonstrate more clearly the value of such services as illustrated by the voices of the users themselves. It was thus considered very important that a series of semi-structured interviews be carried out with existing ICT users across the three authorities. The use of an interview schedule assured consistency of approach, yet allowed sufficient flexibility for individual researchers and interviewees to explore areas most relevant to themselves. A total of 107 interviews were carried out with ICT users, varying in length from 15 to 40 minutes. The interviews were taped in Birmingham and Cheshire, but not in Cumbria, where notes were taken which were later typed up. Although not specifically required as part of the VITAL methods, Birmingham and Cheshire also organised some focus groups, in order to include the views of younger users, who, it was felt were heavy users of the ICT facilities, but who could not be included appropriately in the interviews.

It was recommended that, where taped, the interviews should be fully transcribed before analysis. Due to time constraints, both the transcription and the full analysis proved to be difficult to complete, so a method was devised to speed up the process but allow for meaningful analysis to take place. Data relating to each question was grouped together so that it could be considered across the range of interviews, and any themes identified. Given more time, each library authority would have liked to have developed their analysis further, as the initial results proved to be highly informative and useful when illustrating the value of ICT services to those outside the library service. This kind of data collection does require considerable time and resources but is a highly effective way of illustrating the impact access to such services can have, for individuals and communities, beyond the 'dryness' of usage figures.

4.3.4 Conclusions

The results from the test implementations of the methodology proved to be highly instructive; overall, the methods selected were appropriate and yielded useful data. On the basis of this learning changes were made to the questionnaires and incorporated into the final Workbook. Where difficulties were experienced, this was largely due to time or resource limitations; conducting research of this nature, especially incorporating the commitment to qualitative data collection and analysis, will require both time and staff resources, but can prove to be very rewarding and highly instructive. It also has to be recognised that few library authorities can dedicate staff to this work, so that staff training for those for whom this will be just one part of their duties also becomes a significant issue. Nevertheless, it will become increasingly necessary to undertake such exercises if the value of the People's Network is to be demonstrated as it rolls out and develops over the next few years.

[ Previous Section - Value & Impact ] [ Next Section - Results ]