DEVISE A framework for the evaluation of Internet search engines
Library and Information Commission Research Report 100
by F C Johnson, J R Griffiths and R J Hartley
Abstract
This project investigates the feasibility of the use of user-satisfaction as a multidimensional evaluative construct of search engines. Search engine developments are reviewed to reveal a range of indexing and retrieval techniques that may assist casual users in the information retrieval task. Yet few evaluation studies have considered the impact of system features, in particular those with which the user interacts for search assistance. A broad review of retrieval system evaluation highlights the complex environment in which measures of both the utility of the search results and the usability of the system are sought from a user-perspective. Our proposed approach for a user-centered evaluation is based on a conceptual framework in which user-satisfaction is characterised as a variable dependent on system features and functions and expressed in a moderating context of user-task requirement. Towards this end, the research reported here focuses on the definition of the construct of user satisfaction on the multi dimensions of the retrieval process, an expression of what a typical user is trying to do. Empirical work was then undertaken to test the feasibility and potential value of the implementation of the framework for the evaluation of three search engines. Initial results are presented which provide a degree of understanding of how users are satisfied and on what criteria. This provides the basis on which we make recommendations for the refinement of the multidimensional framework and its use as a methodology for the evaluation of search engines from a user perspective.
For further information or correspondence please contact the Project Director or the Project Research Fellow at the address below:
Dr Frances Johnson
Project Director
Jillian Griffiths
Research Fellow
Centre for Research in Library & Information Management (CERLIM)
Department of Information & Communications
The Manchester Metropolitan University
Geoffrey Manton Building
Manchester
M15 6LL
Tel: +161-247-6156
Fax: +161-247-6351
Email: f.Johnson@mmu.ac.uk, j.r.griffiths@mmu.ac.uk
©Resource: The Council for Museums, Archives and Libraries 2001
The opinions expressed in this report are those of the authors and not necessarily those of Resource: The Council for Museums, Archives and Libraries
Library and Information Commission Research Report 100
RE/085
ISBN 1 902394 61 5
ISSN 1466-2949
Library and Information Commission Research Reports are published by Resource: The Council for Museums, Archives and Libraries and may be purchased as photocopies or microfiche from the British Thesis Service, British Library Document Supply Centre, Boston Spa, Wetherby, West Yorkshire LS23 7BQ, UK.