EFX: Evaluation support for FAIR and X4L Projects
| EFX Project Home
| Toolkit Home | FAQ |
General Links | Contact
|
Evaluation Methods
Broadly speaking, evaluations may use either quantitative or
qualitative methods, or a mix of the two. As a starting point it is useful
to consider what sources of data are, or might be, available to the evaluation.
Bear in mind that you may want to use at least three different methods in order
to 'triangulate' your findings - in other words, to see how one set of data
supports (or not) another. This page provides links to descriptions of methods
and various tools.
- Web server logs. Before starting to use these you might like to
read "Why
Web Statistics Are (Worse Than) Useless" (now somewhat dated, but
worth noting) and
"The
Value of Web Statistics". Useful resources include the
Analog site, which has nice demonstrations
of how data can be presented as well as sample reports.
- Logs of user sessions. These may be used to track how users
navigates through a web site and what results they receive. Various packages
are available to do this, such as
ScreenCam.
- Assessment of user satisfaction. One of the most used methods of
gathering evaluative data, particularly where the focus is on outcomes. User
satisfaction may be gauged by surveys, using questionnaires
or
interviews or by focus
groups. It may also be inferred from exception reports, though these are
not systematic enough to be a major method in an evaluation.
- Assessment of user experiences. The
SERVQUAL methodology is
designed to assess the gaps between users' expectations of a service and their
actual experience. An interesting implementation involving library services is
underway in the USA under the acronym,
LibQUAL.
- Observation. This may range from non-participant
observation, where the observer is outside the group being observed
and does not come into direct contact with it, through to participant
observation where the observer is part of the group. Unobtrusive
testing of a service may be important to ensure that the evaluation does
not affect behaviour, but it is always difficult to judge why users
are behaving as they are (rather than just what they are doing).
- Case studies. It is often useful to provide an in-depth description
of a project's outputs and outcomes, and this may be achieved by describing a
particular case in depth. There will always be a particular viewpoint
expressed in a case study, and this should be explicit. Note that a case study
can be chosen either to be representative of outcomes or to highlight good or
poor experiences.
- Structured walk-throughs. These are used to check whether a product
conforms to a specification. In essence the product (often computer software)
is checked for conformance in exact detail through every stage of its intended
functionality.
- Peer Review. Widely used in higher education, though not as often
in formal evaluation of projects, it uses acknowledged experts to provide a
view on achievements and performance.
Some useful resources
- Cornell University's Research Methods Knowledge Base contains a section on
Interviews
- The Qualitative Methods Workbook by C. George Boeree has a section on
Nondirective
Interviewing
