EFX: Evaluation support for FAIR and X4L Projects

Toolkit Navigation

- Toolkit Introduction
- Project Logic
- The Purpose of Evaluation
- Evaluation Goals and Questions
- Indicators of Achievement
- Evaluation Methods
- Analysis of Data
- Presentation of Results
- Timing Evaluation Activity
- Administrative Considerations
- Ethical Considerations in Evaluation
- General Links

The Purpose of Evaluation

There are many different ways of categorising evaluations. Evaluation findings may be designed (and used) to make a judgement about a system or service, or one of its components; to provide a basis for making improvements; or simply to generate knowledge. One way of categorising evaluation, which is particularly useful when working with projects and programmes, is by its relationship to the state of development of the project/programme itself. Thus formative evaluation concentrates on ways of improving a project or a programme while it is still ongoing; summative evaluation is undertaken after the project or programme has been completed and judges its overall effectiveness. A useful distinction is that formative evaluations are usually intended for the project/programme participants while summative evaluations are intended for an external audience. Or:

"When the cook tastes the soup, that's formative; when the guests taste the soup, that's summative"
Bob Stake, quoted in Scriven, 1991, p. 169.

Evaluation can have many different purposes. What is important is that before you embark on an evaluation exercise you are quite clear as to your purposes, and that you keep focussed on these as you plan it, carry it out and present the results. Among possible purposes are:

To increase understanding of user needs. You may not immediately think of this as an 'evaluation', but in essence it is. The aim is to evaluate needs in the context of current services and to identify where your product or service might best fit. Many of the techniques used in user needs assessment are identical to those used in other types of evaluation.
To fulfil requirements for accountability. In terms of the JISC Programmes, there are contractual obligations on projects to undertake evaluation. However it is worth bearing in mind that these requirements are intended to assist both projects and programme. At the same time, projects which expend public money must expect to be held accountable, so evaluations are needed to gather evidence to support their claimed achievements. Do bear in mind that JISC is interested in the learning that comes from projects, so 'negative' achievements (e.g. evidence that a promising approach does not in fact work) is quite acceptable.
To determine whether processes are working correctly. For example, part of evaluation should focus on whether project management is effective. This might involve a review of how decisions are being taken and implemented.
To check whether a product conforms to specification. This is sometimes called 'quality assurance' but it is a type of evaluation nonetheless. Usually there will be explicit criteria against which conformance can be judged, including a formal functional specification. The key question is, 'Does this product do what it was designed to do?'
To judge the effectiveness of a project. Here the aim is to discover whether or not a project has produced its intended outputs. There will be an explicit link to objectives set at the start of a project, and usually this type of evaluation will need to include product conformance / quality assurance. However, it may well go much further - for example did the project gain as many users as anticipated?
To assess the outcomes of the project - the impact that it has had on its intended users. This is the most difficult type of evaluation, because there are always many variables at work in addition to those directly affected by the project itself. So, if we want to know whether a project has had an impact on the learning experiences of students we will need to find ways to isolate as far as possible the project's effects from others. We will also have to accept that it is rarely possible to have a true control group (i.e. a second group of people who were not exposed to the project but were otherwise exposed to identical variables).