EFX: Evaluation support for FAIR and X4L Projects

Toolkit Navigation

- Toolkit Introduction
- Project Logic
- The Purpose of Evaluation
- Evaluation Goals and Questions
- Indicators of Achievement
- Evaluation Methods
- Analysis of Data
- Presentation of Results
- Timing Evaluation Activity
- Administrative Considerations
- Ethical Considerations in Evaluation
- General Links

Indicators of Achievement

When an evaluation is designed you need to define the performance indicators that you will use to judge achievement. In the simplest case this might simply be a threshold value: so, for example, you determine that the number of hits on your web site will be the indicator and that the threshold value will be 10,000 hits per day. It is important to design indicators carefully and to make sure that they mean what you intend them to mean. This example is a classic one where many different kinds of effects can result in unintended variation in your measurements: for example, caching elsewhere will reduce the number of hits but not the number of page views; a design utilising lots of files (e.g. images) per page may result in apparently high numbers of hits; and so on.

Just as with a research question, indicators and thresholds need to be defined precisely. For example:

it is better to state that "student examination marks will increase by 10% over 2 years"
than, "students will do better in their exams".

The following are some characteristics of useful indicators:

Strategic
The indicator tells you about outcomes rather than outputs: it addresses the significance of what has been undertaken
Focused
The indicator gets to the heart of the evaluation issue, rather than assessing something about the project in general terms. This is sometimes referred to as the rule of empirical relevance.
Attributable
What you measure is attributable to actions that the project has taken, rather than to something else that is happening. However, it can be very difficult in real-world situations to isolate project effects.
Neutral
As far as possible, the indicator does not send value-laden messages. All indicators convey messages to people who may not fully understand the limitations of the data and the data collection, so it is worth bearing this in mind when selecting them.
Transparent
Indicators should mean what they appear to mean, without a lot of interpretation. Web site hits is a good example of an indicator that doesn't always mean what people suppose!
Feasible
You may invent a superb indicator, but if it not possible to collect the data (within the resources at your disposal) it is of no use to you.
Economic
The effort and resources you have to expend to collect and process the data for the indicator must be proportionate to its value.
Repeatable
It must be possible to repeat the data gathering and analysis at a later time, and the meaning must remain the same.
Reliable
The indicator must produce consistent results. If you were to collect two sets of data and nothing was to be changed between the data collection exercises, then the value of the indicator should be equivalent. If my watch tells me that it is 12.10 p.m. every day when it is actually noon, it is a reliable indicator - which is not to say that it is correct, which requires an indicator which is ....
Valid
In essence, validity is about producing accurate results. In other words, the indicator should accurately reflect what has actually happened. If my watch gives a valid indication of the time, it must tell me the correct time.
Actionable
Finally, indicators need to be capable of being used. Although this may not be true in research studies, it will always be the case in development projects and programmes.