mmu | cerlim

The eMapps.com Project Evaluation - A Workbook for Teachers

Evaluation Diary

This is a summary of key evaluation milestones and activities, which show how work progressed with the evaluation through formal meetings and events throughout the eMapps.com project. In addition the, joint MMU and EUN evaluation team hold regular telephone and face-to-face meetings, and liaise by email.

Prague Kick off meeting: February 2006
The Evaluation team met representatives from the management team and the Country Co-ordinators, and explained the nature and purpose of evaluation. This helped participants understand the relationship between the evaluation, the project and themselves, and introduced them to our planned evaluation activities.

A very successful 'History of the Future Exercise' was carried out with all meeting participants. This made explicit each individual's expectations of what the project would achieve, and provided useful data with which to compare different perspectives.

We also learned about concepts integral to games and gaming; alternative reality and rabbit holes, game structure and design, narrative, plot and storylines.

Nida summer school: April 2006
The summer school provided the first opportunity for us to meet and get to know teachers from the eMapps.com schools in the beautiful and peaceful surroundings of Nida, Lithuania. Over the course of five days we took part in a specially created eMapps.com game, "The Amber Room Game" which introduced all participants to the experience of navigating a territory using handheld GPS devices, solving clues and puzzles, and uploading video, audio, image and text material to the prototype game desktop.

We also repeated the 'History of the Future Exercise' with the teachers, and gathered more data to compare their perceptions with those of the management team and country co-ordinators. The report on the outcomes of this exercise can be found at http://www.cerlim.ac.uk/projects/emapps/HOF.doc.

First evaluation questionnaire: June to September 2006
This was circulated to Country Co-ordinators and teachers in early June. It was created using the Zoomerang online questionnaire tool, hosted by EUN, for ease of submission. A Microsoft Word version was also supplied so that teachers could consider and discuss their responses before replying. A total of twenty-three responses was received; at least one from each country. The questionnaire closed at the end of September 2006, and a report was prepared for submission to the November Review meeting.

La Coruna Games Design Meeting: August 2006
During the early summer of 2006 teachers began to return their three completed Games Design Worksheets to Daniel Weiss, the game platform developer who circulated them to the Evaluation team. It became clear that although teachers had many imaginative and creative ideas for game content, there was some confusion about how games should be designed, structured and implemented using the game platform and the handheld devices.

Daniel invited the MMU evaluators to visit his office in La Coruna, Spain to discuss how best to address these issues and to talk about our planned evaluation activities and their fit with his design work. A most useful two-day meeting ensued, during which we considered each team's game design. We also learned much from Daniel about his concept of the eMapps.com game, and his knowledge of mobile technologies and handheld devices. This helped us to form a clearer picture of how the games work as a whole, and how he envisaged that they would be played.

Deliverable D12 submitted: August 2006
The first deliverable due for Workpackage 6, the Validation and Evaluation of Impact, was delivered to the Project Co-ordinator on schedule at the end of August. This deliverable, D12, was the first stage of the Evaluation Workbook for Teachers, with an explanation of how this would be developed as a tool to provide help, information, advice and support throughout the project.

Slovenia Workshop: November 2006
This workshop provided another ideal opportunity for the evaluation team to take part in a cross-project meeting with a relaxed and informal atmosphere. Although there was no evaluation presentation this time, the opportunity to network with teachers and country co-ordinators helped to raise the profile of our activities and to clarify our role in the project. Teachers from all of the NMS schools presented their games at the workshop, and received feedback. We learned much about the progress which had been made with the games design, and the level of understanding of the technologies.

Helsinki Review: November 2006
The ongoing work of the evaluation was presented to the project reviewers at this meeting in Helsinki. As the main body of work will be carried out later in the project, only preliminary findings could be presented, but it was agreed that the evaluation was on track and was being well managed.

Second evaluation questionnaire: January 2007
This was circulated to Country Co-ordinators, again using the Zoomerang online questionnaire tool and accompanied by a Microsoft Word version. This questionnaire focuses upon the pedagogical outcomes of playing the games, so responses are not required until after the children have done so. Country Co-ordinators were asked to implement the questionnaire, by chairing one focus group at each partner school, where one collective response will be gathered for all teachers. This will then be submitted online by the Country Co-ordinator.

Torun Summer School: June 2007
The second Summer School, held in the beautiful Polish city of Torun, was a major opportunity for us to move our work forward. Our focus on usability and accessibility issues was facilitated by small group work sessions with teachers, technicians and country coordinators and by a full day observation of children playing an eMapps.com game at their game base in the hotel, and in the game territory in the city centre. We were able to collect much useful data for formal reporting, but also enjoyed informal discussion with project partners. Libraries Without Walls 7 Conference: Lesvos, Greece, September 2007 We were pleased to include an eMapps.com presentation given by Romana Krizova and Rob Davies in the Conference Programme.