Healthlink Worldwide

Monitoring and evaluation workshop: The time and space to evaluate

Twenty-two participants from Kenya, Uganda and Zimbabwe enjoyed the time to explore challenges in monitoring and evaluation (M&E) at the learning forum in Nariobi, Kenya, May 2005.

Participants were partners of Healthlink Worldwide and the Kenya AIDS NGO consortium (KANCO), who co-hosted the forum with Healthlink Worldwide.

The learning forum, which formed part of the ‘Enhancing HIV and AIDS communication in Sub-Saharan Africa’ project, aimed to enhance the capacity of partners and other local organisations to communicate effectively about HIV and AIDS. The forum provided a chance for participants to share approaches and activities and look at some current trends in evaluation practice.

Allan Ragi, director of KANCO introduced the forum. He told of how Healthlink Worldwide and KANCO had built up a leadership role in the arena of strengthening access to information. He explained that the forum was an opportunity to see increased learning and greater impact in the work of partners.

As part of the forum M&E approaches were shared in an open space ‘marketplace’ and role-plays helped participants draw on their tacit knowledge to reflect on the complex challenges of M&E. More structured sessions helped participants analyse important changes in their work, and to critically question aspects such as the development of indicators. Participants also assessed the potential of new approaches that enhance the potential for learning from M&E, such as the ‘Most significant Change’ method (which using stories of change to gauge impact), and the ‘Outcome Mapping’ approach (which looks at contribution to development through influence on those an organisation works with directly).

Why monitor and evaluate?
An initial session, which explored the understandings and assumptions people have regarding M&E, brought out some key points for participants to:

• ‘Do what we do more effectively’
• ‘Document the learning form our work’
• ‘Be clear about our impact’.

Importantly, the question was raised of ‘who is M&E is for?’

Participants recognised the need to have involvement and ‘ownership’ of the process by ‘beneficiaries’ (the people affected by the development). The value of building on achievements, sharing what works, and using methods for collective discussion, analysis, problem-solving and action was also stressed. Challenges included;

• The need to consider accountability downwards to intended ‘beneficiaries’ as well as upwards to donors
• Using M&E to help check the relevance of work for a range of stakeholders
• Developing adequate capacity and skills to do M&E
• Securing institutional support.

At the same time M&E needs to be ‘de-mystified’ so that fear of M&E is dispelled and, in the words of one participant, we can “move from policing M&E to learning M&E”.

Sharing stories of significant change
Participants shared stories of significant changes in their own work. This helped to introduce them to each other’s work, and to bring out a sense of the ‘criteria of success’ and what they wanted to measure when evaluating their work.

One approach that brings rigour to the collection and analysis of stories to assess development impact is ‘Most Significant Change’ (MSC). MSC looks at development impact through an accessible but rigorous story-telling approach. The method also promotes dialogue and clarification of the core values behind interventions. The stories are filtered and selected as ‘most significant’ and passed up the organisation as examples of impact.

Case study
Three participants’ stories were selected for more analysis and comparison to demonstrate using the MSC approach. One story was about a project in Uganda to promote children’s participation in building coping strategies among children with HIV and AIDS in primary schools. After hearing the story, participants picked out a wide range of things that were significant and looked at the bigger issues behind them. In one example, the fact that the teachers had apologised for discriminating against families living with HIV and AIDS pointed to a greater acceptance, understanding and support on the part of teachers. In another, the fact that children had formed their own support clubs highlighted a growing self-reliance and empowerment on the part of the children.

Participants were impressed to see the range of issues that could be drawn from the stories and a number of people hoped to make use of the method in their work. Discussion highlighted the value of using MSC as a complementary approach to more traditional methods. It had the benefit of producing striking and concise accounts of development changes that were more accessible and more likely to be used than the often neglected traditional evaluation reports. Other issues raised in discussion were the tendency for stories to have more impact if they are told in an animated way and challenges of the MSC approach including, who might dominate the selection of stories, the importance of documenting the reasons why a story is seen as significant, and the challenge of trying to attribute impact.

Outcome mapping
This is a method that looks at ‘contribution’ to development through influence on the organisations and individuals that a project or programme works with directly. The approach focuses on the immediate influence a project can expect to have; it recognises that other individuals and organisations have their own values and methods and that many factors influence development outcomes. The approach also provides a modest but systematic way of tailoring an organisation’s strategy and activities to give maximum impact. Participants saw the promise in the method, and appreciated the logic that if an intervention is truly participatory, the direct input and influence of the donor or agency should be expected to diminish over time. Questions were raised again about the challenges of attributing impact.

Developing Indicators
A session on indicators, or ‘approximations of complex process, events and trends that reveal changes representative of a bigger issue’, was useful for participants. In an exercise participants drew on their experience to develop three or four objectives for projects including children’s participation, organisational change, and HIV and AIDS communication. They then developed indicators for these objectives, and considered appropriate sources of information for each of the indicators. Discussion emphasised the need to have a few, clear indicators, to make sure qualitative indicators aren’t neglected, and to be clear about what an indicator is trying to achieve.

The ‘marketplace’
In the popular marketplace session, participants shared a variety of approaches and methods useful in their own work for M&E. These included:

• Peer educators’ diaries for tracking their work and development
• Forms to gather information from communities about HIV and AIDS
• An organisation wide M&E strategy
• Methods for interviewing children and tracking their development
• A method of tracking the nutrition and health of people living with HIV and AIDS using photographs
• Using stories to track impact
• The use of a mystery client exercise
• Exit surveys to assess the quality of Voluntary Counselling and testing

Challenges in Monitoring and Evaluation
Drawing on the range of challenges in M&E that had already emerged at the workshop, participants developed striking and humorous role-plays to reflect on difficulties often encountered and how they might be overcome.

Role-play case studies
From the perspective of People Living with HIV and AIDS (PLWHA), M&E was often an intrusion into their lives by different organisations asking the same kinds of questions repeatedly, with questionnaires and tools that were hard to understand and not clearly relevant to their own lives. There was a lack of involvement and ownership by PLWHA. This powerful role-play highlighted the need to be sensitive about collecting data on people’s lives; to co-ordinate with other agencies to avoid burdening communities and to develop the M&E tools in a participatory way that took account of culture and the realities of their lives.

Another role play from the point of view of a district official showed them trapped in a numbers game, trying to provide last minute statistics for a government spokesperson with ‘everybody running around but no-body really knowing why’. This emphasised the need to know who evaluation was for, why it was being done, and the danger of using numbers to support “political legitimacy”, rather than using M&E to really learn from work for future improvements.

Taking Monitoring and Evaluation forward
Participants brainstormed a range of next steps based on what they;

• expect to see
• would like to see
• would love to see

A number of people hoped to draw on the MSC approach to strengthen their own M&E, while others felt supported to strengthen M&E in their own work and argue for a greater allocation of time and budget. A number of participants wanted to develop M&E strategies for their organisations and link this to developing stronger proposals.

Participants expressed an interest in a follow up workshop to check on progress and hoped that KANCO and Healthlink Worldwide would continue to support the development of M&E.

Developing an Monitoring and Evaluation Plan
Using a framework provided by the facilitators, participants worked together on central components of designing a Participatory M&E plan. Small groups worked on different core aspects including: establishing a core team; making an overall M&E plan; selecting methods to collect and analyse data; sharing and using the findings. Each group looked at what questions they needed to ask to develop their plan, what difficulties they anticipated and what would help. Breaking down the process and driving it with simple questions allowed participants to confidently develop plans, review a range of challenges, and consider how they might deal with these.

“I will certainly use the knowledge, skills, methods and insights developed to improve M&E in my organisation and to provide a sort of organised framework within which to work." Participant at the workshop.