Up-to-date research that informs policy and practice is vital in the constantly evolving field of international development. Research ensures development programmes respond to the realities of people. This has been recognised by the UK Department for International Development (DFID) who recently committed to supporting research in development with £220 million by 2010.
As research becomes valued then effective communication strategies between policy makers, practitioners, researchers and communities become ever more important. For example, in the last few years key funders have started to specify that their researchers allocate at least 10% of their budgets to research dissemination.
This change encourages non-government organisations (NGOs) and researchers to communicate their research and account for its impact. Healthlink Worldwide has been supporting the effective communication of research for a number of years and in September 2006 ran a series of activities, including a two-day workshop for over 40 participants that explored how UK and international NGOs could develop their capacity in this emerging field.
The workshop, funded by DFID, was convened by the Monitoring and Evaluation communicating research group (M&E research group). This group - a loose network of like-minded organisations and individuals - was formed in 2004 to address growing concerns with monitoring and evaluation in research communication. The group’s members cover many different disciplines and include organisations such as the International Institute for the Environment and Development (IIED), Institute of Development Studies (IDS), Panos London, Healthlink Worldwide, the International Network for the Availability of Scientific Publications (INASP), the Overseas Development Institute (ODI), Wren Media and SciDev.Net.
Nicholas Ishmael Perkins, Head of Communication, Networking and Learning at Healthlink Worldwide, explains: “A divide exists between the research community and practising development organisations as well as with sectors such as the media. Often practitioners in the development field feel excluded from research, believing it has little impact on the ground. The M&E research group’s direct experience of monitoring and evaluation means we are able to turn this into an opportunity to show how monitoring and impact assessment approaches bridge this gap and can support the need to communicate research.”
The M&E research group conducted a scoping study, documented stories, and held a workshop to provide an overview of the key issues in monitoring and evaluation (M&E) of research communications. The case studies gathered, some of which are featured here, show the approaches people are using, what is special about them, and how these might be integrated into projects.
Case study 1:“Shared Care” in
Burkina Faso - the challenges of implementation
Source: Ansgar Gerhardus et al
In the late 1980s ‘shared care’ was proposed by a group of researchers from the Ministry of Health. The approach was based on the idea that mothers and health workers could collaborate on the recognition, care-taking and treatment-seeking of childhood illnesses. However the approach has not been implemented; an evaluation sought to uncover the reasons.
Evaluation methods
Major stakeholders and the researcher’s agenda were identified
by a document analysis, which included a review of the research proposals,
published articles, minutes and reports and other project documents.
Semi-structured interviews with decision makers and researchers established
the environmental context at the time shared care was proposed. Focus
group discussions with staff, women’s groups and mothers - who
were not included in the original analysis - examined awareness of
the ideas of shared care, and how practical the approach was.
Findings
The evaluation found that although researchers managed
to disseminate their results to decision-makers, these were not transformed
into intervention. Failure to undertake effective stakeholder analysis
early on, to account for the policy context and awareness of the limited
time available of key decision makers to process information were major
constraints to implementation.
Made to measure – scoping paper
The workshop, which explored the subject of M&E for communicating research,
was supported by a scoping paper, written by Catherine Butcher and Gil Yaron.
It included a literature review, telephone discussions and case-studies, which
highlighted approaches and challenges faced by practitioners in the field of
research communications. The paper, designed to raise issues rather than form
conclusive answers, highlighted the following key findings:
Frameworks
The most commonly used framework for M&E is the logical framework (log-frame).
This framework, although well used, can present difficulties when capturing
the ‘networking’ aspect of communication. Alternatives, such
as mapping potential communication pathways, may offer a better way to monitor
the relationship between research communications and networks particularly
in smaller projects. Complex projects however, involving large numbers of
stakeholders and numerous communication routes, may respond better to different
approaches.
Stakeholders
Target audiences and ‘knowledge intermediaries’ such as policy
makers are often only identified in the broadest terms, with little information
known about their requirements. Establishing an ongoing relationship with
audiences and intermediaries could enhance the uptake of information and
help determine impact.
Methods and tools
A range of qualitative and quantitative M&E methods and tools are currently
in use. Quantitative methods are more commonly used for ongoing monitoring,
while qualitative, semi-structured interviews are popular for evaluations.
Read the scoping study (PDF 30 pages, 357 KB)
Case Study 2: World Wildlife Federation (WWF) - Evaluation of People
and Plants Initiative (PPI)
PPI was a 12 year programme in selected developing countries that
used applied ethnobotany (the scientific study of how plants are used
in different cultures) to promote the sustainable use of plant resources.
It aimed to build local capacity for community-based conservation. Part
of the programme is still running. The evaluation is unusual because
it assessed long-term change. The effects of research communication
of ethnobotany, including effects on local people’s livelihoods,
were examined.
Purpose
The PPI and its original objectives were reviewed as part of the
evaluation and lessons extracted for use by the WWF. The programme’s
relationship to WWF’s partnership with DfID was examined and
recommendations made for the next phase of the programme.
Approach and methods
A variety of methods were used including an evaluation of impact
at the very local level. Questionnaires were sent to over
800 people and 115 responses were gathered. Semi-structured interviews
and focus group discussions, which included people involved with or
users of PPI, were carried out by a team of local consultants. The
principal consultant and a colleague specialising in livelihood issues
attended the annual meeting of PPI in the UK, and PPI publications,
videos and website were consulted.
Challenges
Establishing impact was difficult as some of the processes of the
programme (for example accreditation of ‘wood certificates’ to
authenticate work by wood carvers in Kenya) were not in place. In some
instances effects were attributed to the programme but other influences
inevitably would have played a role in communicating the research findings.
Critical factors
Five key factors for effective M&E were identified at the two-day workshop.
These were:
Making space for learning
Creating spaces for learning within organisations was seen as a critical
factor by participants at the workshop. This allows M&E to be seen
as a learning approach rather than a ‘judgment’, which is
beneficial to organisations, partners and funders. One useful approach
is the ‘balanced scorecard approach’, which is set on criteria
that measures an organisation or project. It is a competency framework,
moving from a not-so-good position to a better position. The idea is to
reach a balanced situation within the organisation and the project. As
Geoff Bernard, at the Institute of Development Studies says: “It
makes it normal to be not achieving everything on the spread-sheet.”
Tools for the job
In addition, new technologies, such as network analysis software and
tracking of electronic discussion groups, are creating more options for M&E.
While it was noted that tools and approaches are useful learning from others,
sharing case studies and best practice is key to developing better M&E,
and the development of clear indicators for M&E and communicating research
(and for communication for development more broadly).
Involve your stakeholders
Stakeholder involvement is a critical factor. Different groups should
be identified at the beginning of projects and stakeholders should be
engaged in a participatory manner throughout all M&E activities.
Choose your framework wisely
The choice of conceptual framework for research and communicating the
research needs to be thought out in a holistic manner so practitioners
are clear about any changes that may occur. One workshop participant commented: “There’s
no shortage of theories of how communications work. There’s no shortage
of how to do M&E, but there is a shortage in the intersection of these
two areas.” Many of the workshop participants felt there were few
viable alternatives to using the logical framework for evaluation.
Who are you talking to?
Identifying audiences for different types of communication is essential,
and so to is identifying audiences for the M&E of the communication.
Different strategies are required for different audiences, and the language
and the format revised as necessary. Broad potential audiences include:
policy makers, donor and institutional leaders, practitioner level and
fellow project and programme implementers, the research community, and
others such as educational bodies and beneficiaries.
Case Study 3: Lao Fisheries
This is an example of multi-level evaluation: participatory M&E
undertaken by local communities as part of the adaptive learning process
and a more traditional impact assessment at the end of the project. Research
communication played a critical role in the adaptive learning; improvements
to management are possible only if the learning generates desired information
and this disseminated and utilised.
Purposes
To ensure that interventions met the needs of local communities
and to assess end of the project impact (primarily for funders and
implementers).
Approach
An evaluation framework was drawn up as a flow diagram, which formed
the basis for assessing learning outcomes. The diagram posed questions
of whether the information generated was expected, did it get to the
people who needed it, was the information used, and were the benefits
worth the costs? Was the information generated what was expected?
Formal research on which management practices worked best was initially undertaken and results were then evaluated by all stakeholder groups in terms of their implications for future management. Repeated cycles of M&E allowed comparison over time. By incorporating the views of various stakeholder groups, triangulation (cross-checking of results) was possible.
Methods
A combination of quantitative and qualitative methods were used
including group discussions and workshops at district level and in
villages, formal meetings, self-evaluation through questionnaires and
roundtable discussions. The evaluators were made up of district staff,
village representatives and provincial staff.
Challenges faced
Cultural factors: local beneficiaries were concerned not to appear
rude to outsiders – this suggested that evaluators from outside
the area may find a bias towards success rather than failure. The evaluation
was time-consuming for local people.
Conclusions
These are early days for M&E of communicating research. Following on
from the workshop the M&E research group has committed to addressing
some of the challenges and key factors identified. Next steps for the group
include:
Other areas of interest include:
For more information contact Shampa Nath, Director of Operations, Healthlink Worldwide at nath.s@healthlink.org.uk
For further details on the case-studies please refer to the scoping study.