By Howard White
Evidence – what is it good for? Absolutely nothing… if no one knows about it. But that is the fate of too much evidence, such as evaluation studies.
Evaluation has two functions: accountability and lesson learning. But no one is learning any lessons from the many evaluations which are done and then shelved. The knowledge in an evaluation is a public good and should be treated as such. But commissioners use evaluations for their own purposes, then generally neglect to promote their findings more widely. One purpose of evidence maps is to redress this situation.
Evidence maps present all the available relevant evidence for a particular research question. Evidence maps of the sort produced by 3ie and Campbell are easy to navigate, interactive pictorial representations of the evidence which allow the user to access the studies included in the map. Maps are produced following systematic review principles including comprehensive systematic searches. So users have access to evidence which may have otherwise have gone unread and under-used.
Most evidence and gap maps (EGMs) to date have focused on studies of effectiveness. That is, they have mapped impact evaluations. But, as shown in my companion article with Ashrita Saran on evidence maps, mapping is an approach which can be applied to any research questions. Following this principle, Campbell is embarking on the production of country evidence and gap maps, an approach we have piloted with the Uganda country evidence and gap map, C-EGM. The map has been produced in collaboration with the Africa Centre for Systematic Reviews and Knowledge Translation and the Office of the Prime Minister.
When presenting to the Uganda Evaluation Association last summer, I asked the audience how many RCTs people thought had been carried out of development interventions in Uganda. The highest guess was 10. In fact there are over 90 in the 3ie database, and over 200 impact evaluations of all designs. So I wondered about process evaluations. A quick scoping of relevant websites suggested there could be around 500 evaluation studies carried out in Uganda.
The Uganda map is still under development. But already there are indeed around 500 studies in the map, which shows some unsurprising patterns. The largest number of studies is in health, especially impact evaluations. But there are also substantial number of studies in other sectors such as education, macroeconomic policy and governance. An obvious next step is country-level synthesis to identify implementation issues.
We also looked at who wrote the evaluations included in the map. The majority of studies have no Ugandan authors. This is despite the considerable growth in evaluation capacity in the country over the years, as witnessed by the growth the Uganda Evaluation Association. The map will support redressing this problem it identifies.