This Campbell systematic review examines the effectiveness of certification schemes in improving the welfare of farmers and workers. The review summarises findings from 43 quantitative studies, and 136 qualitative studies.
There is not enough evidence on the effects of CS on a range of intermediate and final socio-economic outcomes for agricultural producers and wage workers. There are positive effects on prices. But workers’ wages do not seem to benefit from the presence of CS. Income from the sale of produce is higher for certified farmers, but overall household income is not. Context matters substantially for the causal chain between interventions of certification schemes and the wellbeing of producers and workers.
Generally, the quality of the studies is mixed, with a significant number of studies that are weak on a number of methodological fronts.
The rise of voluntary standards and their associated certification for agricultural products is a well-established phenomenon in the contemporary dynamics of agricultural trade. Supply chain management is increasingly influenced by a proliferation of standards, and by the organisations setting and monitoring them over a growing number of products. While the objectives of standards and certification schemes (CS) vary, the focus of this review is on social sustainability standards, which are closely related to ethical trading and to schemes that focus on socio-economic outcomes of participants, essentially agricultural producers (particularly smallholders) and wage workers, whether employed by corporate plantations or individual agricultural producers.
This systematic review addresses the extent to which, and under what conditions, CS for agricultural products result in higher levels of socio-economic wellbeing for agricultural producers and workers in low- and middle-income countries (LMICs). The primary review question is:
What are the effects of certification schemes for sustainable agricultural production, and their associated interventions, in terms of endpoint socio-economic outcomes for household/individual wellbeing in low- and middle-income countries?
The subsidiary review question is:
Under what circumstances and why do certification schemes for agricultural commodities have the intended and/or unintended effects? What are the barriers and facilitators to such certification’s intended and/or unintended effects?
We systematically searched for available literature from a wide range of sources. Several bibliographical databases were consulted. A very significant amount of time was devoted to a systematic search for relevant items through hand searching in targeted databases and websites, including consultation with relevant stakeholders in the community of standard-setting organisations. In this field the ‘grey’ literature is very important. Thus, the standard bibliographic databases would not be enough to find all relevant material. Papers in English, French, Spanish, German and Portuguese were considered. The references retrieved for this review are up-to-date as of November 2015. Some key references were added in July 2016 as a result of consultations with the ISEAL Alliance.
We included studies that evaluated the effects of CS on socio-economic outcomes for agricultural producers and workers. We defined eligible CS as those based on second (industry-level) or third-party certifications thereby excluding own-company standards. We examined the main types of interventions usually implemented by CS, organized around four groups: (a) capacity building, (b) market interventions (including price interventions, credit support, guaranteed market outlets, etc.); (c) premium-funded social investments, and (d) labour standards. In most cases CS adopt combinations of these groups of interventions. We included studies that report at least one intermediate or final outcome of interest.
For the effectiveness review, we selected studies that use experimental and quasi-experimental methods, and other studies that demonstrated control for selection bias and sufficient confounders. We selected studies that provided relevant comparisons with non-certified groups. For questions on barriers and facilitators and contextual factors we searched for and screened qualitative studies that reported on relevant outcomes, that had sufficient reporting on methods, and provided substantive evidence on key selected themes to complement the effectiveness review.
We used a combination of single screening with substantial piloting and supervision in initial stages, and double screening with arbitration for disagreements in coding and inclusion/exclusion decisions for full-text review.
We developed separate coding tools according to the requirements of our two review questions. To compare effects on variable outcomes across studies we calculated standardised mean differences. The quantitative results were synthesised using inverse variance-weighted random effects meta-analysis. Only one effect size per outcome per study was included in any given synthesis. The analysis of qualitative material was organised around three main thematic areas: barriers and enablers in implementation dynamics; distributional dynamics, including gender equity issues; other internal and external contextual factors and barriers and enablers.
The initial search returned 10,753 studies, which, after dropping duplicates, a large number of irrelevant papers, and applying the selection criteria, were reduced to a final sample of 43 studies from 44 papers for review question 1 (effectiveness), and 136 studies from 114 papers for review question 2. All were published between 1990 and 2016. The majority of our material comes from research reports, working papers, book chapters, and theses.
The included studies for the quantitative and qualitative syntheses provide evidence on a range of rural settings in LMICs, with dominance of cases from Latin America. Despite the fact that there are many CS operating with agricultural commodities, included studies only cover a group among them (12 CS), which have attracted more research in the form of impact evaluations. Fairtrade certification is particularly well represented in the literature, with over half of the total number of included studies. Several agricultural products are covered by the included studies but coffee (38%) and fruits (17%) combined account for more than half of studies. In terms of population, a large majority of studies (77%) focus on agricultural producers, whereas the research on employment outcomes is rather limited.
The quality of the included studies is mixed. The proportion of quantitative studies with high risk of bias ratings was relatively large. There are no randomized controlled trials (RCTs) but there is a range of quasi-experimental designs employing different techniques of data analysis. Given the paucity of calculable effect sizes per outcome and the variety of methods used in different studies the meta-analysis encountered difficulties, and the number of studies with low or moderate risk of bias included for the synthesis of effects for each outcome is very small. Although there are many included qualitative studies of high quality, especially ethnographic research, the overall quality of this group is mixed as well. Several studies, especially non-ethnographic contributions, are only borderline in terms of minimum reporting standards.
In terms of quantitative results, we find that the available quantitative evidence does not give a clear picture of the impact – or lack thereof – of certification schemes. The synthesised effects for our key intermediate and final outcomes are summarised below. For each outcome we present the difference between certified groups and control groups in standardised percentages, with a central estimate and a likely range around the estimate, which reflects the uncertainty inherent in the estimate, added in parentheses.
In most cases, disaggregation by type of CS did not yield conclusive results, although for some CS results were more mixed than for others. Such is the case of Fairtrade for yields and income measures.
The qualitative synthesis discussed a wide array of factors affecting the causal chain in different nodes along the chain, such as: producer organisations (POs) and their characteristics, particularly heterogeneity and power relations within them; relations with buyers and exporters; business models linking buyers and producers (whether open spot markets, contract farming or a mix); national institutions shaping the dynamics of agricultural trade and labour relations; barriers imposed by direct and indirect certification costs, which negatively affect adoption or the size of benefits accruing to producers; availability of additional external support, often critical for adoption and sustained maintenance of standards; inconsistency in monitoring and auditing practices; heterogeneity of participant groups and the effects of inequality on POs management and the sharing of benefits; difficulties in addressing deep-rooted structures of inequality based on gender; the relative invisibility of large segments of agricultural wage workers, notably those employed by small farmers.
The mixed and inconclusive quantitative effects, combined with the wide range of contextual factors to take into consideration, underline that CS operate in complex environments with multiple interventions, goals, actors and contexts, and as such they do not operate in a social, institutional and economic vacuum.
Overall, we found mixed results and a dominance of weak or not statistically significant effects. There were both positive and negative effects for different outcomes. Even within a given CS there is substantial variation in effects across different outcomes. Thus, it is hard to conclude anything about whether any particular CS performs better compared to others over a range of outcomes. Without more systematic high-quality quantitative evidence on intermediate and final outcomes it is difficult to draw meaningful conclusions with actionable findings. Context hugely matters, as the range of contextual factors and barriers and enablers is vast. This is not surprising and most Theories of Change developed for selected CS acknowledge the centrality of context specificity. Nonetheless, the reviewed qualitative research reveal a number of key barriers and facilitators or contextual features that seem important to understanding the impact of CS. Practitioners can extract some lessons about the kinds of contextual factors that seem prominent in mediating the impact of their interventions, such as the characteristics of POs with which they partner, the deep-rooted social relations of inequality, including gender dynamics, in rural areas of LMICs; the direct and indirect certification costs, and their determinants; the specificities of each supply chain and especially existing relations between established buyers and producers; and the national and local contexts of regulation and economic development.
There are various implications for researchers. First, there is scarcity of high-quality impact evaluations, and a disproportionate attention to some CS and almost no attention to several other CS. The volume of research with rigorous study designs has fortunately expanded in the last 10 years but this review calls for more studies and on more outcomes, especially on employment effects, which have received less attention so far. Second, mixed-methods theory-based evaluations with appropriate counterfactual designs are likely to generate more valuable findings, given the importance of context and the need to link effects with barriers and facilitators in each study. Third, reporting standards must be improved, so published papers should devote more space and attention to reporting details of how research was conducted, limitations and all the relevant statistical information. Many studies had to be excluded from this review or from effect size calculations because of basic reporting gaps.