Files
Abstract
In this paper we draw on impact assessment work of the Australian Centre for International
Agricultural Research (ACIAR) to present an example of meta-evaluation – an evaluation of
evaluations – in an agricultural research, development and extension setting. We explore quality
issues relating to evaluation studies in the context of government institutions. Program
evaluation standards (PES) are divided into categories of utility, feasibility, propriety and
accuracy to provide a framework for the meta-evaluation. The PES are presented as a universal
measure of evaluation study quality. The intent of using them here is to judge the adequacy of
PES as a universal quality measure or meta-evaluation base and to extract useful insights from
ACIAR program evaluation activities when developing a meta-evaluation model for the Limpopo
Department of Agriculture (LDA). Our meta-evaluation is undertaken of 63 impact assessment
reports. First, the literature guiding the conduct of a meta-evaluation is reviewed. Second, an
assessment (the meta-evaluation) of the evaluation studies is carried out for 19 sampled reports
from a population of relevant reports fitting the dimension of the analysis, and results are
presented and discussed. Also, lessons learned are presented, using the framework provided by
the meta-evaluation criteria. Third, taking into account the lessons learned, implications are
drawn for a proposed systematic meta-evaluation of the LDA. Finally, we conclude that all the
PES cannot be equally emphasized in a meta-evaluation model. At ACIAR, 70% of the
standards were at least partially addressed. Therefore, we succeeded in using the PES in
judging the ACIAR evaluation quality. As such, they can be an important base when developing
an evaluation model but should be applied in a contextualized manner.