Power, data and knowledge in development sector impact evaluation activities: critically engaging with the impact iceberg

The study opens with a simple observation: International Development is now a digitally data/knowledge intensive sector. This mundane remark suggests development has also become a rich domain for studying how power, data and knowledge are related. In data/knowledge intensive development, our ideas about impacts may appear empirical and objective, based on data and expertise, and often digitally immaterial. However, these ideas involve power and politics which shape data desires and data cells, knowledge products, packaging and pitching, governing processes and digital systems. Such dynamics place demands upon agents like non- government organisations (NGOs), transforming them from an old emphasis on tangible and physical relief, such as providing medicine, food, or clothes to those in need, to constructing and exchanging digital data and expert impact products in diverse development markets and bureaucracies. Processes, like impact evaluation are caught up in this shift to data/knowledge intensity and the power relations, inequalities and silences that accompany the shift. This study elucidates these power/data/knowledge relations, specifically with regard to impact evaluation itself, but more broadly regarding data/knowledge intensive development, or what some observers have christened “development 2.0”. Two bodies of literature are relevant to the study. Firstly, the normative discourse on how-to- do evaluation, and secondly, critical perspectives which acknowledge power and politics, and the impacts of (the doing of) evaluation. The prescriptive discourse is found to rely on old and problematic, yet still pervasive and implicit, models of data/knowledge construction which ignore or elide practice and power. Therefore, an alternative view of data/knowledge construction is proposed, and a research approach developed to explore how critical insights might live in the wilds of development sector evaluation practice. The approach draws on Cultural Historical Activity Theory (CHAT) to scaffold engagement with research partners and to critically analyse evaluation activities. The research design features two qualitative and collaborative cases, where the author offers consulting advice and simultaneously researches evaluation activities with two small NGOs, one expert at evaluation (based in India), and one learning about evaluation (based in Thailand). The CHAT analysis articulates evaluation activities, systems, contradictions, and the notion of temporal activity chains where impact data and knowledge are step by step, incrementally, edited over time, bifurcated, and packaged into exchangeable, legitimate impact products. The expert NGO case shows how this is effectively and instrumentally done, but the novice NGO case reveals the struggles and transformations required to become expert evaluators in development 2.0. Both cases reveal a range of silences, inequalities, assumptions, omissions, priorities, and opportunities which are part of contemporary impact evaluation mechanics. Three contributions are offered in response to impact power/data/knowledge problems. The first contribution, to evaluation literature, is the “impact iceberg”. This is a diagnostic, big picture tool which describes contemporary legitimate impacts, and amplifies the illegitimate edited-out impacts, those silenced or submerged under the iceberg waterline. The second contribution is a set of four provocative conceptual devices which audit evaluation practice and power, namely: “data/knowledge networks”, the “6P sensitivities”, the “impact spectrum”; and the “datamentality/datamateriality” oppositions. These concepts contest conventional audit cultures which align evaluation power and control towards managerial, commercial, digital, and expert domains. The final contribution is “critical engagement”, a methodological proposition, which tests and brokers critiques in engagements with research partners to see if and how they are accepted, rejected, or adapted in NGO evaluation practices. In sum, the contributions are advocated as components of an alternative “evaluation-as- practice” view. This view diverges from existing evaluation views, particularly scientific, business-pragmatic, participatory, and technology-centric perspectives. As such, the study speaks to the concerns of power/data/knowledge relations in the practices of evaluators, NGOs and donors whose work is part of development 2.0 today, as well as to information and knowledge management professionals and researchers who wish to not hide the power relations generated in the mundane data/knowledge intensive worlds they inhabit.