Evaluating Programs Using Systems Analysis
GKI’s Systems Evaluation of USAID Global Healthcare Commodity Supply Chains
How might we better use systems research and analysis in program design and evaluation?
Challenge
Solution
A “system” is a set of actors and interactions that forms a coherent whole, performs a specific function, and has a boundary that sets it apart from the rest of the world. At GKI, we equip our clients to analyze systems to make better decisions on project investment, planning, implementation, and evaluation. Furthering previous work performed with The Rockefeller Foundation in which we developed systems approaches to improve investment decision making, GKI was selected by USAID to offer insight and advice on how to bring GKI’s unique suite of systems tools and methods to USAID Missions and Bureaus around the globe. GKI is doing so through the (Strategic Program for Analyzing Complexity and Evaluating Systems) consortium – a group of systems experts that includes John Hopkins University’s Global Obesity Prevention Center (GOPC), LINC, Resilient Africa Network (RAN), and GKI who each bring comparative expertise to the group.
GKI’s evaluation allows us to transpose two systems together – to look and see what the system looked like, find the problems and the possible fixes. We can inform the way forward by looking backward. ~ USAID Staff Member
In 2017, the SPACES consortium launched a number of pilots, applying different combinations of the consortia’s systems tools to meet a range of demands for improved monitoring, evaluation, research, and learning in agriculture, health, education, etc. Among these, GKI and SPACES completed a systems evaluation of one of USAID’s largest projects in its history, one aimed at strengthening global healthcare supply chains. To begin the evaluation, GKI structured over 40 interviews, pulling out 126 pages of systems insights. Through an interactive process in which key stakeholders were engaged in co-creating a visual depiction of the systems in which they operate, the GKI team then created diagrams of the global healthcare supply chain systems, mapping out the different actors and their complex interactions. GKI used these diagrams to analyze the different motivations and roles of key stakeholders, highlighting drivers of system change that either enabled or thwarted USAID’s supply chain goals. Our evaluation helped to identify (1) opportunities for increased efficiency along the supply chain, and (2) key indicators to be tracked in future monitoring and evaluation efforts in future supply chain projects.
Results / Outcomes
- Facilitated an interactive workshop in which 50+ key systems stakeholders were able to align their perspectives on systems interactions, drives, barriers, enablers, and opportunities for change
- Guided key system stakeholders through 25 iterations of visual maps of the global healthcare supply chains system
- Identified 12 instances of misaligned incentives between key system stakeholders
- Identified over 150 system enablers and barriers to achieving impact
- Delivered 7 key recommendations for overcoming major system bottlenecks
- Recommended over 40 indicators to be tracked in future supply chain evaluations
Resources
-
GKI’s Systems Research and Evaluation Capabilities 2-pager
Overview of GKI’s trainings, resources, and support to boost the application and benefit of systems approaches
-
SPACES Consortium Overview
A short breakdown of GKI’s role supporting systems approaches within USAID in the Strategic Program for Analyzing Complexity and Evaluating Systems
-
Systems and Complexity White Paper
This white paper is a collaborative effort of the SPACES Consortium, designed to frame the international development landscape for application of systems and complexity approaches to design, monitoring, and evaluation.