Insights and Analysis

Evaluating Programs Using Systems Analysis

GKI’s Systems Evaluation of USAID Global Healthcare Commodity Supply Chains

How might we better use systems research and analysis in program design and evaluation?

Challenge

Whether tackling climate change or Ebola, achieving long-term, sustainable impact is nearly impossible without taking whole systems into account.  But while the challenges we seek to solve grow increasingly more complex, the tools and methodologies most often used to measure impact remain static.  As such, standard project planning and project evaluation methodologies mask both unintended consequences and numerous paths to success.  Cognizant of these limitations, the Monitoring, Evaluation, Research, and Learning Innovations Program (MERLIN) at the United States Agency for International Development (USAID) launched a Broad Agency Announcement seeking to source novel approaches to improve the Agency’s ability to measure, evaluate, and learn.  Through this process, USAID selected GKI as part of just four premiere consortia charged with offering insight, advice, and technical assistance needed to apply systems approaches at any stage in the project planning and delivery lifecycle.

Solution

A “system” is a set of actors and interactions that forms a coherent whole, performs a specific function, and has a boundary that sets it apart from the rest of the world.  At GKI, we equip our clients to analyze systems to make better decisions on project investment, planning, implementation, and evaluation.  Furthering previous work performed with The Rockefeller Foundation  in which we developed systems approaches to improve investment decision making, GKI was selected by USAID to offer insight and advice on how to bring GKI’s unique suite of systems tools and methods to USAID Missions and Bureaus around the globe.  GKI is doing so through the  (Strategic Program for Analyzing Complexity and Evaluating Systems) consortium – a group of systems experts that includes John Hopkins University’s Global Obesity Prevention Center (GOPC), LINC, Resilient Africa Network (RAN), and GKI who each bring comparative expertise to the group.

GKI’s evaluation allows us to transpose two systems together – to look and see what the system looked like, find the problems and the possible fixes. We can inform the way forward by looking backward. ~ USAID Staff Member

In 2017, the SPACES consortium launched a number of pilots, applying different combinations of the consortia’s systems tools to meet a range of demands for improved monitoring, evaluation, research, and learning in agriculture, health, education, etc.  Among these, GKI and SPACES completed a systems evaluation of one of USAID’s largest projects in its history, one aimed at strengthening global healthcare supply chains.  To begin the evaluation, GKI structured over 40 interviews, pulling out 126 pages of systems insights.  Through an interactive process in which key stakeholders were engaged in co-creating a visual depiction of the systems in which they operate, the GKI team then created diagrams of the global healthcare supply chain systems, mapping out the different actors and their complex interactions.  GKI used these diagrams to analyze the different motivations and roles of key stakeholders, highlighting drivers of system change that either enabled or thwarted USAID’s supply chain goals.  Our evaluation helped to identify (1) opportunities for increased efficiency along the supply chain, and (2) key indicators to be tracked in future monitoring and evaluation efforts in future supply chain projects.

Results / Outcomes

  • Facilitated an interactive workshop in which 50+ key systems stakeholders were able to align their perspectives on systems interactions, drives, barriers, enablers, and opportunities for change
  • Guided key system stakeholders through 25 iterations of visual maps of the global healthcare supply chains system
  • Identified 12 instances of misaligned incentives between key system stakeholders
  • Identified over 150 system enablers and barriers to achieving impact
  • Delivered 7 key recommendations for overcoming major system bottlenecks
  • Recommended over 40 indicators to be tracked in future supply chain evaluations
Other Insights and Analysis Initiatives
Photo Credit: Cait Goddard, GKI