This evaluation was performed by DevTech Systems, Inc. between 1 October 2015 and 8 April 2016 and the report was submitted on 8 April 2016.
Purpose of the Evaluation and Questions Addressed
This evaluation assesses the impact and sustainability of PRM-supported capacity building programs aiming to improve municipal authorities' assistance to internally displaced persons (IDPs) with respect to Law 1448 of 2011. These programs were implemented over the FY12 to FY15 period by Global Communities, International Relief and Development, and Mercy Corps. This report is organized around eight evaluation questions focused on a) Program design and implementation, b) Challenges and how the Government of Colombia (GoC) and its partners can address them; c) Unintended consequences of capacity building programs and d) Steps PRM can take to phase out its support to GoC municipalities.
This evaluation combines quantitative and qualitative data collection and analysis, drawing from document reviews, a desk review, and from 62 semi-structured key informant interviews and focus groups–54 of which were PRM-program related–with 170 individuals, including NGO staff, GoC officials, and IDPs, conducted between January and February 2016 in Bogotá, Popayán, Mocoa, Puerto Asís, and Cúcuta.
In their municipal authority and IDP capacity building programs, the three NGOs provide training, technical assistance, and guidance tools with success. The NGOs employ several best practices, including: i) Consulting with the GoC at national and municipal levels, ii). Operating with a high level of expertise, iii). Developing guidance materials that explain complex laws and related procedures clearly, and v). Employing pre-tests and post-tests.
The three NGOs, through their direct provision of humanitarian assistance (HA) and of technical assistance, are filling gaps, as municipal authorities and IDPs recognized in the interviews. Interviewees noted an overall improved understanding by municipal and departmental government authorities of their roles and functions with respect to providing HA in line with Law 1448, including planning and budgeting for municipal plans. The NGOs and GoC officials noted high turnover as a barrier to the NGOs' capacity building programs. The NGOs state that the guidance documents they develop help to mitigate the effects of high turnover due to elections and in between elections. However, turnover and changes in regulations and procedures are said to require continual technical assistance. Best practices with respect to preparing municipal authorities to retain relevant knowledge seem to include 1). Training by Global Communities using its guide on transitions; and 2). The targeting of the right level of officials who are less likely to be affected by change. While the NGOs "complement" municipal response in their provision of HA pursuant to Law 1448, the NGOs providing HA face a challenge as this amounts to “substituting” the primary role of municipal authorities, owing to the GoC's resource restrictions. In terms of unintended consequences, some unplanned training of NGO staff by program participants, and vice versa, occurred owing to the close relationship NGOs enjoyed with IDP and GoC program participants. PRM can take steps to transition out of some municipalities using several criteria, such as relevant indicators used by the NGOs, including a Municipal Capacity Assessment, and by the GoC, including, the National Planning Department’s assessment of municipalities, together with those in which the three NGOs have operated.
- PRM and the three NGOs should revise their indicators, in line with PRM’s revised General NGO Guidelines for Overseas Assistance, and adopt the Municipal Capacity Assessment tool developed by Global Communities.
- Resource permitting, NGOs should train and pay IDP leaders from the Municipal Participation Committees to provide in-person orientation to IDPs at Assistance and Orientation Points.
- PRM and the NGOs should explore whether and how best to extend the IHA Information System to the national level, in consultation with relevant GoC officials.
- PRM and the NGOs should consider developing “transition agreements” for with the GoC, to promote GoC ownership of the program.
- PRM should convene NGO staff to share their M&E tools and related results, with a view toward the adoption of these tools by each NGO. PRM and the NGOs should also consider integrating similar or related GoC indicators used in assessments of municipality performance vis-à-vis Law 1448.
- PRM should convene UARIV-Bogotá and the NGOs annually to share best practices, including M&E tools. Each NGO should conduct trainings on these tools with their GoC partners on a regular basis, in light of high turnover of government officials.
- PRM should review capacity building proposals for six key elements
- The NGOs should share their best practices on preparing candidates and newly-elected officials with one another.
- NGOs should continue their practices of selection based on vulnerability, and assess whether they should make any improvements to their processes, including during discussions of their best practices.
- Key program-related decisions taken by PRM and the NGOs it supports over the next five years should draw upon the uniform use of M&E tools as well as relevant GoC indicators and analysis where the tools do not integrate them.