Request for Proposal: Technical support for the design of evaluation methodology and evaluation – Colombia
Request for Proposal: Technical support for the design of evaluation methodology and evaluation – Colombia
Global Alliance for Trade Facilitation/Center for International Private Enterprise
1) Background information
The Global Alliance for Trade Facilitation is a public-private partnership for trade-led growth, supporting governments in developing and least-developed countries in implementing the World Trade Organization’s Trade Facilitation Agreement. Alliance projects cut through red tape and end costly delays at borders by bringing together governments and businesses of all sizes as equal partners to deliver targeted trade reforms.
By emphasizing digitisation and delivering other best practices, Alliance projects enable businesses to trade more easily thanks to streamlined and more predictable processes. Governments save time and resources by modernising trade procedures while still safeguarding their borders. Ultimately, Alliance projects boost trade competitiveness and business conditions, which are key drivers of inclusive economic growth and poverty reduction.
The Alliance is led by the Center for International Private Enterprise (CIPE), the International Chamber of Commerce, and the World Economic Forum, in cooperation with Gesellschaft für Internationale Zusammenarbeit (GIZ). It is funded by the governments of the United States, Canada, Germany, Denmark, and Australia.
Through the Alliance’s process of co-creation, the public and private sectors in Colombia established advance rulings as an approach that synthesizes and addresses meeting the transparency and predictability goals of the current customs reform efforts.
As a part of the Alliance, CIPE has designed and is implementing this project to establish a Center of Excellence, in partnership with relevant government agencies, customs clearing agents’ associations, and local and international businesses, in Colombia. The Center is expected to produce long-term benefits for both the public and private sectors in time and cost savings at the border.
The Center of Excellence is an innovative approach addressing advance ruling challenges in Colombia, specifically those experienced by the automotive sector. The role and purpose of the Center with regards to advance rulings is threefold, with aims to harmonize the decision-making process at the national level, link public and private sector stakeholders working to simplify and streamline customs operations, and facilitate the discussion and adoption of best practices in logistics and foreign trade. In the process, the Center promotes the use of advance rulings, provides recommendations to trade users, and improves the overall competitiveness of the automotive sector. While the Center does not issue advance rulings, it encourages the use of advance rulings and advises the Colombian legal and customs (DIAN) division on the requests received from the private sector.
Based on the model, scope of work and initial mandate of the Centers for Excellence and Expertise in the United States, the Center both centralizes and harmonizes decision-making on the automotive industry, no matter the point of entry for imported vehicles and their parts. Centralized around commodities rather than the importer’s physical location, the Center brings all of Customs’ trade expertise, such as import specialists, national import specialists and national account managers, to support the automotive industry in one strategic virtual location. For this project, the private sector and government identified the automotive industry as the initial sector in the interest of improving the overall competitiveness of the sector and attracting investment.
Underlying Need for the Center of Excellence
The automotive industry importers, exporters and vehicle assembly companies have repeatedly raised classification, valuation, and origin (C/V/O) as a major concern at the border in Colombia. Although DIAN, with the support of donors, had taken steps to address C/V/O challenges and promote transparency with industry actors before approaching the Alliance, Colombia continued to face challenges that prevented satisfactory levels of an advance ruling’s regime. Colombia’s advance rulings process, as a trade facilitation tool, was not harmonized and the process was slow and unpredictable. As a result, the private sector rarely utilized advanced rulings, resulting in increased time and higher costs at the border for the private sector, and inefficient use of resources by Colombian Customs.
2) The assignment
As a part of the Alliance, CIPE plans to conduct a rigorous (and independent) evaluation at the conclusion of project activities in Colombia. The evaluation aims for both accountability and learning purposes. This RFP is in search of an Evaluation Consultant to provide Alliance decision makers and stakeholders with robust information on the results of the Colombia Center project. Lessons learned will contribute to organizational learning, feed into the design and implementation of future Alliance projects and build knowledge on trade facilitation interventions. CIPE and the Alliance plan to publish public versions of any completed evaluation reports.
i. Project Evaluation Plan: The Consultant will develop a suitable evaluation methodology and workplan outline in consultation with the Alliance. Cost-effective designs should be considered. The timing of the research, evaluation, and reporting should be in line with end-of-project activities in June 2021.
The evaluation plan should include the following:
- Evaluation Methodology
- Data collection tools (interviews, focus groups or other participatory methods, protocols, tabulations, etc.)
- Where sampling is recommended, the following must be defined and explained in detail: the purpose, objectives, universe/population, sampling criteria, sample design, sampling frame, sampling unit, sample size, sampling method(s), proposed sample and limitations.
- Detailed timeline for the evaluation, including a detailed (virtual) fieldwork plan if necessary
- Proposed evaluation report outline.
ii. Confirmation of evaluation questions and data availability: The Consultant will review and update the evaluation questions (if necessary), taking into account data availability and accessibility. The purpose, scope and evaluation questions should inform the most appropriate methodology for the evaluation. In addition, the Consultant must examine and document the existence and quality of data (including gender-disaggregated data), especially government administrative data and/or business data, availability of key informants and the timing of the evaluation.
iii. Data Collection, Analysis and Data Analysis Reports: The Consultant will then synthesize and analyze the available data (QUAL and QUANT) and produce a draft data analysis report addressing the specific project outcomes, including findings and validation.
iv. Evaluation Reporting: The Consultant will compile a draft Evaluation Report based on the Data Analysis Reports. Following review, the Consultant will incorporate comments made by the CIPE program and M&E team for a final Evaluation Report. CIPE may also share the draft report with Alliance partners for additional feedback prior to finalization.
3) Suggested Data Collection Methods and Sources
In addition to the Consultant’s own determination of necessary data sources, the following are presented for consideration:
i. Quantitative data collection
- Baseline data:
Validation/triangulation of past Total Transport and Logistics Cost (TTLC) baseline results through a focus group workshop or structured interviews with local experts/stakeholders using a simplified survey instrument similar to the Trading Across Borders (TAB) approach. TAB collects data by administering a survey to a small sample of local experts (less than five on average) per country.
Data produced by the Total Transport and Logistics Cost (TTLC) tool should be considered as a primary data source. TTLC baseline data will be re-validated with local experts and assessed for a quality automotive sub-sample and will be available to the Consultant.
Data from secondary sources should be considered as an important component for triangulating baseline results and measuring outcomes listed in the theory of change.
- Endline data:
The endline assessment should consider both quantitative data (collected from secondary sources, focus group or small-scale survey as per determined in the baseline) and qualitative assessment to address the evaluation questions and to collect stories of change.
ii. Qualitative assessment
- A qualitative assessment may also be used to capture the changes in level of trust between the private and public sectors and potential perceived impact of the project on time and cost.
- Should there be marginal indicators of project results and impacts, the Consultant may consider using a Most Significant Change (MSC) approach to identify and report other potential areas of impact outside the time-cost framework.
4) Key considerations
The following are to be considered during the evaluation design and implementation:
- It is acknowledged that each evaluation occurs in a unique operating environment, however evaluations should reflect methodological rigor.
- Evaluation data should reflect the Theory of Change of the project and corresponding indicators, (see Annex 1) and concentrate on the Project Results.
- Where possible, the evaluation should capture the influence of the unique approach of the Alliance to trade facilitation, including the public-private cooperation in co-creation and co-implementation.
- The audience of the evaluation includes multiple stakeholders: project implementers, partners and funders, and others who may benefit from a clear understating of project results and lessons learned.
Based on the conceptual evaluation plan provided below, a preliminary budget of $50,000 plus contingency costs of $5,000(10%) has been planned. A final budget is dependent on the final Terms of Reference.
5) Project management
The Consultant shall report to Mrs. Danuta Dobosz, Sr. Manager for Monitoring, Evaluation and Learning at CIPE. Progress reporting may be established in any way that meets the project, project manager and consultant needs, but should, at a minimum, include progress reports reporting and analysis using the format identified in the inception report.
Additional details about the Consultant’s expected coordination and collaboration will be provided in the final Terms of Reference for this evaluation. The Consultant is anticipated to work throughout the evaluation with an Evaluation Taskforce, which will include the CIPE Monitoring, Evaluation and Learning for Trade (MELT) Team, Alliance M&E staff, as well as CIPE Program Officers.
6) Evaluation questions
The following is the list of draft evaluation questions for the Colombia assessment. The evaluator may choose to revise questions based on the findings in the inception report.
- Over the course of the project, has DIAN experienced any changes in how it works with automotive industry importers and vehicle assembly companies? Was there a reduction of classification, valuation, and origin (C/V/O) disputes?
- Did traders observe time/cost savings after project completion? How are traders reporting about their observations of time/cost savings?
- What is the private sector perception of the Center of Excellence? Is the Center utilized by the private sector to facilitate their imports? Why or why not?
- Are there any transferable lessons learned from the Center in Colombia? Can other sectors benefit from the results of this project?
7) Deliverables
Required deliverables for the Colombia Center evaluation are:
- Inception Report
- Technical Memorandum on data collection tools and dataset properties
- Preliminary Findings Analysis and Report
- Sensemaking Workshop (Presentation and discussion of preliminary findings)
- Progress Reports (Every other month)
- Evaluation Report
Report Specifics:
- Inception Report
Work may commence upon signature of a valid contract with the Consultant and include an inception period. The inception period should result in an Inception Report that, at a minimum, includes the following:
Confirmed/finalized evaluation methodology
Data collection plan, including design and final survey tools
Revised evaluation schedule (if necessary)
Project management plan, including progress reporting plan and quality assurance approach
Confirmed/finalized evaluation workplan
Evaluation Reports
The Consultant will document initial evaluation findings and recommendations in a Preliminary Findings Analysis and Report. A draft report will be shared with CIPE/Alliance staff for review and co-creation of recommendations prior to finalization. The report, and subsequent variations should include the following:
- Description of the project
- Description of the evaluation objectives
- Description of the evaluation methodology, including limitations
- Summary of the data collection tools, and data collected
- Summary of the results
- Conclusions from the analysis
- Lessons Learned
- Recommendations
A Final Evaluation Report should be submitted following a review of the Preliminary Evaluation Data Review and Findings Report presented prior to the Sensemaking Workshop with CIPE and Alliance M&E and Program staff, no later than September 2021. A follow up Evaluation (Preliminary) Report should incorporate Alliance feedback received during the Sensemaking Workshop and presented to Alliance Steering Group in November 2021. Any other additional data obtained through subsequent triangulation of evaluation data received by January 2022 can and should be incorporated into the Final Evaluation Report for a March 2022 deadline. Additional information on this evaluation timeline can be found in Section 8 below.
- Quarterly Progress Reports
The Consultant will be required to provide short, written narrative progress reports to Mrs. Dobosz and the MELT staff on the progress of the evaluation. Format and content will be discussed in the TOR to this Request for Proposal.
8) Evaluation timeline and schedule
Preliminary Schedule (May 2021 – March 2022)
Currently, due to the COVID pandemic and unanticipated, government related project hurdles, there are expected delays. As a result, the evaluation schedule presented below is conceptual only. A schedule for delivery will be agreed during the inception period and updated as needed, by agreement between the Project Manager and the Consultant. Should the evaluation plan require revision, it will be conducted in agreement with CIPE MELT staff.
| ACTIVITY | DURATION | NOTES |
| Pre-Field Work and Field Work | ||
| Inception | May-June 2021 | Starting with a kick-off call with the Alliance, the project teams and the head of the Colombia Center project working group, the main inception objectives are:
The results of these activities will be documented in an Inception Report. |
| Review and approval of the Inception Report | Mid-June 2021 | Alliance approval time (estimated 2 weeks) |
| Colombia pre-field work and ethical clearance
Professional certified survey translation to Spanish (Colombia) Recruitment of field work staff |
June 2021 | Prior to engaging in the field, initiate stakeholder contacts and scheduling of interviews, finalize tools, identify, and recruit local survey consultants.
Finalize informed consent plan. Recruit survey leaders and support in Colombia. Although many survey targets will speak English, response rates and quality will improve if a second language choice is offered. Language selection will be part of the informed consent process for each survey. |
| Colombia field work | July 2021 | Field work includes:
Field work will start and conclude with a briefing for the Alliance project manager. A starting briefing for DIAN is also recommended. |
| Colombia survey beta test | July 2021 | Testing involves roll-out of the sample survey and informed consent process, evaluation of the results and any questions or confusion that arose, refinements of the survey and update of the translations. |
| Colombia survey training | August 2021 | The Consultant should confirm best approach as outlined in section 2. |
| Colombia Center key informant interviews | August 2021 | The KIIs will run at the same time as the bilingual Spanish-English survey and cover key stakeholders:
|
| Colombia Center Survey | September 2021 | Survey design and methodology will be decided upon by the Consultant in coordination with the CIPE M&E team. Specifically, how target respondents will be selected, how the surveying will be conducted in an appropriate manner to achieve desired response rates, and finally how the Human Subject Research protocol will be followed. |
| Expert Analysis and Stakeholder Review | ||
| Preliminary Evaluation data review and findings report | September/ October 2021 | A preliminary data-for-analysis document will be produced covering the methodology, data collection, possible findings/lessons learned and any recommendations. The Consultant, with assistance from the MELT team, will review the collected data project and develop initial findings based on the data. |
| Sensemaking Workshop | October/
November 2021 |
The sensemaking workshop/process allows the evaluation team to present the initial findings and potential lessons and recommendations to the Alliance, discuss these, and agree on the meaning of the data collected. Inclusion in the sensemaking process should include the project team. Attendance by others is at the discretion of CIPE and the Alliance. |
| Evaluation (Preliminary Results) Report | November 2021 | The preliminary evaluation report will reflect the comments received from the evaluation team and project team regarding initial evaluation findings. The report will also be presented to the Alliance steering group. |
| Project Results Triangulation and Final Documentation | ||
| Results Triangulation | January 2022 | Following the preliminary report, a final draft report for the evaluation will be produced. The Consultant should triangulate any additional data at that time (such as TTLC) to supplement the Evaluation (preliminary results) report. |
| Alliance review and Approval | February 2022 | The Alliance will have an opportunity to review and comment on the evaluation report. Alliance approval time (estimated 2 weeks) |
| Final Evaluation report | March 2022 | The final report will reflect the comments received from the Alliance, and submitted as a completed report. |
9) Legal and ethical considerations
The evaluations will be conducted following the criteria and approaches for international development assistance as established by OECD/DAC Evaluation Quality Standards. The OECD/DAC Evaluation Quality Standards provide guidance on the development and delivery of evaluations and reports with the aim to improve the quality of development intervention evaluations and collaboration.
In line with those standards, the evaluation will be undertaken with integrity and honesty. Evaluation managers and evaluators will respect human rights and differences in culture, customs, religious beliefs, and practices of all stakeholders. The evaluator(s) will be mindful of gender roles, ethnicity, ability, age, sexual orientation, language, and other differences when designing and carrying out the evaluation. Evaluator(s) will consult local guidelines and practices on protecting human research subjects. Where necessary, the evaluator(s) will use the services of an Internal Review Board (IRB) to ensure that any risks faced by participants during, and as a result of, the evaluation are minimal.
All evaluation drafts and final outputs, including supporting documents and analytical reports ownership, rests jointly with the Alliance and the consultants, and will be consistent with the quality standards of evaluation ethics, data protection and confidentiality. Key stakeholders may also make appropriate use of the evaluation reports in line with the original purpose of the evaluation and with proper acknowledgment.
- Application Requirements
Candidates are requested to submit a response by June 09, 2021 to tradejobs@cipe.org and DDobosz@cipe.org, Sr. Manager for Monitoring, Evaluation and Learning, Trade Team:
Submissions should include:
- A technical proposal (max 5 pages) outlining the proposed evaluation methodology and plan in English.
- Personal CV of individual consultant or team representative, indicating all experience from similar projects, as well as the contact details (email and telephone number) of at least three (3) professional references; and
- At least one sample evaluation design and/or report(s) in English.
- Required Skills and Experience:
Education
- Bachelor’s Degree or equivalent in relevant field to the assignment; Master’s degree preferred;
Professional Experience
- Demonstrated at least 6 years of experience in program evaluation and/or mastery of data collection techniques and methods, interviews, focus groups, and quantitative and qualitative analysis;
- Experience evaluating trade facilitation or customs modernization projects preferred;
- Familiarity with donor-funded trade programs and the theory of change applied to trade programs.
Other knowledge and skills:
- Excellent proven oral and writing skills in English and Spanish, with excellent analysis and synthesis skills;
- Knowledge of the country context, development issues, trade practice and challenges;
- Availability to focus on the evaluation as required to meet deliverable deadlines within the agreed timeframe;
- Authorization to legally work in Colombia and the ability to travel within Colombia;
- Proactive, self-starter with strong attention to detail;
- Flexible and adaptable to changing environments.
ANNEX 1: PROJECT THEORY OF CHANGE
