Consultant – Zambia & Malawi – Impact Evaluation
Terms of Reference: Impact Evaluation Expert
Technical support for the design of impact evaluation methodology(ies) for two trade facilitation projects in Zambia and Malawi
1 Background
1.1 The Global Alliance for Trade Facilitation
The World Trade Organization’s Trade Facilitation Agreement (TFA) provides a unique opportunity to enable trade as a central driver of growth. According to the Organization for Economic Cooperation and Development (OECD), the TFA could reduce total trade costs by more than 14% for low-income countries and more than 13% for upper-middle-income nations by streamlining the flow of trade across borders.[1] Implementation of the TFA and associated reduction in trade time and costs could increase developing countries’ exports by up to US$730 billion per year.[2] TFA implementation can also contribute to a range of other development objectives as set out in the SDGs.[3]
To support the implementation of the TFA, the Global Alliance for Trade Facilitation (the Alliance or GATF) brings together governments and businesses as equal partners to identify opportunities to address delays and unnecessary red-tape at borders, and design and deploy targeted reforms that deliver commercially quantifiable results.[4]
The Alliance is a public-private partnership dedicated to international trade facilitation. It is a not-for-profit venture jointly led by the International Chamber of Commerce (ICC), the World Economic Forum (WEF) and the Center for International Private Enterprise (CIPE) – in cooperation with Deutsche Gesellschaft für Internationale Zusammenarbeit (GIZ). The Alliance was launched at the World Trade Organization’s 10th Ministerial Conference in December 2015 and is supported by the governments of the United States, Canada, Germany, Denmark, the United Kingdom and Australia.
1.2 Alliance projects in Malawi and Zambia
Customs clearing agents are a vital cog in the machine of international trade, handling all the documentation needed to clear goods arriving and leaving a country. If knowledgeable and well-trained, these agents can ensure that the right documentation is properly submitted, that traders pay the correct fees and that goods keep moving smoothly and efficiently across borders.
In Malawi and Zambia, as in other countries, customs clearing agents are expected to perform this key function in cross-border trade. Yet these two countries do not have properly structured licensing frameworks and accreditation programs for customs clearing agents, and individuals are able to obtain a license to operate without demonstrating they possess the skills required.
As a result, many agents lack the knowledge of how to clear goods in compliance with regulations. This can lead to errors in documentation, which, in turn, leads to mistrust between traders and customs, increases the frequency of inspections at borders and, ultimately, raises the time, unpredictability and cost of trade, and damaging the countries’ international competitiveness.
Through the Alliance’s process of co-creation, the governments, businesses and customs clearing agents in both countries agreed that proper training and regulation of customs clearing agents could help reduce the time and cost of trading. Under the leadership of CIPE, the Alliance has designed and is implementing two separate projects, in partnership with relevant government agencies, customs clearing agents’ associations, and local and international businesses in Malawi[5] and Zambia[6] to modernize the countries’ custom broker licensing frameworks.
The new frameworks will include a training course combining practical, hands-on training with e-learning, and an exam through which prospective agents can demonstrate their competence. The course and exam will be designed to maximize accessibility for all, particularly for women who are often more time-constrained than their male counterparts, with the goal of encouraging women in both countries to become licensed customs clearing agents.
As a result of the two projects, Malawi and Zambia will be equipped with a fully trained and professional pool of customs clearing agents who are able to facilitate faster clearance and transit of goods across the countries’ borders. With this, it is expected that the governments will experience higher levels of compliance, while businesses will encounter a level playing field where goods are subject to proper classification, valuation and origin regulations. It is also expected that there will be a larger number of and diversity of traders along such dimensions as size, gender identity of owner, and other variables.
2 The assignment
The Alliance plans to undertake two rigorous (and independent) impact evaluations of the projects under implementation in Malawi and Zambia. The evaluations aim for both accountability and learning purposes: they will provide Alliance decision makers and stakeholders with robust information on the impacts of the two projects. The results and lessons learned will contribute to organizational learning, feed into the design and implementation of future Alliance projects, and build knowledge on trade facilitation interventions. The Alliance plans to publish public versions of any completed impact evaluation reports.
The impact evaluations will be conducted in two phases:
● Design Phase: Leveraging best practices on impact evaluation, the Alliance will conduct an evaluability assessment, with the goal of identifying possible counterfactuals, as well as the most suitable impact evaluation methodology(ies), taking into account the context and nature of the interventions (e.g. availability of data, feasibility of data gathering, alignment to the project work plan).
● Implementation phase: To implement the methodology(ies) developed in the Design Phase, the Alliance will mobilize the appropriate evaluation team(s) to conduct data gathering, analysis, reporting and dissemination of key findings and conclusions.
This TOR pertains to the procurement of an evaluation expert to undertake the design phase of the process. If one or both projects are considered evaluable, the expert may also be commissioned to lead the activities of the Implementation Phase. In this sense, evaluation experts that are capable of rapidly and independently mobilizing an evaluation team may be given preference. If one or both of the projects are not deemed to be evaluable, the Alliance may direct the consultant to design an impact evaluation of one or more separate, future Alliance project(s).
2.1 Key considerations
The following are some key considerations in designing the impact evaluations:
It is acknowledged that each evaluation occurs in a unique operating environment, at the same time the impact evaluations should reflect methodological rigor. These two factors must be considered in proposing the methodology(ies).
Even though both projects address the same topic, the two impact evaluations are not meant to be compared. The two projects are included in this TOR for the purposes of maximizing resources, time and expertise. Each evaluation methodology should consider the contextual nuances of each country, including the capacity of key stakeholders, availability of data, among others.
In addition to data collected by the evaluation team, the evaluator should propose an evaluation methodology(ies) that utilizes data and information gathered by the Alliance partners and stakeholders reflecting the logical framework of the project and corresponding indicators (see Annex 1).
The methodology(ies) should capture the influence of the unique approach of the Alliance to trade facilitation, including the public-private cooperation in co-creation and co-implementation.
The audience of the impact evaluations includes multiple stakeholders: project implementers, partners and funders, and other countries who may benefit from a clear understating of project results and lessons learned.
3 (Preliminary) Evaluation Questions
The following is the preliminary list of impact evaluation questions. These questions will be refined and updated during the design phase, as described above.
● To what extent did the formalization of the customs bond system affect the efficiency of goods cleared at the border in Malawi/Zambia?
● How effective was the capacity building and the licensure initiative of the Alliance project in reducing inaccuracies in custom’s processes? And if identified, to what extent have these contributed to improved trust and use of customs brokers?
● To what extent did the Alliance project outcomes in Malawi/Zambia contribute to reducing the time and cost of international trade for Malawian/Zambian businesses?
● To what extent has the project contributed directly or indirectly towards inclusivity (e.g. women entrepreneurs; rural and smaller traders) in trade?
4 Key responsibilities and tasks
The impact evaluation expert will have the following responsibilities
● Impact evaluation design: Provide technical guidance on designing a suitable methodology(ies) for a rigorous impact evaluation of the Alliance projects in Malawi and Zambia, in collaboration with the Alliance evaluation team (refer to section 6 for the management arrangement and responsibilities).
● Quality assurance: Ensuring methodological rigor and high quality throughout the evaluation design process, as well as assuring high quality deliverables and evaluation reports.
Based on a review of the literature and project documentation, and discussions with the project team and key stakeholders, the expert will conduct the following:
- Evaluability assessment. The expert will design and undertake an evaluability assessment. This assessment should include, but is not limited to: criteria for assessing the evaluability of each project, stakeholder mapping, and review of the coherence and logic of the interventions and the logframe of the projects. Given the important value of administrative data, this task will also include a data mapping exercise to assess the availability and accessibility of government and business data.
- Specification of the evaluation questions. Following the evaluability assessment, the expert will review and update the impact evaluation questions. The purpose, scope and evaluation questions are to be used to determine the most appropriate methodology for each evaluation. In addition, the expert must examine and document the existence and quality of data (including gender-disaggregated data), especially government administrative data and/or business data of custom agents, availability of key informants and the timing of the evaluation, and the extent to which key stakeholders are accepting of or resistant to the exercise and its reasons.
- Impact evaluation methodology report. Based on the tasks above, the expert will propose an evaluation methodology(ies). The methodology(ies) shall specify the evaluation objective, proposed evaluation design, the specific evaluation questions and quality criteria, and will clearly specify the evaluator team’s tasks, deliverables and timelines. The methodology(ies) must be developed in line with the evaluation approach chosen and support the answering of evaluation questions using credible evidence. The expert must attach the following annexes to the methodology report:
- Evaluation Methodology
- Sampling. For each sample the following must be defined and explained in detail: the purpose, objectives, universe/population, sampling criteria, sample design, sampling frame, sampling unit, sample size, sampling method(s), proposed sample and limitations.
- Proposed draft data collection tools (interviews, focus groups or other participatory methods, protocols, tabulations, etc.).
- Detailed timeline for the evaluation, including the detailed fieldwork plan.
- Proposed evaluation report outlines.
5 Methodology
This design phase of the impact evaluation will include the following methods:
● A literature and desk-based review of existing or emerging best practices about the subject matter and methodologies on measuring the effect of the subject matter; as well as a review of project documents.
● Virtual interviews with relevant Alliance staff, including the project team, M&E team, management team and donors.
● Virtual scoping interviews with relevant government agencies and businesses involved in the projects in Malawi and Zambia to assess the availability and accessibility of administrative data, as well as the feasibility of the impact evaluation methodology(ies).
● A workshop (via videoconference or in person at CIPE- Washington DC, as the case may be) to present and discuss the proposed methodology(ies) to the Alliance team.
6 Deliverables
● Detailed work plan for the design phase.
● Detailed impact evaluation methodology(ies) for the projects in implementation in Malawi and Zambia, incorporating the comments and requests raised by the Alliance team and donors.
7 Management Arrangement
This section provides an overview on how this engagement will be managed:
The Steering Group of the Alliance will provide comments and inputs to the draft evaluation methodology.
An Evaluation Taskforce, chaired by the Alliance Director and composed of the Alliance M&E Lead, Knowledge Lead and Sr. M&E Manager from CIPE will be the consultant’s main counterparts, and will hold the following responsibilities:
– Selection of the evaluation expert
– Orientation of the evaluation expert about the tasks, expectations, timeline and evaluation management arrangement
– Provide inputs/ comments to the work plan, including the appropriateness of the process, methodology, list of reference materials and key informants
– Review the evaluation methodology
– Attend regular meetings (e.g. bi-weekly video calls)
– Brief the Alliance Steering Group on evaluation progress
In support to this, the Alliance Project Leads will provide the following:
– Compile and share relevant references and resource materials about the Alliance and the projects
– Consolidate and provide the list of Alliance partners (public and private sector)
– Coordinate the scheduling of the virtual meetings with relevant staff and the project team
– Provide coordination securing the necessary administrative support for the virtual meetings with the relevant government agencies and businesses; and provide the evaluation expert with the required background information BUT will not be involved nor attend any meetings.
8 Legal and Ethical Considerations
The evaluations will be conducted following the criteria and approaches for international development assistance as established by OECD/DAC Evaluation Quality Standards. The OECD/DAC Evaluation Quality Standards provide guidance on the development and delivery of evaluations and reports with the aim to improve the quality of development intervention evaluations and collaboration.
All evaluation drafts and final outputs, including supporting documents and analytical reports ownership, rests jointly with the Alliance and the consultants, consistent with the quality standards of evaluation ethics, data protection and confidentiality. Key stakeholders may also make appropriate use of the evaluation reports in line with the original purpose of the evaluation and with proper acknowledgment.
9 Selection process
Candidates are requested to submit a response by April 17, 2020 to Mrs. Barbara Ramos (Barbara.Ramos@weforum.org), Team and Knowledge Lead, Global Alliance for Trade Facilitation. Submissions should include:
- A technical proposal (max 6 pages) outlining the proposed methodology for the design phase.
- An up-to-date Curriculum Vitae (CVs) and/or company profile.
- Financial Proposal (in USD)
- Personnel Cost (proposed daily rate and number of days)
- Reimbursable travel and administrative expenses
- Sample impact evaluation design and report(s)
The selection process will utilize the following scoring matrix:
| Technical score (max 65) | |
| The expert should have conducted at least ONE impact evaluation of a private sector development project or project in an international context. Experience in evaluating border management or customs modernization projects is preferred. | 10 |
| The expert must have at least 8 years’ demonstrated experience on designing and management of impact evaluations. Experience in Africa is preferred. | 15 |
| The technical proposal should clearly enumerate the steps in undertaking the design of the impact evaluation, including the process of conducting a sound and rigorous evaluability assessment; developing the evaluation matrix and instruments to be used in sampling, data collection, analysis; as well as the data collection plan. | 30 |
| The expert’s proven ability to independently and rapidly mobilize an evaluation team for phase 2, if one or both projects are considered evaluable. | 10 |
| Financial score (max 35) | |
| Financial proposal | 35 |
10 Expertise
● Master’s degree in Economics, Statistics, International Trade or another related subject
● Demonstrated experience (minimum 8 years) in evaluation, especially experience in designing and managing quantitative and/or impact evaluations
● Academic training and/or practical experience in quantitative and qualitative methods in impact evaluation; familiarity with evaluation designs such as quasi-experimental design and randomized control trials, including expertise in econometrics
● Familiarity with donor funded trade programs and the theory of change of trade programs
● Ability to travel overseas for short-term assignments
● Demonstrated ability to write clearly
● Proactive, self-starting, and strong attention to detail
● Excellent verbal communication and writing skills in English
● Demonstrated ability to lead a multi-cultural team and working environment
[1] OECD (2015). Implementation of the WTO Trade Facilitation Agreement: The Potential Impact on Trade Costs
[2] WTO (2015). World Trade Report: Speeding up trade: benefits and challenges of implementing the WTO Trade Facilitation Agreement
[3] GATF and the SDGs https://www.tradefacilitation.org/helping-achieve-un-sustainable-development-goals/
[4] GATF https://www.tradefacilitation.org/who-we-are/about-us/
[5] Malawi project description : https://www.tradefacilitation.org/project/modernising-licensing-for-customs-clearing-agents-2/
[6] Zambia project description : https://www.tradefacilitation.org/project/modernising-licensing-for-customs-clearing-agents/