Terms of reference – Consultant – Technical support for the design of an impact evaluation methodology 

Terms of reference – Consultant – Technical support for the design of an impact evaluation methodology 

Global Alliance for Trade Facilitation/ World Economic Forum

 

1.  Background
1.1 The Global Alliance for Trade Facilitation
The Global Alliance for Trade Facilitation is a public-private partnership for trade-led growth, supporting governments in developing and least-developed countries in implementing the World Trade Organization’s Trade Facilitation Agreement. Alliance projects cut through red tape and end costly delays at borders by bringing together governments and businesses of all sizes as equal partners to deliver targeted trade reforms.

By emphasizing digitisation and delivering other best practices, Alliance projects enable businesses to trade more easily thanks to streamlined and more predictable processes. Governments save time and resources by modernising trade procedures while still safeguarding their borders. Ultimately, Alliance projects boost trade competitiveness and business conditions, which are key drivers of inclusive economic growth and poverty reduction.

The Alliance is led by the Center for International Private Enterprise, the International Chamber of Commerce, and the World Economic Forum, in cooperation with Gesellschaft für Internationale Zusammenarbeit (GIZ). It is funded by the governments of the United States, Canada, Germany, and Denmark.

1.2 Alliance project in Madagascar
An advance ruling is the right of economic operators to benefit, on written request, from a transparent and formal process of the issuing of a legal opinion on the application of customs legislation prior to the import/export operation. This opinion is binding, uniformly applied by Customs and valid for a prescribed period.

At the time of customs clearance, as soon as the customs declaration is lodged, economic operators are likely to see elements of their declarations contested by front-line customs officers. Resolution of these disputes through the judicial system requires time and money. Disputes generally relate to factors such as the taxable amount and the application of external trade regulations, including tariff classification, origin or customs value. Bad decisions are also a source of corruption to obtain favourable treatment of goods, thus creating unfair competition. The erroneous or fraudulent issuance of advance rulings creates commercial risk unpredictability and legal uncertainty.

Today, economic operators in Madagascar are faced with inconsistency between the tariff classification notices of the various customs officials involved at the time of customs clearance. The opinion may vary from one office to another, from one customs officer to another and be different depending on the requesting operator. This creates conditions of unfair competition and opportunities for collusion between customs officials, freight forwarders and operators.

Advance rulings should enable operators, before the import/export operation, to be certain of the customs fees they will have to pay and the regulations to which they will be subject. Such predictability is essential for doing business under sustainable and competitive conditions. The principle is that these formal decisions by the central administration are binding for both the Customs Administration and the requesting economic operator. Indeed, according to the provisions of the Malagasy legislative framework, the ruling is binding on the economic operator who is obliged to use the advance ruling issued as part of its customs declaration at customs clearance.

As such, the project supports Customs to:

1. Respond to the needs of economic operators,
2. Issue advance rulings on tariff classification and customs origins,
3. Apply decisions uniformly at all points of entry/exit in Madagascar (ports and airports)

The project will do so through a series of activities which will solidify the use of advance rulings in Madagascar:

1. Review existing regulations related to the implementation of Advance Rulings in Madagascar and draft applicable texts, which will make it possible the effective use of advance rulings;
2. Design the procedures for the application, issuing and management of advance rulings (classification and customs origin) – draft the various forms required and elaborate a draft functional definition of the needs to design the data flows and the database in accordance with the organization, processes and procedures that will be implemented;
3. Assess the existing organization within the Customs Administration (at central and regional levels) and propose the adequate organization to set up in order to process and manage advance rulings;
4. Define and implement the digitization strategy according to the maturity of the IT systems used by Customs and economic operators and define and set-up a database centralizing the ARs issued in Madagascar.
5. Training customs officers in procedures for handling and managing advance rulings, and assist them in implementing the new organization and procedures.
6. A communication campaign to raise awareness and assistance to private companies in the use of advance rulings to promote the attractiveness of the program to the public, with a specific focus on SMEs and business-women associations. One of the strengths of the project is that it is inclusive of all categories of companies that import or export (microenterprises, SMEs and large national or international companies), including the Malagasy women entrepreneurs.

2. The assignment
The Alliance plans to undertake a rigorous (and independent) impact evaluation of the project under implementation in Madagascar. The evaluation aims for both accountability and learning purposes: it shall provide Alliance decision makers and stakeholders with robust information on the impacts of the project, and its results and lessons learned shall contribute to organizational learning, feed into the design and implementation of future Alliance projects and build knowledge on trade facilitation interventions. The Alliance plans to publish public versions of any completed impact evaluation reports.

A portion of the assignment (question 2 on section 3) will be to calculate the cost savings that result from the implementation of the project (cost of trade). This is expected to be done following a rigorous methodology but not necessarily following an experimental, or a quasi-experimental approach.

The impact evaluations will be conducted in two phases:

• Design phase: leveraging best practices on impact evaluation, the consultant will conduct an evaluability assessment, with the goal of identifying the counterfactuals (in this case, those who use advance rulings as oppose to those who don’t) as well as the most suitable impact evaluation methodology(ies), taking into account the context and nature of the intervention (e.g. literature review, availability of data, feasibility of data gathering, alignment to the project work plan). The consultant will also propose a methodology for the second question on costs of trade.
• Implementation phase: based on the result of the evaluability assessment. To implement the methodology(ies) developed in the previous phase, the consultant will conduct data gathering, analysis, reporting and dissemination of the key findings and conclusions.

The present TOR pertains to the procurement of an evaluation expert to undertake the design phase of the process. If the project is considered evaluable, the expert is not automatically awarded the contract for the second phase but will be considered as a candidate to conduct the implementation. In this sense, evaluation experts that are capable to rapidly and independently mobilizing an evaluation team may be given preference. If the project turns out not to be evaluable, the Alliance may direct the consultant to undertake the measurement of the time and cost expected results with a different methodology.

2.1 Key considerations
The following are some key considerations in designing the impact evaluation:

It is acknowledged that each evaluation occurs in a unique operating environment, at the same time the impact evaluation should reflect methodological rigor. These two factors must be considered in proposing the methodology.
The evaluation methodology should consider the contextual nuances of the country, including the capacity of key stakeholders, availability of data, among others.

In addition to data collected by the evaluation team, the evaluator should propose an evaluation methodology that utilizes data and information gathered by the Alliance partners and stakeholders reflecting the logical framework of the project and corresponding indicators (see Annex 1). Nonetheless, the evaluation model proposed should be true RCT (Randomized Control Trial), measuring the differences in outcomes between randomly assigned project participants and non-participants after the project took effect or a quasi-experimental approach, a difference-in-difference approach carried out using multivariate regression (accounting for other observable factors that might also affect the outcome).

For the cost of trade methodology, the consultant can rely on a non-experimental approach capturing the project impacts on the costs of trade for the private sector. In this effect, the consultants may consider adopting the Alliance’s Total Transport and Logistics Costs approach.

The methodology should capture the influence of the unique approach of the Alliance to trade facilitation, including the public-private cooperation in co-creation and co-implementation.

The audience of the impact evaluation includes multiple stakeholders: project implementers, partners and funders, and other countries who may benefit from a clear understating of project results and lessons learned.

3. (Preliminary) Evaluation Questions
The following is the preliminary list of questions. These questions will be refined and updated during the design phase, as described above.

1. For the impact evaluation questions, which should follow an RCT/quasi-experimental impact evaluation methodology:

a. Has the project led to a reduction in customs clearance time for importing goods for those consignments accompanied by an advance ruling?

1. 1. Has the project led to a time reduction for customs clearance processes for goods covered by an advanced ruling?
   2. Has the project led to more predictable customs clearance times for consignments covered by an advance ruling? What is the impact on predictability?

b. Has the project led to more consistency in tariff classification, and application of origin rules?

1. 1. For consignments covered by advance rulings
   2. Across different entry points

2. We expect the following questions to be answered with a non-experimental methodology:

a. Has the project led to a reduction in classification, valuation, origin amendments and disputes?

1. 1. Has a reduction on classification, valuation, origin amendments and disputes have an impact on the average time and cost of trade?

4. Key responsibilities and tasks
The impact evaluation expert will have the following responsibilities

• Impact evaluation design: Design a suitable methodology for a rigorous impact evaluation of the Alliance project in Madagascar, particularly taking a lead on this project in collaboration with the Alliance measurement team and advisory board (refer to section 6 for the management arrangement and responsibilities).
• Cost of trade measurement: Design a suitable rigorous methodology to measure the impact of the Alliance project on the cost of trade, particularly taking a lead on this project in collaboration with the Alliance measurement team.
• Quality assurance: Ensuring methodological rigor and high quality throughout the evaluation design process, assuring high quality deliverables and evaluation reports.

Based on a review of the literature and project documentation, and discussions with the project team and key stakeholders, the expert will conduct the following tasks:

a. Evaluability assessment. The expert will design and undertake an evaluability assessment to assess the feasibility of conducting a rigorous impact evaluation in this project. This assessment should include but is not limited to: criteria for assessing the evaluability the project, stakeholder mapping, and review of the coherence and logic of the interventions and the logframe of the projects and a thorough literature review of existing similar evaluations for grounding the methodological approach in current academic research. Given the important value of administrative data, this task will also include a data mapping exercise to assess the availability and accessibility of government and business data.

b. Specification of the evaluation questions. Following the evaluability assessment, the expert will review and update the impact evaluation questions. The purpose, scope and evaluation questions are to be used to determine the most appropriate methodology for each evaluation. In addition, the expert must examine and document the existence and quality of data (including gender-disaggregated data), especially government administrative data and/or business data of custom agents, availability of key informants and the timing of the evaluation, and identification of whether key stakeholders want/resist having their development intervention evaluated (i.e., the level of resistance to the evaluation and its reasons).

c. Impact evaluation methodology report. Based on the tasks above, the expert will propose an evaluation methodology. The methodology shall specify the evaluation objective, proposed evaluation design, the specific evaluation questions and quality criteria, and shall clearly specify the evaluator team’s tasks, deliverables and timelines. The methodology must be developed in line with the evaluation approach chosen and support the answering of evaluation questions using credible evidence. The expert must attach the following annexes to the methodology report:

• Evaluation Methodology, including power calculations.
• Sampling. For each sample the following must be defined and explained in detail: the purpose, objectives, universe/population, sampling criteria, sample design, sampling frame, sampling unit, sample size, sampling method(s), proposed sample and limitations.
• Proposed draft data collection tools (interviews, focus groups or other participatory methods, protocols, tabulations, etc.).
• Detailed timeline for the evaluation, including the detailed fieldwork plan.
• Proposed evaluation report outlines.

d. Cost of trade methodology: capturing the project impacts on the costs of trade for the private sector. The methodology for the cost of trade can be included in the impact evaluation methodology report as a section.

5. Methodology
This design phase of the impact evaluation will include, but is not limited to, the following methods:

• A literature and desk-based review of existing or emerging best practices about the subject matter and methodologies on measuring the effect of the subject matter; as well as a review of project documents.
• A preliminary analysis of the available data, making sure that we can collect all relevant information, with clear sources, to conduct the impact evaluation.
• Interviews (could be conducted virtually) with relevant Alliance staff, including the project team, M&E team, management team and donors.
• Scoping interviews (could be conducted virtually) with relevant government agencies and businesses involved in the projects in to assess the availability and accessibility of administrative data as well as the feasibility of the impact evaluation methodology.
• A workshop to present and discuss the proposed methodology to the Alliance team and advisory group.
• Survey interviews with relevant stakeholders and beneficiaries of the project.

6. Deliverables

• Detailed work plan for the design phase.
• Evaluability assessment
• Specification of the evaluation questions.
• Detailed impact evaluation methodology report for the project in Madagascar, incorporating the comments and requests raised by the Alliance team and advisory group.

7. Management Arrangement
This section provides an overview on how this engagement will be managed.

Impact evaluation team (IET): will supervise and provide comments and inputs to the draft deliverables and evaluation methodology. It will manage everyday tasks and project needs. It is composed of:

• Alliance Monitoring and Evaluation Lead – as the manager of the impact evaluation.
• Consultants – as the developers and implementers of the experimental impact evaluation approach.
• Metrics and Benchmarking Lead – as technical expertise support of the evaluation.
• Management Team Champion – as support to the evaluation management.

Advisory group (AG): integral part of the team. They will all be involved in different capacities depending on availability. It will consist of the impact evaluation team plus:

• Project leads
• Communications officer
• Knowledge and Best Practice Specialist
• Private Sector Working Group member interested in the impact evaluation and/or championing the project
• External scholars – e.g. UNCTAD, WTO, WCO, academic, researchers.

And they will hold the following responsibilities:

– Select the evaluation expert
– Orient the evaluation expert about the tasks, expectations, timeline and the evaluation management arrangement
– Provide inputs/ comments to the work plan, including the appropriateness of the process, methodology, list of reference materials and key informants
– Review the evaluation methodology
– Attend regular meetings (e.g. bi-weekly video calls)
– Brief the Alliance Steering Group on evaluation progress

In support to this, the Alliance Project Leads will provide the following:
– Compile and share relevant references and resource materials about the Alliance and the projects
– Consolidate and provide the list of Alliance partners (public and private sector)
– Coordinate the scheduling of the virtual meetings with relevant staff and the project team
– Provide administrative coordination securing the necessary administrative support for the meetings with the relevant government agencies and businesses; and give the evaluation expert the required background information. They will not be involved nor attend any meetings.

7. Legal and Ethical Considerations
The evaluations will be conducted following the criteria and approaches for international development assistance as established by OECD/DAC Evaluation Quality Standards. The OECD/DAC Evaluation Quality Standards provide guidance on the development and delivery of evaluations and reports with the aim to improve the quality of development intervention evaluations and improved collaboration.

All evaluation drafts and final outputs, including supporting documents and analytical reports ownership, rests jointly with the Alliance and the consultants, consistent with the quality standards of evaluation ethics, data protection and confidentiality. Key stakeholders may also make appropriate use of the evaluation reports in line with the original purpose of the evaluation and with proper acknowledgment.

8. Selection process
Candidates are requested to submit a response by November 7th, 2021 to Mrs. Macarena Torres Rossel (macarena.torresrossel@weforum.org), Monitoring and Evaluation Lead, Global Alliance for Trade Facilitation. Submissions should include:

a. A technical proposal (max 6 pages) outlining the proposed methodology for the design phase.
b. An up-to-date Curriculum Vitae (CVs) and/or company profile.
c. Financial Proposal (in USD)

• Personnel Cost (proposed daily rate and number of days)
• Reimbursable travel and administrative expenses

d. Sample impact evaluation design and report(s)

The selection process will utilize the following scoring matrix:

Technical score (max 65)
The expert should have conducted at least ONE impact evaluation of a private sector development project or project in an international context. Experience in evaluating border management or customs modernization projects is preferred.	10
The expert must have at least 8 years’ demonstrated experience on designing and management of impact evaluations. Experience in Africa is preferred.	15
The technical proposal should clearly enumerate the steps in undertaking the design of the impact evaluation, including the process of conducting a sound and rigorous evaluability assessment; developing the evaluation matrix and instruments to be used in sampling, data collection, analysis; as well as the data collection plan.	30
The expert’s proven ability to independently and rapidly mobilize an evaluation team for phase 2, if one or both projects are considered evaluable.	10
Financial score (max 35)
Financial proposal	35

9. Expertise

• Master’s degree in Economics, Statistics, International Trade or another related subject.
• Demonstrated professional experience (minimum 8 years) in the field of impact evaluation. Ideally, experience in managing quantitative evaluations.
• Academic training and/or practical experience in quantitative and qualitative methods in impact evaluation; familiarity with evaluation designs such as quasi-experimental design and randomized control trials; expertise in econometrics.
• Familiarity with donor funded trade programs and the theory of change of trade programs.
• Ability to travel overseas for short-term assignments.
• Demonstrated ability to write clearly.
• Proactive, self-starting, and strong attention to detail.
• Excellent verbal communication and writing skills in English.
•  Demonstrated ability to lead a multi-cultural team and working environment.

 

Annex 1: Logical Framework
Results statement: Once implemented, the advance ruling mechanism in Madagascar will enable importers to request a legally binding opinion on the application of customs regulation prior to the import process. The project foresees to publish issued rulings under consideration of confidentiality as well as a mechanism to seek a review of the ruling by the trader.
Problem Statement: Today, economic operators in Madagascar are faced with inconsistency between the tariff classification notices of the various customs officials involved at the time of customs clearance.

MADAGASCAR THEORY OF CHANGE

MADAGASCAR LOGFRAME

IMPACT	Impact Indicator 1	Assumptions
Time related cost savings for the private sector due to a reduction of disputes on tariff classification and customs origin	Average time required for the customs clearance process for traders that request advance rulings	AR is a powerful tool as long as customs top management is willing to set up the appropriate organisation and tools (a database to start with) and, even more important, to train staff to make sure that rulings will be applied evenly at all entry points across Madagascar. Also, the positive attitude of private companies towards AR is key. It’s got to be designed in line with their expectations. Their massive use of AR will be a strong sign that the project has reinforced trust between customs & private businesses.
	Impact Indicator 2
	Number of disputes over tariff classification and customs origin during import/export clearance processes at the point of entries The indicator should disaggregate product categories, SME and companies affiliated with a business women’s association
	Impact Indicator 3
	Time and cost of resolving classification, valuation or origin disputes The indicator should disaggregate product categories, SME and companies affiliated with a business women’s association
Increased transparency and predictability of tariff classification and origin rulings across all entry points	Impact Indicator 4
	More consistency in tariff classification and origin rulings across all entry points (measures to compare between entry point)The indicator should disaggregate product categories The indicator should disaggregate product categories
INTERMEDIATE OUTCOME	Outcome Indicator 1.1	Assumptions
Advance ruling system is in place and fully operational	Number of total requests issued by private companies The indicator should disaggregate product categories, SME and companies affiliated with a business-women’s association	Even-though a fully digital solution is not necessary to implement the project (though it remains an objective in the medium run), it will be up to the project implementer to design & test & deploy a database to store and display advance rulings when they are validated. As stated in “output 3” private businesses sensitisation will also be very important to induce them in requesting advance rulings
	Outcome Indicator 1.2
	Number of advance rulings publicly published by customs in the data base The indicator should disaggregate product categories
OUTPUT 1	Output Indicator 1	Assumptions
Law and regulations allow the use of Advance Rulings	Approval of the text for the effective use of the Advance Rulings	As the code is already changed to allow AR. It is assumed that the approval of the text will be easy.
OUTPUT 2	Output Indicator 2	Assumptions
Organization and procedures are set to process and manage Advance Rulings within Directorate General of Customs both at central and regional levels	Number of customs agents dedicated and trained to process advance rulings (and recruited if needs be) The indicator should disaggregate central and regional postings, also agent’s seniority	Training and change management are key factors of success which relies a great deal on the drive of customs CEO & top managers. The former has repeatedly manifested his support to the project.
OUTPUT 3	Output Indicator 3	Assumptions
Communication and information towards private businesses to raise awareness and promote the use of Advance Rulings is carried out	Number of private companies trained to use advance ruling (consult the database & issue a request & know their rights) The indicator should disaggregate SME and companies affiliated with a businesswomen’s association	The support of pilot private companies is key to make sure that the project will be aligned with private sectors expectations. They will attend the steering committee and workings groups. It will be up to the project implementer to design & implement a communication strategy towards private businesses
OUTPUT 4	Output Indicator 4	Assumptions
Information system is integrated with ASYCUDA	Integration with the Custom’s risk analysis system (ASYCUDA) is done TBD – to be clarified with customs and the service provider	Two options have been identified for the integration, either through the Single Window (GasyNet), or directly with the customs management system (Asycuda). The best option is to be clarified with customs, Gasynet and the project implementer.