Consultant for Project Evaluation – Modernization of Import Licensing Procedures
The Global Alliance for Trade Facilitation (GATF) is a public-private organization created to support the effective implementation of the World Trade Organization (WTO) Trade Facilitation Agreement (TFA), through partnerships between governments and the private sector.
Funding for GATF projects comes from the governments of the United States, United Kingdom, Canada, Germany, Australia and Denmark. The GATF secretariat is composed of the Center for International Private Enterprise (CIPE), the International Chamber of Commerce (ICC) and the World Economic Forum (WEF), in cooperation with the Gesellschaft für Internationale Zusammenarbeit (GIZ).
As part of this Alliance, CIPE is responsible for implementing a project to automate and streamline the import licensing procedures of the National Institute of Metrology, Quality and Technology of Brazil (INMETRO).
INMETRO is Brazil’s main consumer product safety regulatory agency, whose role in international trade involves issuing import licenses for a wide array of consumer and durable goods, from automotive parts, electronics, to children’s toys. The agency has seen over 35% annual growth in the demand for such licenses (therefore doubling approximately every two years), as Brazil’s economy becomes more integrated into the global supply chains. This growth has understandably overtaxed INMETRO, resulting in importing delays and disruption to border operations. In efforts to address identified needs, the project’s objective is to address application backlogs and license irregularities, in addition to streamlining procedures. The INMETRO project includes the reengineering of licensing procedures, the modernization of the regulatory framework for import licenses and the introduction of technology to refine and modernize the process.
CIPE, within the context of the Global Alliance, is seeking a consultant to conduct an evaluation on the effectiveness of the INMETRO project. The evaluation should reflect accountability and knowledge-building purposes, as CIPE and the Alliance intends to share evaluation results and lessons learned with decision makers and stakeholders.
This information will contribute to organizational learning, feed into the design and implementation of future projects and improve the understanding of trade facilitation interventions. Evaluation results may be published in whole or in part by CIPE and/or the Alliance.
The assignment is deliverable based and requires a two-phase approach: 1) research and analysis, and 2) evaluation. Data analysis and initial results are required prior to the conclusion of the project (no later than mid-October 2020), however the final evaluation report is not required until shortly approximately March 2021. With such a gap, the Consultant will be expected to confirm earlier produced data is still valid, and any relevant new data or information is added to the final report. Additional details are found below.
The assignment requires a suitable evaluation methodology design and implementation plan to qualify and quantify project results. Both anticipated and unanticipated results should be addressed.
PRIOR to the final evaluation report, the Consultant is tasked in organizing available quantitative data and reporting on specific project outcomes. Research specifics are included in Section 4: Deliverables. The Consultant will research, analyze and produce initial data analysis reports on the following project results:
Effectiveness of the interoperability between Orquestra (current IT system used by INMETRO to approve import licenses), and SISCOMEX (existing single window system used by Receita – Brazil Customs).
Effectiveness of INMETRO Ordinance 260 which allows importers to request import licenses after goods have been shipped from the exporting country.
Effectiveness of the new license management system and its integration with Portal Unico (Brazil’s new single window for trade).
Effectiveness of the AEO INMETRO program.
3. Evaluation Questions
The evaluation should answer the following questions:
How has the introduction of the risk management strategy reduced the backlog of import licenses?
Has the interoperability of INMETRO and SISCOMEX IT systems improved the approval process for licenses? Has it led to a reduction on time in the issuing of INMETRO import licenses? Did the Brazil project lead to regulatory efforts and legal ordinances that promote private sector trade?
Has the project contributed to the reduction of time and cost of international trade for Brazilian businesses? To what extent has the project contributed to reducing the overall clearance time of goods that require an INMETRO import license?
To what extent has the project benefited the private sector? Have the activities delivered by the project contributed to a better working relation between the private and public sector?
To what extent can the results of the project continue over time? What is the likelihood of a continuation of those effects in the future? What kind of changes are needed to ensure continuation of these effects?
4. Key considerations
The following are some key considerations in designing and conducting the evaluation:
Acknowledgment that each evaluation occurs in a unique operating environment, and that evaluations should reflect methodological rigor. These two factors must be considered in proposing the methodology for this TOR.
In addition to data collected by the Alliance evaluation team, the evaluator should propose an evaluation methodology that utilizes data and information from Alliance partners and stakeholders.
Where possible, the methodology and/or evaluation results should capture the influence of the unique approach of the Alliance to trade facilitation, including the public-private cooperation in co-creation and co-implementation.
The audience of the evaluation includes multiple stakeholders: project implementers, partners, private sector, and funders, and other countries who may benefit from a clear understating of project results and lessons learned.
5. Key Tasks and Responsibilities
Based on a review of the literature and project documentation, and discussions with the project team and key stakeholders, the expert will conduct the following:
1) Inception Report. The expert will provide a suitable evaluation methodology and workplan outline. Cost-effective, quasi-experimental designs should be considered. The timing of the evaluation and research should be in line with the Deliverables schedule in Section 7.
The inception report should include the following:
- Evaluation Methodology: The consultant should adequately describe the way that the chosen data collection methods, data sources, sampling, who are the target groups, and indicators which will support the evaluation questions
- Where sampling is recommended, the following must be defined and explained in detail: the purpose, objectives, universe/population, sampling criteria, sample design, sampling frame, sampling unit, sample size, sampling method(s), proposed sample and limitations.
- Detailed timeline for the evaluation, with a schedule of deliverables, and any anticipated evaluation challenges. A detailed (virtual) fieldwork plan should also be included if necessary
- Proposed evaluation report outline.
2) Confirmation of evaluation questions and data availability. The expert will review and update the evaluation questions (if necessary), taking into account data availability and accessibility. The purpose, scope and evaluation questions should inform the most appropriate methodology for the evaluation. In addition, the expert must examine and document the existence and quality of data (including gender-disaggregated data), especially government administrative data and/or business data, availability of key informants and the timing of the evaluation.
3) Data Collection, Analysis and Data Analysis Reports. The expert will develop data collection tools, identify the target groups, collect data through quantitative and qualitative methods and then synthesize and analyze the available data and produce a draft data analysis report addressing the specific project outcomes, including findings and validation. Consultant will receive feedback on tools and data collection throughout the evaluation.
4) Final Evaluation Report. The expert will compile a draft Evaluation Report based on the Data Analysis Reports. Following review, the Consultant will incorporate comments made by the CIPE program and M&E team for a final Evaluation Report. CIPE may also share the draft report with Alliance partners for additional feedback prior to finalization.
6. Suggested data sources
This program evaluation should include, but is not limited to, the following evaluation data sources:
- A literature and desk-based review of existing or emerging best practices about the subject matter and methodologies on measuring the effect of the subject matter; as well as a review of project documents.
- Virtual interviews and/or focus groups with relevant CIPE staff, including the project team, M&E team as well as private and public sector stakeholders.
- Virtual or in-person interviews with relevant government agencies, businesses and other stakeholders involved in the Brazil project, in order to acquire available administrative data.
- Additional verified data and documentation that may support the proposed evaluation methodology and questions.
The Evaluation should specifically utilize qualitative and quantitative data on the following project outcomes in order to address the evaluation questions:
- Implementation of the risk management strategy
- Impacts of the interoperability of new licensing systems implemented under the project, specifically SISCOMEX, Portal Unico, and Orquestra
- Integration of INMETRO’s operations under Portal Unico
- The effects of Ordinance 260 in importers and import processes
- Additional regulatory efforts and legal ordinances that might result due to project activities/outputs
- Potential reduction in the time and cost of border processes as a result of project activities
|1||Inception Report: Report detailing the updated project schedule with all key activities and deliverables.||7 calendar days after contract signature||Electronic (MS Word) in English and Portuguese.
|2||Report on the effectiveness (time and cost reductions) of the interoperability between Orquestra and Siscomex.||45 calendar days after contract signature||Electronic (MS Word) and presentation to key stakeholders in English and Portuguese.|
|3||Report on the effectiveness (time and cost reductions) of Ordinance 260.||45 calendar days after contract signature||Electronic (MS Word) and presentation to key stakeholders in English and Portuguese.|
|4||Report on the effectiveness (time and cost reductions) of the new licensing system and its integration with Portal Unico.||7 months after contract signature||Electronic (MS Word) and presentation to key stakeholders in English and Portuguese.|
|5||Report on the effectiveness of the INMETRO AEO program||8 months after contract signature||Electronic (MS Word) and presentation to key stakeholders in English and Portuguese.|
|6||Final Evaluation Report||9 months after contract signature||Electronic (MS Word) and presentation to key stakeholders in English and Portuguese.|
8. Legal and Ethical Considerations
The evaluations will be conducted following the criteria and approaches for international development assistance as established by OECD/DAC Evaluation Quality Standards. The OECD/DAC Evaluation Quality Standards provide guidance on the development and delivery of evaluations and reports with the aim to improve the quality of development intervention evaluations and collaboration.
All evaluation drafts and final outputs, including supporting documents and analytical reports ownership, rests jointly with the Alliance and the consultants, consistent with the quality standards of evaluation ethics, data protection and confidentiality. Key stakeholders may also make appropriate use of the evaluation reports in line with the original purpose of the evaluation and with proper acknowledgment.
9. Characteristics of the Consultancy
|Place of work||Consultant’s office, with regular meetings at INMETRO headquarters in the state of Rio de Janeiro and other key stakeholders.|
|Procurement methodology||Selection based on the Consultant’s qualifications|
10. Proposed Payment Schedule
- 10% against the submission and approval of the Inception Report.
- 30% against the submission and approval of deliverables 2 and 3.
- 35% against the submission and approval of deliverable 4.
- 10% against the submission and approval of deliverable 5.
- 15% against the submission and approval of the Final Evaluation Report.
11. Application Requirements
Candidates are requested to submit a response by Thursday August 20, 2020 at 5pm Washington DC time to email@example.com and DDobosz@cipe.org, Sr. Manager for Monitoring, Evaluation and Learning, Trade Team.
11.1. Submissions should include:
- A technical proposal (max 5 pages) outlining the proposed evaluation methodology and plan in English.
- A current Curriculum Vitae (CVs) and/or company profile in English.
- At least two sample evaluation design and/or report(s) preferably in English. If the sample evaluations are presented in Portuguese, a one-page executive summary in English of said evaluations is required.
11.2. Expertise and qualifications
- Higher level university degree in Economics, Business, Statistics, Engineering, Measurement and Evaluation, International Trade or another related subject.
- Demonstrated experience (minimum 6 years) in project evaluation, especially experience in designing and managing quantitative data collection and analysis.
- Experience evaluating trade facilitation or customs modernization projects preferred.
- Academic training and/or practical experience using quantitative and qualitative evaluation methods; familiarity with quasi-experimental evaluation design. Expertise in econometrics preferred but not required.
- Familiarity with donor funded trade programs and the theory of change of trade programs.
- Authorization to legally work in Brazil and the ability to travel within Brazil.
- Proactive, self-starter with strong attention to detail.
- Excellent verbal communication and writing skills in English and Portuguese.
- Demonstrated ability to work with a multi-cultural team and working environment.
Annex 1: Logical Framework
Problem Statement: INMETRO’s difficulties to provide import licenses in a timely manner due to increased demand, limited capacity, outdated processes and IT systems has occasionally led to considerable disruption to border operations. Recent efforts to expedite license approvals and reduce disruption circumvent proper risk assessment practices, potentially undermining legitimate traders and endangering Brazilian consumers.
Results statement: A new streamlined and simplified license approval process will reduce the transaction costs for the private sector while ensuring compliance with INMETRO’s policies and safety standards.
Annex 2: Project Proposal
Annex 3: Key IT systems involved in the approval of INMETRO licenses
SISCOMEX: Customs IT management system. Traders request the license in Siscomex and INMETRO analysts must log in to SISCOMEX to formalize the approval of every license application.
Orquestra: Off-the-shelf business process management software that has been customized by INMETRO to analyze and issue import licenses. All importers requesting an INMETRO license must also register in Orquestra and submit all required documents. After an INMETRO analyst looks at the application, a public servant enters the relevant data into SISCOMEX (Customs current IT management system) to approve the license.
ProdCert: Database that holds product certification information such as item manufacturer, model, material, size, etc. Because most of INMETRO’s technical regulations have conformity assessment schemes associated to them, the information in this system is crucial to approve licenses. Basically, if the company information in ProdCert is in order and the product is certified by an accredited certification body, the license will be issued. ProdCert is currently based on free-text data and, as such, cannot effectively communicate with a cutting-edge risk management system (RMS).
Importers Registry Database (IRD): INMETRO also requests products to be registered and not just certified. Therefore, INMETRO analysts must evaluate each certificate in ProdCert and issue a registry number in the IRD.
 It is also part of a broad restructuring of the entity and enjoys broad support from the Ministry of Economy, Brazil Customs and key private sector organizations such as the National Confederation of Industry (CNI), the National Electronics and Appliance Manufacturers Association (ELETROS) and the National Association of Vehicle Manufacturers (ANFAVEA).
 Allowable traveling expenses will be funded separately.