Final Evaluation Request by Deutsche Welle Akademie: Final evaluation of the EU-funded project “European Media Facility in Armenia - Building Sustainable and Professional Media”
DW Akademie is commissioning a final evaluation to gather relevant data which will be used to measure to what extent the project has achieved its specific objective.
Evaluations of DW Akademie projects are based on principles that both, the contracting authority, and the contractor must adhere to. These principles stem from the OECD/DAC and the German Association of Evaluators (DeGEval) and are: Impartiality and Independence, Credibility, Usefulness, Participation, Feasibility, Fairness and Accuracy.
The evaluation should be based on six criteria developed by the OECD/DAC:
- Relevance: Did the project address key problems of the target groups and society as a whole?
- Coherence: What synergies did the intervention have with other initiatives in the field? How well did the intervention identify needs of the target groups within the general framework of media support in Armenia?
- Effectivity: Is the intervention achieving its objectives? What are the benefits of working with regional media in contrast to national media? What was the effectivity of activities conducted online in comparison to in-person formats? Were past and recent lessons learned integrated into the intervention? How was the cooperation between the lead and co-applicants
- Efficiency: Were the activities and procedures cost-efficient (cost-benefit ratio) and resources used proportionately? Was the project able to adapt to changing frameworks (war, COVID-19, currency exchange fluctuations, inflation, etc) in an efficient way?
- Impact: What impact did the project have on target groups and beneficiaries and what are concrete examples/illustrations of it?
- Sustainability: Did Armenian partner organisations retain sufficient ownership of the results of the project to continue positive effects?
Evaluation Process and Format
a) Process description
After the selection and contracting of the evaluator(s), DW Akademie will provide basic documents and information needed for the evaluation. During an online kick-off workshop, the evaluator(s) and the DW Akademie will agree on the final scope of the assignment, the research questions, methodology and evaluation steps. An inception report prepared by the evaluator(s) will detail methodological strategies and the evaluation design.
The evaluation is expected to be carried out on-site with visits to all 4 partners in Yerevan as well as 10 regional media partners, in addition to meetings with other stakeholders and beneficiaries (universities, students, journalists, fact-checkers, etc). According to our experience 2 weeks should be planned for the field phase. DW Akademie will establish contact with the interview partners and assist with general logistical support during the field visits. Depending on the security situation the scope and schedule can be adapted. The evaluator(s) will analyse the data collected independently and draft an evaluation report, which will be subject of discussion with the project team in a debriefing meeting.
b) Format of the evaluation report
The final evaluation report shall comprise no more than 25 pages and should be written in English. Additionally, a two-to-three pages executive summary shall be handed in, as well as an annex containing additional information that is important for the understanding of the evaluation results. This includes research tools such as questionnaires, interview guides, list of interviewees, etc. DW Akademie recommends a mixed methods design including both quantitative and qualitative techniques.
Deliverables
The following deliverables, to be drafted in English, will be expected:
1. Presentation of the evaluation design during the kick-off workshop with project team.
2. Inception report (max 6 pages): The evaluator(s) will present a report giving detailed information about their interpretation of the ToR, proposed evaluation methods and specified research questions as agreed in the kick-off workshop.
3. Preparation and execution of interviews and data collection (this includes also the collection of target values of the indicators for Specific Objectives).
4. Draft evaluation report including recommendations and possible lessons learned.
5. Debriefing: the evaluator(s) will present the results of the evaluation: the feedback of the project team will be considered in the final report.
6. Final evaluation report (max 25 pages).
Please note that while the main project documents are in either English or Armenian, full command of both languages is required.
Application Requirements
a) Applicant’s profile
Applicants should be experienced in conceptualizing and conducting evaluations in a development assistance context. They should be very adept at gathering and processing information and data, with proven analytical abilities and knowledge of empirical research methods. The following expertise is particularly appreciated:
- Proven record in conducting logframe-based evaluations.
- Significant experience of undertaking desk reviews, developing/implementing M&E systems and/or project assessments/evaluations (EU and/or other international donor projects.
- In addition to fluent English, strong command of Armenian language of at least one person within the evaluating team is required; - A good understanding of issues relating to freedom of expression, local media landscapes, journalistic education, fact-checking, etc.
- Expertise on the Armenian media landscape is a strong asset.
b) Format of the proposal
The proposal should outline a coherent and concise concept for the evaluation of the project. Furthermore, the applicant’s own evaluation expertise as well as the context of the project region, evaluation objectives, methodology and evaluation design should be described. In addition, a preliminary time schedule and detailed cost summary need to be presented. The exact documents to be submitted are listed below. Submitting a proposal does not constitute an entitlement to the conclusion of a contract.
c) Documents to be submitted
The following documents should be included in the proposal:
- A cover letter in which the applicant conveys her or his suitability for the specific evaluation;
- CV(s) of person(s) to be involved in the assignment, documenting relevant experience in the afore mentioned fields (incl. contact details of reference persons of minimum 2 prior project evaluations). In the case of team applications, kindly note that DW 7 Akademie retains the right to issue only one contract with the lead evaluator representing the whole team.
- A technical proposal containing the evaluator’ understanding of the task and ToR, the evaluation design, including methodology, relevant evaluation questions and a preliminary time-table (max. 6 pages).
- A detailed cost summary and the projected working days using the template provided in Annex A.
d) General Information
The estimated budget for the evaluation should not exceed 12.500 Euros (excluding VAT, this lump sum should contain all costs necessary for conducting the evaluation, including travel costs, hotel costs, materials, if applicable). No additional per diems or subsistence support is granted by DW Akademie. Proposals should be submitted in electronic form only to [email protected] by August 25th 2023, at the latest. Evaluators will be selected by September 1 st, 2023. The contract will start October 1 st and a kick-off workshop will take place mid- October 2023. The field phase is scheduled to start immediately thereafter with field visits to partners, target groups and beneficiaries. The final evaluation report must be completed by December 15th, 2023. Contact details for questions or clarification: Email : [email protected]
Proposals should be submitted in electronic form only to [email protected] by August 25, 2023, at the latest. Evaluators will be selected by September 1st, 2023.
The contract will start October 1st and a kick-off workshop will take place mid-October 2023. The field phase is scheduled to start immediately thereafter with field visits to partners, target groups and beneficiaries.
The final evaluation report must be completed by December 15, 2023.
Please click for the complete Terms of Reference and the annex, featuring more information about the ToR.