In the context of the current economic and financial situation, a proper evaluation of major public spending schemes is not just a formal obligation, it is a clear necessity – and the current FP7 and the future Horizon 2020 Programmes are no exceptions to this rule.
In 2010, an independent expert panel chaired by Rolf Annerberg, Director of the Swedish Research Agency FORMAS, carried out the Interim Evaluation of FP7. This document raised a number of issues and recommendations, which were not only discussed at the Council and the European Parliament, but had a clear influence on the shaping of the debate on the future of European research funding.
At this point in time – prior to the Commission Decision on the Horizon 2020 Programme – it is clearly premature to enter into a detailed description of future evaluation approaches. Instead, this paper aims at highlighting some of the key building blocks for such a future Horizon 2020 evaluation approach.
Although evaluation is a permanent task, attention will undoubtedly focus on the three major evaluation reports which can be expected in the years to come:

• The Ex-post Evaluation of FP7 in 2015,
• The Interim Evaluation of Horizon 2020 (around 2017), and
• The Ex-post Evaluation of Horizon 2020 (around 2023).

In line with the very positive experience of recent evaluations of FP6 and FP7, these evaluations will be

• Involving independent external experts,
• Based on comprehensive evidence base, and
• Formally communicated to the Council and the European Parliament.

Prior to the launch of the Programme in 2014, a detailed and explicit evaluation strategy for Horizon 2020 should be developed, notably to ensure adequate coverage of all areas and components and to provide detailed timetables. This strategy should be revised later on to adapt to new developments, as in a fast moving area like research and innovation things are very likely to evolve substantially over the next decade or so.

A key component of such a strategy should be a comprehensive governance structure. This should of course clarify the roles and responsibilities across the different Commission services and agencies involved in Horizon 2020. Moreover, in view of the growing number of activities involving Member States and other stakeholders, their respective roles should also be clearly defined. Last but not least, independent expert advice at the strategic level might be a further asset to strengthen the overall efficiency of the Horizon 2020 evaluation system. The aim is to implement a coherent approach, so that all thematic components of Horizon 2020 will be evaluated on a regular basis and in an adequate and coordinated manner. Key tools in this approach will be thematic evaluation studies for each of the various action lines, but also analytical work to shed more light on cross-cutting and transversal issues – such as participation patterns or international standing, just to quote two possible topics.
Wherever appropriate, there should be an increased use of common templates and methodologies, and of course of a common set of key indicators, so that there will be an increased comparability and harmonisation of the reports.

Not surprisingly, a solid evidence base will be of paramount importance for the planned evaluation activities. This includes notably a continuation and development of the current Monitoring system, but moreover the development of new structures to capture the throughput, output and impact of Horizon 2020 activities. The very successful example of the common data warehouse CORDA indicates that substantive progress can be achieved. In this context, it seems important to step up research activities in order to stimulate the development and use of new techniques and to also foster the development of the European research evaluation community.
Finally, it is obvious that there needs to be maximum transparency at all stages of the Horizon 2020 evaluation. This should start with the development of the Horizon 2020 Evaluation Strategy, and notably include annual Monitoring Reports with key data on implementation and performance indicators. In addition, a new annual Evaluation Report could present key findings and features of the current evaluation work. The availability of all reports and findings should also contribute to further increase the relevance of evaluation work for the political and societal debate on European research policy.

As transparency is not just a promise for the future, but already a reality for the evaluation activities on FP7, please consult our dedicated website (see link below) to check for further details on the evaluation of FP6 and FP7. Most notably, this website allows you to download the full texts of all recent FP6 and FP7 evaluation studies.

Author: Peter Fisch, Head of Unit Ex-post Evaluation, DG Research and Innovation