Year: 2020
Type: report
Abstract: Executive Summary This final report has been prepared by AIT – Austrian Institute of Technology and the Katholieke Universiteit Leuven (KU Leuven, sub-contractor) for the programme evaluation of the Special Research Programmes (SFB) of the Austrian Science Fund (FWF). The evaluation was carried out between January 2019 and February 2020 and critically reviews the SFB programme based on projects granted in the period 2004-2018. The aim of this evaluation is to assess the SFB programme’s outputs, outcomes, and impacts against its stated objectives and to provide recommendations for the future implementation of the programme. The SFB programme is expected to fulfil the following three objectives: Support outstanding research in Austria by creating long-term, generally multidisciplinary/interdisciplinary and (by international standards) exceptionally productive research networks (SFBs) which should reach “critical mass” Expand human resources in science and research and improve gender-balanced orientation of research and education Achieve broader effects on the Austrian science and research landscape by supporting universities and other research institutions in the autonomous creation of focus areas in their research profiles and improve public awareness of top-quality research by supporting science communication, knowledge transfer and appropriate dissemination strategies Based on these objectives, the terms of reference and this study identified five evaluation dimensions based on the overarching objectives of the programme. The first three relate directly to the specific SFB objectives and expected impacts of the programme, while the fourth dimension concerns the adequacy of FWF processes for selection, management, and support of SFB throughout its lifecycle, and the fifth dimension relates to the correspondence of the SFB programme to FWF objectives and the Austrian research and innovation system: Support for outstanding research in Austria Enhancement of human resources in science and research and improved gender balance Achievement of broader effects on the Austrian science and research system Effectiveness and efficiency of programme implementation Coherence of the SFB programme with overarching objectives In order to address the evaluation dimensions outlined above, a theory-based evaluation and contribution analysis mixed-method evaluation design was chosen. Evidence was collected by means of a literature review, portfolio analysis, bibliometric analysis, control groups (FWF Stand-alone projects and rejected SFB), counter-factual analysis, a large interview programme of individual and focus group interviews, and two online surveys. This Executive Summary presents the key findings and conclusions along the five evaluations dimensions with relevant recommendations for the future orientation and implementation of the SFB programme embedded in the corresponding sections. Key findings and recommendations Coherence and contextualization of the SFB programme The SFB programme is still the only basic research oriented and network-based research programme in Austria and has therefore a unique selling proposition with a strong relevance for Austria’s research and innovation system and an excellent reputation among researchers and university leaders. The programme is strictly bottom-up oriented and exclusively focused on research excellence. This constitutes a unique feature in international comparison, where the use of thematic priority setting is wide-spread, and the strategic orientation of basic research networks has changed. Two major changes in the configuration of the programme occurred since the 2004 evaluation: Firstly, the overall acceptance rate decreased from 54% to 14%. Secondly, the share of SFB funding in the total FWF funding decreased from 14.7% in the year 2014 to 5.6% in 2018. The FWF did not follow the recommendation of the 2004 evaluation which indicated that provisions are certainly needed to keep up or even enlarge the share of budget that goes to the networks. The programme exhibits a strong dominance of the Natural Sciences (57%) and Human Medicine & Health Sciences (26%), a concentration that is higher than that found in the Stand-alone projects. The distribution does not quite reflect the overall intention of the programme, which explicitly addresses researchers “from all disciplines”. The skewed disciplinary distribution is to some extent due to structural differences in Social Sciences and Humanities and the Natural Sciences (e.g., small size of Social Sciences and Humanities research in Austria, a lack of natural networks, lack of incentives, approach to research). However, the eligibility requirement of concentration of at least 50% of sub-projects at one location seems to be a barrier to researchers in Social Sciences and Humanities to participate in SFB to some extent. The existing disciplinary distribution and the high rejection rates of the programme also constitute an evident barrier to application. Compared with the last evaluation, the programme has lost momentum and transformed to an extremely competitive niche rather than a systemic oriented flagship programme that is capable to shape profile building of research organisations. The following factors are responsible for this transformation: 1) A low level of absolute annual funding, 2) a negative trend in the share of funding provided for by the FWF, 3) a tight concentration on a limited number of disciplines, and 4) acceptance rates of just about 14%. This is worrying, as all system level analyses confirm an increasing relevance of building new and sustaining critical mass, establishing better career prospects for young high-potential principal investigators, and sharpening of research profiles at Austrian R&I institutions. Against these findings, the following recommendations are given: Recommendation 1: FWF’s funding share allocated to network programmes should aim to reach a minimum level of 25%. Given the level of ambition of the programme, such as achieving broader effects on the Austrian research landscape, the resources do not suffice. For pursuing the overarching objectives set, a better endowment with resources from the FWF budget is needed for network programmes. Recommendation 2: FWF should elaborate measures that allow stronger participation of more disciplines and support the emergence of new fields of excellence. Measures could comprise the introduction of calls for certain disciplines as well as measures focussing on structural impact for sharpening research profiles at Austrian R&I institutions. Recent system level analyses confirm an increasing relevance of building new and sustaining critical mass and establishing better career prospects for young high-potential principal investigators. It therefore could also be considered to incorporate aspects of relevance and impact in the funding decision process. Support for outstanding research The analyses performed clearly show that SFB succeeded in supporting exceptional research as evidenced by the outstanding publication and citation record of the funded projects. The publication profile of the SFB program and its projects in the period 2004–2017 displays a strong continuity in the growth and development of the scientific activity. The publication analysis shows that SFB researchers receive a high citation impact and that their work reflects a very high scientific standard, and SFB PIs in general publish in high impact journals and receive more citations than expected for these journals. SFB projects also exceed national averages and outperform Stand-alone projects along all metrics of citation impact. The excellence in scientific outcomes is not paralleled by achievements in terms of interdisciplinarity and international collaboration. SFB projects involve fewer internationally co-authored publications compared to the Austrian average, and average interdisciplinarity scores of the SFB program lies below the average of Stand-alone projects and the Austrian average. Interdisciplinarity is typically a result of very closely related (sub-) disciplines working in a sub-project. It is mainly found in fields of science with clearly overlapping content such as biology and medical theoretical sciences and clinical medicine. In this respect, interdisciplinary co-operation in SFBs resembles to a large extent the cross-disciplinary structure of Stand-alone projects. In somewhat stark contrast is SFB participants’ own assessment of interdisciplinarity in the programme. Interdisciplinarity is seen as one of the key attributes that makes the SFB attractive and unique and SFB researchers perceive their projects as highly interdisciplinary. On the other hand, interviewees stressed that interdisciplinarity is merely a means to an end. The evidence on this apparent contradiction in opinions on interdisciplinarity shows that researchers have different understandings of the differentiation between multi- and interdisciplinarity; a lack of intrinsic motivation for interdisciplinary integration; and that the peer review process is perceived as detrimental to interdisciplinary projects integrating non-closely related fields of science, thus causing strategic choices of disciplinary distribution in SFB proposals to maximize success rates. Against these findings, the following recommendations are given: Recommendation 3: FWF should keep the overall programme structure (network size, funding provided, duration etc.) and principles of a two-stage peer review process. The two-stage peer review selection system is in principle capable of identifying research networks that can deliver outstanding results in terms of publication impact. For funded SFBs, framework conditions (network size, amount of funding per projects, etc.) which allow delivery of outstanding research results ar
Source: Lirias
Institutions: AIT Austrian Institute of Technology GmbH, KU Leuven
Language: English
Subfield: Information Systems and Management
Field: Decision Sciences
Domain: Social Sciences
Open Access status: green
OpenAlex ID: W3210771296