GUIDELINES

for

the Second, Mid-Term, Evaluation

of

the NUTEK Competence Centre Programme

 

The Competence Centre Programme is a NUTEK-initiative to develop university-industry interaction in Sweden. The aim is to achieve a stronger industrial impact and concentration of resources by creating multidisciplinary academic research environments in which industrial companies participate actively. The programme is a means to strengthen the important link in the Swedish innovation system between university research and industrial R&D.

The programme at present comprises 28 Competence Centres at 8 universities. About 220 industrial companies are participating in the programme. The Swedish National Energy Administration is the governmental partner in five energy-related Competence Centres. The parties in a Competence Centre - a number of industrial companies, a university or institute of technology, and NUTEK or the National Energy Administration - contribute jointly to the centre´s research programme, financially or in the form of active in-kind contributions. Their collaboration is manifested in a Competence Centre Agreement.

The programme is long-term and NUTEK and the National Energy Administration intend to contribute to the funding of the individual centres for up to 10 years. The programme has now been running for 4-5 years and the centres are in their second stage, comprising 3 years. As a next step, NUTEK and the National Energy Administration intend to negotiate with the other partners in the centres for a following 3-years stage.

The main purpose of this second, mid-term, evaluation is to give an input to the negotiations and decisions about stage 3 for the individual centres and to contribute to the further development of the programme. It will focus on the following issues:

• Technical results and scientific outputs

• Industrial relevance, benefits and effects

• Performance and development as a Competence Centre.

The First International Evaluation in 1997 - 1998

The development of the Competence Centres is based on a stepwise funding and evaluation. When the programme started in 1995, NUTEK formulated a number of criteria as a basis for the evaluations (see Appendix 1). The criteria are attached as an appendix to all Competence Centre Agreements and thus wellknown among those participating in the centres´ activities.

The first evaluation was initiated and performed by NUTEK during 1997 and 1998 at the end of stage 1. As it took place at a very early stage its purpose was not to review scientific and industrial results. The main focus was to look into and review, from an international point of view, the approaches used and the measures taken so far by the individual Competence Centres and to assess their potential for a long-term development. Four evaluation criteria were used:

All centres were reviewed by the same team of international experts with experience from similar programmes for university-industry research collaboration. The conclusions and recommendations of the evaluation team were a central basis for the negotiations and decisions about stage 2 for the individual centres and an important means to improve the performance of the centres and the programme. Their recommendations especially focused on the following issues:

• Strategic planning and the important of role of the Board in this process

• Industrial involvement in the strategic issues

• Strength and balance in the interaction industry — academia ("leading or following industry"). Involvement of senior academic staff in the centres.

• Centre Leadership and the important role of the Centre Director.

The recommendations and the extent to which they have been implemented by the centres during stage 2 will be followed up in the second evaluation.

According to the overall recommendations of the evaluation team NUTEK has initiated a Centre leadership training programme which is now in progress.

The first evaluation of the programme is documented in six reports (NUTEK report 1997:18, 1997:47, 1997:76, 1998:2, 1998:28 and 1999:25).

Time schedule for the Second Evaluation in 2000 - 2001

The centres are now running their Stage 2, for most centres comprising three years. According to the Competence Centre Agreements for stage 2, "NUTEK intends to conduct the second evaulation with a view to giving the Parties a basis for possible continued activities at the Competence Centre". This evaluation, which will be carried out 4-5 years after the establishment of the centres, should be looked upon as a mid-term review of the centres and the programme. This means that the evaluation will be used as an instrument to further develop, re-direct or phase out the individual centres.

In order to fulfil its main purpose - to give an input to the negotiations and decisions about stage 3 - the evaluations of centres have to be completed in good time (preferably 3 months) before the expiration of stage 2. This implies that the 28 centres will be evaluated according to the following time schedule:

To be evaluated not later than March 2000 (stage 2 expires June 2000)

To be evaluated not later than May 2000 (stage 2 expires August 2000)

To be evaluated not later than June 2000

To be evaluated not later than September 2000 (stage 2 expirers Dec. 2000)

To be evaluated not later than November 2000 (stage 2 expires Febr. 2001)

To be evaluated not later than December 2000 (stage 2 expires March 2001)

To be evaluated not later than March 2001 (stage 2 expires June 2001)

To be evaluated not later than September 2001 (stage 2 expires Dec. 2001)

The first evaluation of the Centre for Circuit Design, CCCD, Lund Univ., which started in January 1998, was carried out in November 1999.

Aspects to be covered by the evaluation. The task of the evaluators.

A NUTEK Competence Centre has two main challenges to meet:

The aim of the evaluation is to assess the progress of the centres in these respects.

The evaluation will look into and review, from an international point of view, the progress and the results of the centre´s activities attained since the centre started. It will focus on the following issues:

• Technical results and scientific outputs

• Industrial relevance, benefits and effects

• Performance and development as a Competence Centre.

The Competence Centre Programme means that new forms of university-industry research collaboration are tested and developed. An overall aim of the evaluation is to judge and measure the added values of the collaboration efforts and the Competence Centre concept. This could include "soft" added values, for instance building up new linkages and collaboration networks, developing dynamic research environments, increased industrial impact in universities, development of leadership and management skills etc.

The prime reference frame for the review of the scientific outputs and the industrial relevance and benefits will be other R&Defforts in the research areas and disciplines covered by the centre´s research programme. The development as a Competence Centre will be reviewed with other centres in the programme and internationally as the prime reference frame.

Evaluation teams

The evaluation of individual centres will be carried out by groups of 4-5 international experts, two or three of them preferably being "generalists" and at least two of them being internationally distinguished and renowned specialists within the centre´s research area. It is advantageous if the scientific experts also have industrial experiences.

Since the Competence Centre programme covers a wide spectrum of technology areas and industry sectors, the scientific experts will differ from centre to centre. Nevertheless, opportunities will be sought to find scientific experts who are capable and willing to participate in the evaluation of adjacent centres. The experts will be selected and appointed by NUTEK/The National Energy Administration in consultations with the centres. It is important that the choice of scientific experts is accepted by the Centre. The experts should have no personal couplings to the centre.

A very positive experience from the first evaluation in 1997-1998 was to have one evaluation team reviewing all centres. Also for the second evaluation the ambition is to include in all evaluation teams persons who have acquired an overview of the programme and the Competence Centre concept by taking part in the review of several centres. For these reasons NUTEK has invited the review team from the first evaluation to form a central group of generalists. All evaluation teams set up to review individual centres will hopefully include at least two persons from this central group. In addition to taking part in the evaluation of individual centres, the generalist group will be asked to make an overall assessment of the progress of the programme and its impact on Swedish higher education institutions and industry.

NUTEK will appoint a Swedish moderator for the evaluation chosen from NUTEK´s Industrial and Scientific Advisory Board. The moderator will not be a member of the evaluation teams but an independent resource as adviser to the teams, for instance in policy issues.

Organization of the site visits

The basis for the evaluators will be a written report from the Competence Centre and a site visit at the centre. NUTEK´s instructions to the centres for their reports to the evaluators are given in Appendix 2. The centre reports will be distributed to the members of the evaluation group a couple of weeks before their arrival.

The central part of the site visit at the Competence Centre will be a main review meeting with the whole evaluation group (scientific experts + generalists) and the centre. It is planned to take 4-5 hours. During the main meeting the evaluators will meet with the following persons and partners in the Centre:

In order to make it possible for the scientific experts to discuss and scrutinize in more detail the technical and scientific output they will be invited to arrive one or half a day before the main meeting to meet with the Centre Director, the research leaders an the PhD-students. Staff from NUTEK or the National Energy Administration will be present at the site visits.

 

 

Reports of the evaluation groups

The work of the evaluation group shall result in a report to NUTEK or the National Energy Administration. The evaluation team is supposed to be unanimous in its conclusions and recommendations. Preferably, the report should comprise three parts dealing with the main issues for the evaluation:

Discussions and recommendations on priorities of actions to be taken by the centre itself as well as by NUTEK or the National Energy Administration, both in terms of financial support and of more structural matters, are appreciated.

The report of the evaluation group will be published by NUTEK or the National Energy Administration. It will be openly circulated to all Competence Centres and, on request, to any other agencies or persons who have expressed an interest in this type of information. The Swedish scientific community is used to outspoken international evaluation reports.

Remuneration

NUTEK or the National Energy Administration will pay for all costs for the members of the evaluation groups including travels, accommodations etc. According to NUTEK´s standards for international evaluations, each member of an evaluation group will receive a small remuneration of USD 2000.

NUTEK's Criteria for Competence Centres

A successful Competence Centre

- PhD students and academic researchers conduct research in active collaboration with and within industrial companies;

- company R&D staff are active in the centre's academic environment;

 

The above criteria will be used as a basis for the evaluations of the activities of the Competence Centres that NUTEK will perform.

 

Second, Mid-Term, Evaluation of NUTEK Competence Centres in 2000-2001

Instructions/Guidelines for

Centre Reports to the International Evaluation Group

Each Competence Centre should submit a report to NUTEK (and for energy-related centres to the National Energy Administration, STEM) in 8 copies not later than 4 weeks before the site visit of the evaluation team. The reports will be forwarded to the international evaluators.

It is recommended that the report contains no more than 30 pages plus enclosures. The following information should be given in enclosures:

Content

The evaluation and the report should deal with the progress and the results attained since the centre started. The table of contents given below has not to be strictly followed, but it is recommended that all aspects mentioned be covered in the report. The list of headings could be changed and expanded in order to include relevant information on progress and achievements of your centre.

Added values of the Competence Centre concept. Executive summary.

A summary of main progress, results and benefits.

List and rank the main added values to be a Competence Centre. Relate the added values to the main outputs and benefits described in sections A-C below.

Basic facts

Participating research groups/departments and industry partners (for the companies incl. number of employees and areas of interest in the centre).

Research staff from the academic and industrial partners (list showing different categories, names, and their % time committed to the centre).

Economic accounting. Income budget according to the agreement for stage 2 showing contributions (cash and in kind separated) from NUTEK or STEM, the university partners and the individual companies. Account for income, budget and economic result for the latest fiscal year. Allocation of resources to subareas/subprogrammes and to centre management.

 

A. Performance and development as a Competence Centre

1. Long-term goals and progress of the centre

The long-term strategies and goals of the centre. Assess the progress of the centre so far, in relation to goals and milestones.

Experiences and development of the centre since it started, incl. measures due to the first NUTEK evaluation and the centre´s self-assessments. State some major problems you have encountered and how you solved them.

2. International and national collaboration. Attractiveness of the centre

Describe the research and competence profile of the centre. Rank the centre in relation to internationally leading groups or centres.

Indicators/examples of the attractiveness of the centre (guest scientists, external invitations etc.).

Main collaborating partners worldwide. Participation in international programmes, incl. EU framework programmes.

Marketing activities and and measures to inform about the centre.

3. Linkages and collaboration within the centre

Experiences and effects of measures taken to achieve strong links and integration between research groups/departments/faculties, between academia and companies, and between companies.

Integration of the industrial partners´ interests in a "research programme of mutual interest". Proportion of the research projects involving staff from more than one industrial partner.

Management of the centre.

4. The centre and the host university

The status and role of the centre in the university organization. Measures and organisational development in order to strengthen the identity and effectiveness of the centre.

University commitments and involvement in the centre.

Evidence of impact of the centre on the university activites and priorities, e.g. new interdisciplinary collaboration between departments/faculties,

on graduate and undergraduate education, creating new partnerships with industry.

5. Future plans and strategies (alternatively to be included in A1 above)

Preliminary plans for the next 3-5 years period.

 

B. Technical results and scientific outputs

1. Research programme

Summary of the research programme and subprogrammes/projects; (focus, objectives, new approaches and methods). Main progress so far.

Methods used to secure the quality and relevance of the research programme and the projects.

2. Technical and scientific achievements

Highlight some major technical and scientific achievements.

3. Scientific papers

Enclose a list of publications categorized in:

- publications in refereed scientific journals

- invited and other conference contributions

- other reports.

The list should only include papers that are strictly results of the centre´s research programme.

Submit 3-5 key publications which are central and representative (e.g. cross-disciplinary, co-authored university-company)for the research programme and the centre.

4. Examinations

Theses for doctor´s and licentiate´s degrees; diploma works.

For the PhDs awarded; specify the proportion of their education programme they have been involved in the centre as well as their present employments.

Assess if possible, the centre graduates compared to "traditional" graduates.

5. Education and training activities

Courses, lecture series etc. conducted by the centre.

 

 

  1. Industrial relevance, benefits and effects

1. Industrial involvement and interaction

Describe the industrial involvement and how it has developed since the centre started (incl. number of industrial partners; strengths and depths of the industrial involvement. If industrial partners have withdrawn, state the reasons.)

Evidence of mutual personal mobility between the industrial and academic R&D environments.

2. Characterize and, if possible, rank the industrial partners´expectations on the centre. Assess the extent to which the expectations have been fulfilled so far. (It is anticipated that the information is not only in general terms and that it illustrates the industrial partner´s own views.)

3. Commercialisation and technology transfer activities

e.g. IPR-results (patents applications, licenses etc.)

4. Evidence of industrial benefits by participating in the centre.

Estimates of potential economic benefits (if possible), e.g. through new or improved industrial processes and products.

Give examples. The list of headings below could be used as a guideline.

e.g. interaction with the academic and other industrial partners, access to the international scientific community, etc.

e.g. through centre research staff and graduates which have been employed by industrial partners, collaborative projects etc.

e.g. through diploma work, industrial training courses and seminars, access to centre equipment and facilities.

e.g. technology and methods that have been implemented within the companies.