Special report No 9/2001

on the Training and Mobility of Researchers Programme together with the Commission's replies

(pursuant to Article 248(4)(2), EC)

(2001/C 349/01)

CONTENTS

>TABLE>

LIST OF ABBREVIATIONS

TMR Training and Mobility of Researchers

DG RESEARCH Directorate-General for Research

4FP Fourth Framework Programme

MCFA Marie Curie Fellowship Association

IHP Improving Human Potential

CORDIS Community Research & Development Information Service

SUMMARY

I. This report analyses the quality of the Commission's management systems used for pursuing the Training and Mobility of Researchers (TMR) programme as part of the Fourth Framework Programme (1994-1998). It concentrates on the procedures for selecting proposals and the management systems for monitoring contracts (see paragraphs 1 to 8).

II. In the evaluation of proposals the procedures followed sometimes lacked transparency: there was an absence of detailed records of the deliberations leading to the decisions made. In the allocation of proposals to evaluators potential conflicts of interest may arise (see paragraphs 9 to 15). The Court recommends that the procedures in place should be improved and rigorously followed (see paragraph 41).

III. In the case of the research networks activity, organisations which were not applying the eligibility criteria for researchers were found. Individuals who had already received fellowship awards from the Commission to execute a project were recruited (see paragraphs 16 and 17). The Commission should carry out cross-checks to ensure these instances do not recur (see paragraph 42).

IV. Weaknesses were found in the procedures for establishing unit costs in the access to large scale facilities activity. Only scientific officers from the Commission, and not financial officers, are evidenced as involved with the contractors in establishing daily user fees (see paragraph 21). Advice from financial officers should be sought and documented before the user fees are agreed (see paragraph 44). In addition, the external mid-term review was carried out at an inopportune time (see paragraph 22). This review should be carried out in line with the actual contract duration (see paragraph 45).

V. The success rates for proposals for Marie Curie Fellowships submitted varied significantly according to the submission procedure adopted by the prospective fellow (see paragraph 24). The management of the Marie Curie Fellowships should be centralised in the department involved in the TMR programme where significant improvements in management have occurred (see paragraph 46).

VI. The Commission does not systematically check that all the criteria it establishes for the euroconferences, practical training courses and summer schools are being respected. In addition, no information is obtained on sponsorship received from other entities for these events (see paragraphs 29 and 30). Procedures should be altered to ensure these checks are carried out (see paragraph 49).

VII. From the results of the survey carried out it can be noted that the beneficiaries/contractors are on average satisfied with the service they receive from the Commission (see paragraphs 32 to 36).

INTRODUCTION

Research activities in the field of the training and mobility of researchers

1. The aim of the Community's research policy, as laid down in Article 163 of the Treaty, is to strengthen the scientific and technological bases of Community industry and encourage it to become more competitive at international level. Community strategy in the field of research and technological development is laid down in framework programmes and implemented by the Commission through specific programmes. The Fourth Framework Programme(1) (1994-1998) is still being implemented while the Fifth Framework Programme(2) (1998-2002) has commenced following the European Parliament and Council Decision of 22 December 1998. In the Fourth Framework Programme (4FP) the "stimulation of the training and mobility of researchers" was selected as one of the four activities for which 792 million euro were allocated out of a total budget for the Framework Programme of 11879 million euro (see Table 1). The Council adopted a decision(3) on "a specific programme of research and technological development, including demonstration, in the field of training and mobility of researchers".

Table 1

Breakdown by measure of the Fourth Framework Programme's budget

>TABLE>

Source:

European Commission.

Objectives and budget of the TMR programme

2. The TMR programme is divided into three interdependent areas (research networks, access to large scale facilities, training through research), to which is added an activity concerning accompanying measures aimed at improving communication between researchers themselves and with industry, at encouraging young researchers and at publishing the results and achievements of the programme.

3. Research networks aim to allow researchers from five or more research teams in at least three countries to pool their efforts in a common research project and to constitute, in this manner, groups capable of performing research of higher quality. The access to large scale facilities activity is devoted to large scale installations whose uniqueness or rarity in the Community, high investment or upkeep costs, and importance for research justify substantial effort at Community level. Use of these facilities is funded to the extent that researchers from other states can obtain access to the installations that was previously unavailable. The aim of the training through research activity (which was subsequently renamed Marie Curie Fellowships) is to allow European researchers to undergo training or specialisation outside their country of origin.

4. Accompanying measures are taken to contribute to the objectives of the programme in relation to the stimulation of researcher mobility and training while being at the same time an appropriate vehicle for the dissemination of information on the conditions and procedures for participating in the various activities of the programme and for the dissemination of its results. These are referred to as euroconferences, practical training courses and summer schools.

5. The TMR programme is managed by a unit of four departments comprising 59 members of staff as at 31 December 1999 in DG Research.

6. The budget for the four main areas in this programme was amended following the accession of Austria, Finland and Sweden. Table 1 shows the budget split between these areas. The total amounts committed and paid each year can be found in Table 2. The main point derived from these figures relates to a certain under execution of the payment appropriations in the first years of the 4FP.

7. Table 3 shows the amount of the TMR budget used in administrative and personnel expenditure from 1995 to 1999. The Council Decision sets a ceiling on this expenditure of 5,6 % of the total budget. It can be seen that the Commission is within these limits.

Table 2

Implementation of the budget - Training and mobility of researchers

>TABLE>

Source:

European Commission.

Table 3

Training and mobility of researchers - Administrative expenditure

>TABLE>

Source:

European Commission.

The Court's audit

8. This audit was focused on analysing the management systems used by the administrative services both at the Commission and at contractor level in pursuing the programme and on examining the quality of the way they were managed. The main audit objective was to assess the effectiveness and efficiency of the TMR programme. The procedures in place for selecting contracts were audited and a follow-up was carried out on the weaknesses noted in the earlier report by the Court(4) in this area. Ten on-the-spot audits were carried out in six Community countries, during the course of which 19 programme beneficiaries involved in 51 contracts were audited. Finally, by means of on-the-spot audits and a postal survey, information on both the contractors/beneficiaries involved in the programme and the level of satisfaction with the service provided by the Commission was obtained.

SELECTION OF PROPOSALS

Audit approach and general results

9. Proposals received under specific calls for proposals were audited to determine if the written procedures were correctly followed and if the actual selection processes were transparent. Proposals are submitted by organisations and individuals throughout the Member States and associated States. Success rates vary between areas and calls.

10. When a proposal is submitted, as a first step it is reviewed by between two and five evaluators (called Primary Reviewers) selected by the Commission. A preliminary single overall score is then awarded which is normally the average of the scores awarded by the Primary Reviewers. As a second step the Primary Reviewers then meet (called a panel meeting) to discuss the results of the evaluations. In the panel meeting the preliminary single overall score can change in line with specific written procedures. The final list produced by the panel meeting is submitted to the Commission. In the cases examined by the Court, the summary minutes produced, however, do not include the deliberations leading to the production of this final list. The final decision on which proposals to accept is taken by the Commission.

11. There were some variations in the way in which, in practice, proposals were evaluated between the four activities. Weaknesses were found in all areas, with findings being that no documents were produced explaining the evaluators' decisions and that evaluators had assessed proposals to which they were linked; cases were found where an evaluator assessed a proposal in which his/her employer/association was involved.

Research networks

12. In the area of research networks, the ranking stated in the summary minutes of the panel meeting differed from the single overall scores stated in this document. The proposals were assessed in accordance with the procedure in force, but this is not fully reflected in the minutes in so far as the panel's proceedings are only partially documented. The minutes record the adjustments made by the Primary Reviewers in order to ensure, where necessary, that their individual scores are consistent with various proposals before the plenary discussion. The minutes do not, however, record the subsequent adjustment made to the score of certain proposals in the light of the panel's plenary discussion. This leads to an apparent inconsistency between the final classification, resulting from the panel's discussions, and the individual scores of some proposals. There is, in any event, a certain lack of rigour in the minutes, but this in no way impairs the assessment result (list of proposals which the panel suggests that the Commission finance, in order of priority). A total of 13 proposals were funded by the Commission as a result of the recommendations of this panel.

Access to large scale facilities

13. For the area of access to large scale facilities, in 13 cases out of 61, the explicitly detailed instructions given for determining the ranking of the proposals were not respected. This resulted in 10 proposals being given higher preliminary rankings than the assessments indicated. Six of these 10 proposals were funded. No documents were found justifying these higher rankings.

Contract extensions

14. In this area, the third call for proposals in 1997 resulted in new proposals being received from existing contractors, requesting additional funds for granting access. Thirty-eight of these were successful. Instead of issuing new contracts for these proposals, contract amendments were signed extending the duration of the existing initial contracts and the financial contribution. This approach was taken to ensure the available budget was used up. In future, the Commission should ensure this situation does not recur and it should obtain written advice from its legal service on the correct procedures to apply. In addition, two different forms of overlap were found with these contracts. The first concerns the duration of the contracts, where between one and nine months of the time period of the first contract was also considered part of the extension in duration of the amended contract. This overlap occurred to avoid "new contracts" commencing after the end of 4FP. The second instance of overlaps concerns the additional financing, whereby in some instances cost statements covering periods in the initial contracts were financed from the additional funding for future periods. This situation was not caused by the contractors but as a result of a misinterpretation of the contract on the part of the Commission services involved. The total amount paid to the contractors over the duration of the contract was correct but, because of the above misinterpretation, some amounts were prematurely paid.

15. In the analysis of the proposals in 1995 and 1997 for the above contracts, it was noted, in 27 of the 38 proposals, that one of the evaluators of each proposal was the same in both periods. In one of these cases, the same two evaluators assessed the proposals in 1995 and 1997 (in 1997 a third evaluator was involved). From the analysis of the relative annual reports on these contracts for the years 1996 to 2000, it was noted that in 11 instances one of the evaluators was linked to the contractor. In seven instances the evaluator was a referee in the contractor's user selection panel and in the other four cases the evaluator was a user of the facility, and therefore a beneficiary of the Commission funding. In one particular case both 1997 evaluators were using the facility they assessed in 1997 and 1998, and 1997 and 1999 respectively. Indeed, these evaluators were benefiting from the funded access to this facility in 1997 in the four weeks around the time they carried out the evaluations. One of them had to interrupt his stay at the facility to attend the panel meeting in Brussels. All of these cases could lead to conflicts of interest which the Commission procedures should prevent.

CONTRACTS AND MANAGEMENT CONTROL

Research networks

16. In the research networks activity (also known as research training networks), various conditions apply to the personnel costs of visiting researchers. Two of these conditions are that:

(a) his/her appointment must be temporary and for a fixed-term starting after the commencement date of the contract, and

(b) he/she must not be a national of the State in which the contractor appointing him/her is situated and he/she must not have carried out his/her normal activities in that State for more than 18 of the 24 months prior to his/her appointment.

17. During the on-the-spot audits, instances were found where researchers who had previously received fellowships were declared as visiting researchers on the research network even though they had spent up to 36 months prior to their appointment in the same organisation. These researchers were thus not eligible to be classed as visiting researchers.

18. A common complaint during the on-the-spot audits and from the survey was that the contractors found it difficult to recruit researchers for the project. Many had misunderstood the instructions and thought they were limited to advertising on Cordis and had very few applications. In addition many contractors, not just those in the research networks activity, were critical of the Cordis website itself. It was considered too voluminous and difficult to use and many organisations avoided using it if possible.

Access to large scale facilities

19. In the model contract for the access to large scale facilities activity, the allowable costs are split into two categories for the cost statements: travel and subsistence costs and user fees.

20. The travel and subsistence costs may be charged to the contract in accordance with the "normal internal rules and procedures of the contractor". Therefore, if the systems in place at the contractor are inadequate, and since there is an absence of minimum requirements requested by the Commission, it is extremely difficult to check the validity of these costs. Instances were found where supporting documentation was not available to justify expenditure. In addition, the absence of verification from the users of the facilities that they did not receive reimbursement for their travel and subsistence costs from their home organisation led to instances of double financing. A university professor has been brought before a disciplinary committee as a direct result of the above situation.

21. In the majority of these contracts, user fees per day were agreed between the Commission and the contractor. These were established in ecu per day. The exchange rate exposure of the Commission was therefore limited and cost statements were easier to review as the rates had been agreed in advance. However, no evidence was found that the advice of financial officers had been sought by the scientific officers in determining these user fees. As the user fees absorbed a high portion of the funds (in some instances up to 95 % of a contract with a maximum Commission contribution of 2 million ECU), it was imperative that they were established at a fair, accurate level. The rules for calculating the user fees per day specify that all costs should be calculated in accordance with accounting principles relating to historic costs. The audits identified that, while the majority of contractors had declared the actual costs in determining the user fees, a small number had used incorrect calculations. For example, for a contract at the Joint Research Centre, the calculation of the user fees per day was actually based on estimates, resulting in user fees in the contract more than 30 % higher than could be justified using historic costs. DG Research is currently investigating this.

22. A requirement of the contract is that mid-term reviews be carried out on all contracts over 1 million euro in this area. This review was carried out on all contracts regardless of value in 1998. For the contracts that commenced in 1998, the review came too early as many were only operational for two to three months and were merely evaluating requests for access. For other contracts ending in 1998, the review came too late as it was not possible to improve areas criticised before the contract ended. The only contracts suited to the timing of this review were the 38 contract extensions mentioned in paragraphs 14 and 15.

Training through research (Marie Curie Fellowships)

Follow-up of previous observations

23. Since the Court's previous report on this area, efforts have been made to improve the management of this activity. One of the main criticisms concerned the absence of follow-up by the Commission when scientific reports are not received from fellows. Substantial improvements have been noted in this area with payments being linked to receipt of reports and automatic follow-ups being sent by the Commission to contractors when reports are not received.

Decentralisation of selection and management of Marie Curie Fellowships

24. Marie Curie Fellowships are available not just to applicants in the TMR programme but also under specific calls for specific thematic programmes existing in the Fourth Framework Programme. In the analysis of the evaluations of Marie Curie Fellowships the average success rate of applications in the TMR programme was found to vary considerably from that in the thematic programmes. In one instance, the divergence was so wide that a candidate would have been three times more likely to succeed with an application in the context of a specific programme than in the TMR programme.

Taxation of Marie Curie Fellows

25. It was noted that the application of income tax to the stipend of Marie Curie Fellows varied from country to country and even between tax districts in the same country. The non-harmonisation between tax districts in a country makes the process of determining monthly stipends for each fellow visiting that country difficult for the Commission and causes significant variations in the net amount received by the fellows in these countries.

Marie Curie Fellowship Association

26. The Marie Curie Fellowship Association (MCFA) was established in November 1997 as an international, non-profit-making association under Belgian Law. The aim of the MCFA, as defined in its statutes, is to:

(a) help advance science in Europe;

(b) create a clear and visible identity for Marie Curie Fellowships;

(c) promote and increase the reputation of Marie Curie Fellowships;

(d) improve contact and the flow of information between the members of the Association;

(e) establish a structured interface between fellows and the world of research, including industry;

(f) develop and foster the international dimension of Marie Curie Fellowships;

(g) increase the appreciation and understanding of science by the general public.

27. The Commission contribution to the MCFA, financed from the TMR programme (see Table 4), represents the quasi-totality of the resources of the Association.

Table 4

Commission support to the Marie Curie Fellowship Association

>TABLE>

Source:

European Commission.

28. The audit of the MCFA exposed a serious absence of management controls. Financial procedures concerning the eligibility of expenditure were not in place, the effect of which was that all expenditure incurred was deemed eligible for Commission funding. This culminated in a significant portion of the funding received from the Commission not always being spent effectively.

Accompanying measures

29. In the model contract for euroconferences, practical training courses and summer schools, contractors are asked to ensure a balance of people under the age of 35, from less-developed regions, to encourage female participation and to encourage industrial participation. However, on the cost statements the contractors only have to prove that participants in the conferences are under 35 years of age and consequently there was no evidence that all the other criteria were systematically being checked.

30. It was noted during the on-the-spot audits that the Commission is not the only source of funding for these conferences and in many cases other sponsorship has been received. However, no steps were taken to ensure the total sponsorship/funding did not exceed the total costs of the event.

On-the-spot controls by the Commission

31. In the period 1996 to 1999 no on-the-spot financial controls were carried out by the Commission, either by Commission officials or external contractors, on the TMR programme. It is imperative that a minimum of checks be carried out early in a programme to ensure both that the correct procedures are being carried out by the contractor and that these procedures and the objectives are realistic.

SURVEY RESULTS

32. In the period September to December 2000 a total of 1896 questionnaires were sent out to contractors participating in the TMR programme. Questionnaires were sent, on a random sample basis, to all categories of contractors/beneficiaries in all Member States and all states associated with the framework programmes. The number of contracts surveyed and their volume can be found in Table 5. 1372 (72,36 %) replies were received. An analysis of the number of questionnaires sent and replies received can be found in Table 6. The aim of these questionnaires was to evaluate the performance of the Commission and the usefulness of the TMR programme itself. The initial part of the questionnaires was designed to obtain information on how involved the contractors/beneficiaries were in earlier framework programmes and if they were involved in other activities in 4FP. Each process in the contract procedure from call for proposal, preparation of proposal, evaluation, contract negotiation and signature through to reporting and payment procedures was questioned to determine how satisfied the contractors/beneficiaries were. Finally, the contractors/beneficiaries were asked if they would participate in the Fifth Framework Programme.

Table 5

Summary by type and value of TMR programme projects audited (on-the-spot/questionnaire)

>TABLE>

Source:

European Court of Auditors.

Table 6

Analysis of questionnaires sent and received

>TABLE>

Source:

European Court of Auditors.

33. The main results of this survey can be found in Table 7. Many respondents also attached remarks to their returned questionnaires which provided further insights on the functioning of the Commission and the TMR programme.

34. A point worth noting from the results concerns the access to large-scale facilities activity. More than half of those surveyed had previously held contracts in this area under the earlier framework programmes and almost all had already submitted a proposal in the Fifth Framework Programme. It is also interesting to note that more than half the respondents in this area were also involved in other activities in the TMR programme.

35. From the results it can be noted that the beneficiaries are on average satisfied with the service they receive from the Commission. The area where the least satisfaction is expressed concerns the calls for proposal and the selection procedures (44 % to 65 %).

36. The area where beneficiaries are most satisfied relates to the answers they received from Commission staff in response to their queries (62 % to 77 %).

Table 7

Results of the survey conducted on the spot and by post

>TABLE>

Source:

European Court of Auditors.

CONCLUSIONS AND RECOMMENDATIONS

Conclusions

Selection of proposals

37. The transparency of the procedures for selecting proposals was not always evident:

(a) in the allocation of proposals to evaluators, potential conflicts of interest may arise; and

(b) there was an absence of detailed records of the deliberations leading to the panel decisions made.

Contracts and management control

38. In the training through research activity (Marie Curie Fellowships) the Commission has taken significant steps in improving the management of the programme.

39. The wide divergences in success rates for proposals in the area of Marie Curie Fellowships between the TMR programme and the specific thematic programmes is unsatisfactory and may adversely affect the reputation of the fellowship scheme.

40. The department managing Marie Curie Fellowships in the TMR programme shows how a centralised unit can operate effectively and efficiently within the Commission.

Recommendations

Selection of proposals

41. The following procedures, related to the situations described in paragraph 37, are proposed in order to improve the system of evaluation.

(a) All applications from individuals to become evaluators of proposals should be assessed and a decision taken on the suitability of the applicant for the work. The applicant should be informed of this decision. The experts remaining on the list should be used for the evaluations. The subsequent selection should be based on an analysis of the risk of potential conflicts of interest, covering areas such as home institutes, membership of associations and, to some extent, nationality.

(b) No evaluator should be allowed to attend an evaluation where his/her organisation has submitted a proposal. All evaluations should be done in one building and the evaluators should have a defined length of time to evaluate each proposal. More care should be taken in the allocation of proposals to ensure potential conflicts of interest do not arise and that evaluators have the necessary skills and qualifications for the proposals they are assessing. A more thorough review of the results of the evaluations should be carried out by the Commission to ensure that proposals are properly ranked.

(c) Finally, to ensure transparency, full minutes should be taken of all meetings and the discussions behind all decisions should be produced.

Contracts and management control

42. Procedures should be put in place to ensure participants in research networks are adhering to the eligibility criteria. As the main problem lies in the recruitment of researchers who have already received fellowships from the Commission, it should be relatively simple to crosscheck researchers recruited on networks against the existing database of past and present fellows in the participating organisation. In addition, the Commission should give more guidance on how to recruit researchers for the projects and should also take steps to improve the user and friendliness of the Cordis website.

43. The Commission is recommended to require contractors in the access to large scale facilities activity to keep supporting documentation for all costs incurred. For travel and subsistence costs a declaration should be obtained from the user that he/she has received no other financial support.

44. While the Commission is encouraged to use the agreed user fees approach for access to large scale facilities contracts with contractors, it should ensure that the user fees are correctly established. The advice of financial officers should be sought and fully documented before contracts are signed and follow-up audits should take place to ensure the user fees are correct.

45. The Commission should carry out its mid-term reviews in line with actual contract duration.

46. The Commission should continue the management of Marie Curie Fellowships in the department responsible for this activity in the TMR programme. It should also make it responsible for the management of the other fellowships in the specific programmes.

47. The Commission is encouraged to continue to offer good service in answering contractors' queries.

48. Should the Commission decide to further fund the Marie Curie Fellowship Association, it should ensure that adequate management controls have been put in place, and it should provide the Association with strong guidance on how to spend the money and also fix limits for expenditure in certain areas.

49. The Commission should ensure that if target criteria are included, particularly in the area of euroconferences, practical training courses and summer schools, documentation is provided by the contractor to ensure these criteria have been respected. It is also recommended that the Commission request, along with the cost statements, declarations from the contractor showing the total amount sponsored and the percentage of the total costs of the event this sponsorship has covered. This could also be used as an additional check on the accuracy of the costs declared.

50. The Commission should take steps to ensure on-the-spot audits are carried out as soon as possible on contractors in the Improving Human Potential (IHP) programme.

This report was adopted by the Court of Auditors in Luxembourg at its meeting of 19 July 2001.

For the Court of Auditors

Jan O. Karlsson

President

(1) Decision No 1110/94/EC of the European Parliament and of the Council of 26 April 1994 concerning the fourth framework programme of the European Community activities in the field of research and technological development and demonstration (1994 to 1998) (OJ L 126, 18.5.1994). Decision No 2535/97/EC of 1 December 1997 adapting Decision No 1110/94/EC for the second time (OJ L 347, 18.12.1997).

(2) Decision No 182/1999/EC of the European Parliament and of the Council of 22 December 1998 concerning the fifth framework programme of the European Community for research, technological development and demonstration activities (1998 to 2002) (OJ L 26, 1.2.1999).

(3) Council Decision 94/916/EC of 15 December 1994 adopting a specific programme of research and technological development, including demonstration, in the field of training and mobility of researchers (1994 to 1998) (OJ L 361, 31.12.1994).

(4) Annual Report (Chapter 9) concerning the financial year 1994, together with the institutions' replies (OJ C 303, 14.11.1995).

The commission's replies

SUMMARY

II. Generally speaking, the procedures for evaluating the fourth framework programme gave ample guarantee of transparency. The presence of an independent observer also provided an extra guarantee of proper application of the procedures.

In the cases mentioned by the Court, it is true that the minutes did not always fully reflect the different stages leading to the decisions of the full panel. This had no impact on the results nor on the financing of projects (see points 10 and 37).

In the fifth framework programme the whole evaluation procedure was improved by the use of a single evaluation manual adopted by the Commission and made available to all potential participants. Moreover, an open call for experts and a common expert database for all programmes have been set up. Finally, the status of independent observer has been codified and for some activities the panel's reporting system has been improved with the designation of a rapporteur responsible for drafting the reasoning behind the decision (see points 10 to 12).

In an area like research, where there is often a concentration of experts at the forefront of science, the potential risk of a conflict of interests cannot be excluded from the outset. Having said that, the two-stage evaluation structure and in particular the peer review of each proposal, individually to begin with and then by the panel as a whole, guarantees a balance between the potential risk of a conflict of interests and the quality of evaluations. This procedure has been improved considerably under the fifth framework programme by steps such as the coding of the status of independent expert and attaching terms of reference, a code of conduct and a declaration of impartiality duly signed by the party in question to the contract of each evaluator (see points 11 and 37).

III. Checks are being made to clarify, together with the two organisations, the cases mentioned by the Court of Auditors. Any measures needed will be taken (see points 17 and 42).

IV. The Commission services responsible for the negotiation of access to large-scale facilities contracts, including staff from both the relevant scientific and administrative/financial units, take all the necessary steps to ensure correct calculation of the unit cost, which in turn provides the basis for the user fees due by contractors. A number of access contract audits will be carried out to provide an assurance that the established system of user fees is correct and sound (see points 21 and 44).

The Commission has carried out the TMR mid-term reviews considering the effective duration of access contracts. The Commission considers technical reviews and the overall process of programme monitoring and evaluation as extremely valuable for the improved management of contracts and of the programme as a whole and has further strengthened this process in FP5 (see points 22 and 45).

V. Efforts have been made since the fourth framework programme to improve the system of grants (e.g. creation of a single entry point for the receipt of proposals). Harmonised evaluation procedures and criteria have also been put in place in the fifth framework programme. In the Commission's proposal for the new framework programme all activities relating to mobility, including Marie Curie fellowships, have been grouped under a single activity (see points 24 and 46).

VI. The Commission systematically checks the eligibility of costs (age, nationality) before making any payments. Other criteria of note (geographic balance, male/female parity, participation of less developed countries, representation of industry) are checked on the basis of replies to the questionnaires distributed to the participants (see points 29 and 30).

Furthermore, the Commission will consider how to incorporate checks on sponsorship in the elaboration of the rules for the next framework programme (see point 49).

VII. The Commission was interested to see the results of the survey on the TMR programme undertaken by the Court of Auditors among the beneficiaries/contractors. It notes the high proportion of responses and welcomes their generally positive view of all parts of the TMR programme (see points 33 to 36).

SELECTION OF PROPOSALS

Audit approach and general results

10. The two-stage procedure, as summarised by the Court of Auditors, rightly shows that the first stage was in preparation of the final evaluation by the full panel. As regards the minutes of the panel's decisions, the main point is that they correctly reflect the results of those decisions, namely, the classification in order of priority of the proposals that the panel would like the Commission to finance. This has always been the case even if in the cases examined by the Court of Auditors the minutes did not always show the details of the discussions that led to that particular order of priority. This does not compromise the final result of the evaluation.

11. The evaluation system under the fourth framework programme, with its common basic structure for each activity, took account, within a coherent legal framework, of the characteristics of the different activities (large-scale facilities, research training networks, Marie Curie fellowships, accompanying measures).

As said in point 10, the Commission stresses that the panels' discussions (plenary sessions) have always been minuted and that they reflect the final decisions of the panels strictly, even if in some cases the discussions leading to the decisions are not always fully documented.

In a programme covering all areas of science and requiring a peer review of projects, an expert may be asked to evaluate proposals in which his organisation is participating without there being any direct conflict of interests, especially where a big organisation is concerned (e.g. national research institutes). Where there is a potential conflict of interests, the mechanism of a two-stage evaluation procedure provides a good guarantee, particularly as this system has been improved in the fifth framework programme by steps such as the coding of the status of independent expert and attaching terms of reference, a code of conduct and a declaration of impartiality duly signed by the party in question to the contract of each evaluator.

Research networks

12. Particular attention has been attached in the fifth framework programme to the reporting of the discussions of the panel and a rapporteur has been appointed to provide consistency between the observations, scores and final classification adopted by the panel.

Access to large scale facilities

13. It is true that individual evaluators have sometimes ranked proposals without following precisely the guidelines given in the specific evaluation manual for large-scale facilities. Nevertheless, the final panel rankings obtained at the second meeting follow scrupulously such guidelines (e.g. that the top ranking is given only if all five criteria are marked as "high"). The Commission did in fact overview the evaluations to ensure the correspondence between the final marks and the actual ranking finally given by the panel. The evaluation procedures under FP5, on the other hand, are now simpler in this respect, in that primary evaluators are not required to rank proposals but only to mark them, the ranking being left entirely to the panel discussion.

Contract extensions

14. When selecting proposals from the third call under the fourth framework programme, preference was given to the method of using an amendment rather than a new contract for infrastructure support projects that already had a contract from the first call for proposals. This amendment stated that it did not finance projects already supported under the initial contract but rather complementary activities (Article 3). Consequently, the period of overlapping should not have posed any problems since there was no conflict between the activities being funded. On the issue of the use of funds, Article 3 also stated: "... no funds may be transferred between the additional contribution and the contribution referred to in Article 3(1) of the contract".

It is true however that a few contractors have expressed some confusion in relation to charging costs for the original contract and for the amendment in the same annual report and that payments concerning the original and additional financing could not be clearly differentiated. This is the reason why the system, despite its intention to facilitate a more continuous technical implementation of access contracts, has been abandoned in FP5.

15. As the specific evaluation manual for large scale facilities (LSF) clearly states: "... the chairman will ask the programme staff to provide factual information (...) concerning the implementation of contracts for access with any facility already funded under this or earlier programmes that have submitted proposals to the current round (...), including the results from [earlier] technical audits". In other words, knowledge of previous proposals, and of how they have performed, is considered beneficial to a better understanding of proposals. In any case, about one third of all expert evaluators must change from one evaluation round to the next. This standard rule has always been followed to guarantee a proper renewal of expertise in the panel.

The issue of the links between applicants and evaluators is very complex. The specific evaluation manual for the LSF clearly defines cases of "direct link" between an evaluator and an applicant. Such potential conflicts of interest are thus carefully avoided Some of the 11 cases quoted by the Court, however, do not fall under this category but are considered as "indirect links". Nevertheless, potential cases of "indirect link" are also very carefully examined, but demand a case-by-case analysis. In particular:

- the case of evaluators acting as referees in a user selection panel is not normally felt to constitute such a link. By definition, in fact, the members of a user selection panel are asked to act as independent experts (as in all "peer review systems"),

- the case of users may be rather difficult to assess. While a "regular" user, including someone who intends to become a user, is certainly to be excluded, users who have used a facility in the past but who have no plans to continue using it, may be considered, in fact, as the best judges of that facility.

Last but not least, it should also be pointed out that several other criteria are taken into account in order to minimise the risk of potential conflicts of interest. One of them is to avoid evaluators of the same nationality as the coordinator of a proposal. Of course, in a multi-disciplinary panel such as this, evaluators must also be chosen among those who have the necessary expertise. It is ultimately up to the panel, in its collegiality, to verify and discuss those cases where potential "indirect links" may be suspected.

The Commission agrees that "regular users" represent a case of direct link and has already brought such potential conflicts of interest to the attention of recent evaluation panels (see the latest report of the independent observer for the infrastructures action of FP5, available on the Internet).

CONTRACTS AND MANAGEMENT CONTROL

Research networks

17. The Commission has taken note of the Court's audit results and the two concerned institutions are being contacted in order to obtain full information on the eligibility of researchers financed under the TMR contract. The Commission will then reimburse costs only for those researchers that have been appointed in accordance with the terms of the TMR network contract, and where appropriate will make any necessary adjustments.

18. The scarcity of students in many scientific fields and the practical obstacles to transnational mobility, are two among the many factors that makes it difficult for the networks to recruit young researchers. Such issues, which are of wider significance, are addressed in a communication adopted by the Commission on 20 June 2001. This communication also addresses the problem of the dissemination of information on vacancies for research positions within Europe (proposal of an Internet portal, providing easy access to the relevant websites). Furthermore it is intended to launch a "raising awareness" campaign on the opportunities offered by the research training networks and other activities of the programme, especially towards the associated countries.

There is a general need for regular interaction between contractors and the Commission on various aspects of the implementation of the projects. Increased attention will be paid to the early warning of network contractors on questions relating to the appointment of young researchers, emphasising that the publication of posts on the Commission's website is only one of various promotional measures to be considered. Continued attention will also be paid to the improvement and user-friendliness of the Cordis website.

Access to large-scale facilities

20. The Commission agrees on the potential risk that travel and subsistence costs may be incorrectly charged and that appropriate warnings will be given so that contractors should verify whether such costs have not already been paid by the home institutions of the users. Such considerations will be taken into account in the current development of the new funding instruments for the next framework programme.

21. The adoption of "user fees" requires the negotiation of a "unit cost", the calculation of which rests on information provided by contractors during negotiation. This calculation is based on the annual direct operating costs of the facility, based on historic costs, and on the total amount of access provided over the same period. The Commission services responsible for the negotiation of contracts, including staff from both the scientific and administrative/financial units, take all necessary steps to ensure that such calculations are correctly established.

As the system of user fees was new in TMR, the negotiation procedures and the corresponding documentation have been much more carefully described in FP5 in order to improve the general understanding of this accountancy basis.

22. Large contracts extended (in some cases through ad-hoc modifications to the contract duration) until the start of FP5 (i.e. early 2000). Taking into account this overall time perspective the timing of the mid-term reviews was appropriate. Moreover, operating as a dedicated mid-term review panel allowed the expert reviewers to take a global view of the programme (including all contracts), which is extremely valuable for the improved management of the contracts and of the programme as a whole. This could only be achieved by conducting the review after the final call of the programme. For this reason, the new "transnational access" contracts of FP5 specify a "technical verification of the project" (Article 22 of the new standard contract) in order to allow a more flexible and efficient implementation of the monitoring process.

Training through research (Marie Curie Fellowships)

Follow-up of previous observations

23. The Commission is pleased to note the Court's observation.

Decentralisation of selection and management of Marie Curie Fellowships

24. The Commission is aware of the need for coordination and harmonisation of Marie Curie Fellowships. In the fourth framework programme, there was a unique entry point for all such fellowship proposals; however, there were no further coordination measures implemented.

In the fifth framework programme, fellowships were coordinated and harmonised under the responsibility of the improving human potential programme and, in particular, a common set of evaluation procedures, including minimum threshold for quality of proposals, was defined. As, however, budgets available within the thematic programmes varied substantially, success rates of the proposals submitted were in some cases also quite different within the limit of the respect of the quality of proposals.

To further coordinate and harmonise Marie Curie Fellowships in the future, the Commission has proposed, for the next framework programme, to operate all Marie Curie Fellowships within one activity line of one programme.

Taxation of Marie Curie Fellows

25. The Commission is aware of the problem of the differences in taxation applied to the fellows' income. In the beginning of FP4, the Commission set up a working group on this issue where all the Member States and associated States were represented. In the conclusions of this working group there was strong opposition from three Member States regarding a Community legislative initiative for an equivalent legal EC status for EC fellows. Within the context of the work on the European research area, a High-Level Expert Group on improving the mobility of researchers was set up by the Commission with representatives of the Members States. The High-Level Group has identified obstacles to mobility of researchers and has also noted taxation in this context. Many of the taxation differences arise from differences in status of the fellows in the Member States. Taxation is a difficult issue to address at Community level, as decision-making relies on unanimity and for some Member States taxation policy lies at the very heart of national sovereignty. This reluctance to discuss taxation issues has also been clearly seen in the course of the work of the High-Level Group.

These items are also addressed by the Commission in its forthcoming communication "A mobility strategy for the European research area" (COM(2001) 331 final, of 20 June 2001).

Marie Curie Fellowship Association

28. The final financial and scientific reports of the Marie Curie Fellowship Association (MCFA) have been submitted to the Commission, who is evaluating them. On this basis, all necessary measures will be taken. The Commission has already announced to the MCFA that should there be any future funding, improved management control and financial rules will have to be put in place as an a priori condition.

Accompanying measures

29. The Commission systematically checks the eligibility of costs (age, nationality) before making any payments. Other criteria of note (geographic balance, male/female parity, participation of less-developed countries, representation of industry) are checked on the basis of replies to the questionnaires distributed to the participants.

30. It is true that there are cases among the accompanying measures where the Commission is not the only source of funding. In such cases, the check asked for by the Court, namely whether total sponsorship did not exceed total costs, could indeed not have been done systematically, at the level of the regular cost statements. For the successor activity under FP5, the Commission has made sure that the cost statements include a requirement to report on total costs and the contracts exclude any profit for the contractor. The Commission has taken note of the Court's statement and will take it into account when elaborating the rules for the next framework programme.

On-the-spot controls by the Commission

31. The Commission accepts that early checks are imperative. For this reason subsequent to the period of 1996 to 1999 considered by the Court, the Commission has intensified its audit efforts and has in recent years increased the number of audits carried out on a random basis.

In the case of the TMR programme, 30 audits are in the process or have been in the process in the 4FP. Among these 30 audits:

- since 1999, 13 audits have been closed,

- five audits are ongoing,

- 12 are scheduled for start in 2001.

SURVEY RESULTS

33. The Commission was interested to see the replies by contractors to the two questionnaires sent by the Court of Auditors on the TMR programme. It notes the high proportion of responses and welcomes their generally positive view of all parts of the TMR programme.

34. The frequent involvement of facilities supported for access over consecutive programmes and also in related RTD projects and cooperation networks is due to the fact that, by its own very nature, the LSF action addresses facilities that are unique or rare and that provide essential services to the users.

35. The Commission welcomes the observation of the Court concerning the services they provide to their contractors, in particular on the general satisfaction concerning the procedures for calls and selections.

36. The Commission services endeavour to provide a fast and effective assistance to all queries from contractors and from the research community at large. This service is facilitated by the use of a dedicated website on the Internet, which is constantly kept up-to-date.

CONCLUSIONS AND RECOMMENDATIONS

Conclusions

Selection of proposals

37. In an area like research, where there is often a concentration of experts at the forefront of science, the potential risk of a conflict of interests cannot be excluded from the outset. Having said that, the two-stage evaluation structure and in particular the peer review of each proposal, individually to begin with and then by the panel as a whole, guarantee a balance between the potential risk of a conflict of interests and the quality of evaluations.

This procedure has been improved considerably under the fifth framework programme by steps such as the coding of the status of independent expert and attaching terms of reference, a code of conduct and a declaration of impartiality duly signed by the party in question to the contract of each evaluator.

The fact that the minutes do not give details of the discussions leading to the classification by order of priority of the proposals that the panel wishes the Commission to finance does not compromise the final result of the evaluation and is not synonymous with a lack of transparency in the respective funding. See also response to point 41 (a), (b) and (c).

Contracts and management control

39. Any variations in the success rates of the various programmes stem from the differences in funding made available in the specific programmes themselves.

Under the next framework programme the Commission has proposed that all Marie Curie Fellowships should come under a single activity.

Recommendations

Selection of proposals

41. (a) While agreeing with the auditors' comments that the conflict of interest is an important consideration in the selection of evaluators, it is by no means the only criterion. The Commission services have to examine the experts' details (CVs) carefully to ensure that only the most appropriate people, with the necessary experience and competencies, and other characteristics (academia/industry, gender, nationality, linguistic capabilities, etc.) are used. The open call for experts for the evaluation of proposals under the fifth framework programme and the common expert database for all programmes have already improved the openness and transparency of the process.

(b) The evaluation manual of the fifth framework programme contains specific steps for avoiding any potential conflict of interest. This is achieved by way of the contract concluded with the expert and the attached terms of reference, code of conduct and declaration of impartiality duly signed by the party in question.

Under the fifth framework programme, for most programmes and calls, experts are invited to a central location to carry out the evaluation. However, where time permits and where issues of commercial sensitivity are not a limiting factor, proposals may be sent to experts, either as electronic or paper copies, for them to carry out an initial evaluation at their place of work. If the latter alternative is used, it is made clear in any public supporting documentation for the particular call. For the future, the Commission is examining the possibility of using remote electronic proposal evaluation as used by a number of national authorities. Evidently any remote evaluation is subject to considerations of commercial confidentiality and the nature of the proposals being evaluated.

(c) Since the fifth framework programme standard procedures are used for the various decision levels. Experts examine the proposals assigned to them individually, filling in an individual evaluation sheet with their comments and marks. Once all the experts to which proposals have been assigned have completed their individual assessments for a given phase, a panel is brought together to discuss the marks awarded with a Commission official acting as moderator for the group. During this consensus meeting, the experts attempt to agree on a final mark for each of the groups of criteria and an overall mark for the proposal. They justify their marks with comments suitable for feedback for the proposal coordinator and agree on an overall evaluation summary report, which is signed by them. In the activity "networks and access" a rapporteur appointed by the panel is now responsible for drafting the reasoning behind the decision.

Contracts and management control

42. Already, under the TMR network activity the Commission has taken different actions to ensure that network participants are well-informed on the financial/administrative management of the network project. During the contract negotiation phase all network coordinators were invited to Brussels in order to discuss the implementation of the network contract and its financial aspects. In particular, special attention was given in these seminars to the discussion of the eligibility criteria for young researchers.

The Commission has also provided the network partners with "Guidelines for the preparation of cost statements". These guidelines serve as a quick reference on the basic contractual stipulations concerning eligible costs and summarise in particular all relevant aspects concerning the appointment of young researchers. The Commission has put all relevant documents on the Internet (http://www.cordis.lu/fp5/management/home.html) and will further improve the user-friendliness of the Cordis website.

43. The provisions concerning supporting documentation given in the model contract for access to large-scale facilities under TMR have been further developed in FP5, for example to include "records of the presence at the infrastructure of all users within the user groups benefiting under the contract, their nationality, their place of work and the access provided to them". Contractors will be asked to pay special attention to potential problems concerning travel and subsistence costs. Such considerations will be taken into account in the current development of the new funding instruments for the next framework programme.

44. The Commission has confirmed the system of user fees, which rests on the negotiation of a "unit cost" to be derived from the annual operating costs of the facility and the corresponding amount of "access" available, provides an efficient method of work. This system has been continued in FP5 and should be maintained in the next framework programme. The Commission services responsible for the negotiation of contracts, including staff from both the scientific and administrative/financial units, take all necessary steps to ensure that such costs are correctly established. It is foreseen to carry out a number of "access" contract audits to provide an assurance that the established system of user fees is correct and sound.

45. The Commission has carried out the TMR mid-term reviews considering the effective duration of access contracts. The Commission considers technical reviews and the overall process of programme monitoring and evaluation as extremely valuable for the improved management of contracts and of the programme as a whole and has further strengthened this process in FP5.

46. In the project for the next framework programme, the Commission has proposed to manage all Marie Curie Fellowships within one activity line under the responsibility of one directorate.

47. The Commission services endeavour to continue their tradition of providing reliable and effective services to their contractors and to the scientific community in general, including prompt and thorough replies to the queries received from applicants, contractors, users, and the scientific community at large.

48. The Commission agrees with the Court recommendation if the Association decides to ask for further financing.

49. Since the fifth framework programmes these checks are made systematically.

50. In a general way, the Commission has intensified its audit efforts and has in recent years increased the number of audits carried out on a random basis. The IHP programme, after the TMR programme (see point 31), is included in the general audit effort like any other RTD programme.