Table of contents

1.Introduction

Purpose and scope of the evaluation    

2.What was the expected outcome of the intervention?

2.1    Description of the intervention and its objectives    

2.2 Points of comparison    

3.How has the situation evolved over the evaluation period?

3.1 Policy measures taken    

3.2 Trends in relevant indicators    

4.Evaluation findings

4.1 To what extent was the intervention successful and why? (effectiveness, efficiency, coherence)    

Effectiveness    

Efficiency    

Coherence    

4.2 How did the EU intervention make a difference? (EU added value)    

4.3 Is the intervention still relevant?    

5.What are the conclusions and lessons learned?

5.1 Conclusions    

5.2 Lessons learned    

Annex I. Procedural Information

Annex II. Methodology and Analytical models used

Annex III. Details on answers to the evaluation questions (by criterion)

Annex IV. Overview of benefits and costs and, where relevant, Table on simplification and burden reduction]

Annex V. Stakeholders consultation - Synopsis report

Annex VI. Intervention Logic

Annex VII. Starting Points and Relevant Trends in Indicators and Policy Measures by Member State: Overview Tables

Glossary

Term or acronym	Meaning or definition
ACVT	Advisory Committee for Vocational Training
ALMP	Active Labour Market Policy
Cedefop	European Centre for the Development of Vocational Training
CIE	Counterfactual impact evaluation
CPD	Continuing Professional Development
DG EAC	Directorate-General for Education and Culture
DG EMPL	Directorate-General for Employment, Social Affairs and Inclusion
DGVT	Directors General for Vocational Training
EAEA	European Association for the Education of Adults
EaSI	EU Programme for Employment and Social Innovation
EBSN	European Basic Skills Network
EPALE	European Platform for Adult Learning and Education
EQF	European Qualifications Framework
ERDF	European Regional Development Fund
ESF	European Social Fund
ESIF	European Structural and Investment Funds
ETF	European Training Foundation
ETUC	European Trade Union Confederation
EU	European Union
Eurofound	European Foundation for the Improvement of Living and Working Conditions
Eurostat	Statistical office of the European Union
GDPR	General Data Protection Regulation
ICT	Information and communications technology
ILO	International Labour organization
ISG	Inter-service Steering Group
JTF	Just Transition Fund
NEET	Not in Education, Employment, or Training
NGOs	Non-governmental organizations
OECD	Organisation for Economic Cooperation and Development
PES	Public Employment Service
PIAAC	Programme for the International Assessment of Adult Competencies
RRF	Recovery and Resilience Facility
SMEs	Small and medium-sized enterprises
SRSP	Structural Reform Support Programme
STEM	Science, technology, engineering and mathematics
SWD	Staff working document
TSI	Technical Support Instrument
TVET	Technical and Vocational Education and Training
UN	United Nations
VET centres	Vocational education and training centre
WEC-Europe	World Employment Confederation-Europe
AT	Austria
BE	Belgium
BG	Bulgaria
CY	Cyprus
CZ	Czechia
DE	Germany
DK	Denmark
EE	Estonia
EL	Greece
ES	Spain
FI	Finland
FR	France
HR	Croatia
HU	Hungary
IE	Ireland
IT	Italy
LT	Lithuania
LU	Luxembourg
LV	Latvia
MT	Malta
NL	Netherlands
PL	Poland
PT	Portugal
RO	Romania
SE	Sweden
SI	Slovenia
SK	Slovakia

1.Introduction 

Purpose and scope of the evaluation

This document evaluates progress in relation to the Council Recommendation of 19 December 2016 on Upskilling Pathways: New Opportunities for Adults 1 (hereafter “the Recommendation”), adopted on the basis of Articles 165 and 166 of the Treaty on the Functioning of the European Union on education and vocational training. Evaluating the implementation and impact of European Union (EU) legislation is a basic requirement of the European Commission Better Regulation framework. The Recommendation itself asks the Commission to report to the Council assessing and evaluating “the actions taken in response to this Recommendation”. The results of the evaluation will feed into the policy debate to put in practice the European Pillar of Social Rights, namely the first and fourth principles 2 , building on the 2020 European Skills Agenda 3 and taking measures towards the 2030 headline skills target of 60% adults participating in learning every year. 4 This document is accompanying the Report from the Commission to the Council 5 .

In compliance with the Better Regulation Guidelines, the evaluation assesses the following criteria:

–effectiveness: to what extent the objectives of the Recommendation have been achieved through action by Member States and the Commission;

–efficiency: relationship between costs and benefits, relevant factors and proportionality of costs;

–coherence: whether the Recommendation remains coherent with other EU and national initiatives;

–EU added value: whether the Recommendation has generated additional value compared to what action at national level alone would have produced;

–relevance: whether five years after the adoption of the Recommendation (end of the evaluation period: 2021), its objectives and the measures recommended to achieve them are still relevant.

The evaluation is based on a mixed-method data collection approach supported by an external study (hereafter referred to as “supporting study”). 6  It combines a mapping of implementation measures based on extensive desk research, cross-checked with national experts and validated by public authorities 7 , with implementation case studies, a literature review, a public consultation and surveys targeting the organisations involved in the coordination and implementation of the Recommendation, organisations representing low-skilled adults and adults participating in initiatives implementing the Recommendation. It also included targeted consultation events and interviews with stakeholders at national and EU level as well as an analysis of relevant survey data. The geographic scope includes all EU Member States (EU 27). This evaluation covers the period from the adoption of the Recommendation in 2016 to 2021 8 included. 

It should be noted that a robust counterfactual impact evaluation of the Recommendation is not possible due to data limitations and the multitude of factors other than the Recommendation which may have influenced trends in relevant indicators. Moreover, impacts of any structural changes in national skills policies influenced by the Recommendation may take longer to unfold. Where relevant, this evaluation discusses alternative explanations for the observed trends, and presents stakeholders views to support their interpretation. Finally, responses to the public consultation or targeted consultations may not be representative for the relevant target populations or all EU Member States. Where possible, the evaluation triangulates results from several information sources to increase the robustness of its conclusions. See Annex II for further details on the evaluation methodology, its limitations and mitigation measures.

2.What was the expected outcome of the intervention? 

2.1 Description of the intervention and its objectives

Rationale

At the time of adoption of the Recommendation, labour market trends pointed towards a decreasing share of jobs of an elementary nature, an increased frequency of job changes, and the emergence of new ways of working 9 , increasing the relevance of having at least basic digital and other transversal skills. Moreover, demographic trends pointed to a shrinking workforce, with a risk of associated staff shortages. To mitigate the challenges resulting from these trends and increase productivity, investment in skills was considered essential.

It was hence a point of concern that around one quarter of Europeans in the 25-64 age group did not have an upper secondary education qualification, and similar shares of adults lacked basic literacy, numeracy and digital skills according to Organisation for Economic Cooperation and Development’s (OECD) Survey of Adult Skills (PIAAC) 10  (cf. Section 2.2). Moreover, adults lacking basic skills often have less opportunities to develop their skills throughout their working life, as such skills are taken for granted by many employers. 11 They risk getting stuck in a “low skills-poor jobs trap” 12 , calling for adequate policy action to support basic skills acquisition.

The 2016 New Skills Agenda 13  therefore called for strengthening basic skills as the foundations of further learning and career development. The Recommendation was one of its main legislative proposals.

Objectives

The general objective of the Recommendation was to reinvigorate and provide guidance to Member States’ policies supporting basic and further skills acquisition among adults with low skills, in order to enhance their employability and active participation in society. Since most members of Europe's workforce in the next two decades are already adults today, such policies were seen as a crucial complement to EU policies to reduce early leaving from education and training that reduce the “flow” into adulthood with low basic skills. 14  

The Recommendation’s specific objective was to offer adults with a low level of skills, knowledge and competences, for example those who have left initial education or training without completing upper secondary education or equivalent, an opportunity to acquire a minimum level of literacy, numeracy and digital competence and/or acquire a wider set of skills, knowledge and competences relevant for the labour market by making progress towards a qualification at European Qualifications Framework (EQF) level 3 or 4 15 depending on national circumstances. 

The Recommendation builds on other Recommendations in the area such as the 2012 Council Recommendation on validation 16 and the 2016 Council Recommendation on the integration of the long-term unemployed. 17 It supports the implementation of the United Nations (UN) sustainable development goal 4 on quality education and lifelong learning opportunities for all 18 , in particular the targets on digital skills and universal literacy and numeracy.

 Recommendations

To motivate more low-skilled adults to engage in training and reach these objectives, it was seen as key to assess the skills adults already have (including those acquired through non formal and informal learning), guide them towards learning opportunities adapted to their needs, and validate and recognise the skills thus acquired. Member States were hence recommended to offer a three-step personalised support via an integrated upskilling pathway to adults in the target group.

	The three steps are:
	1) Skills assessment – aimed at identifying the existing skills of adults and any gaps in the skills set that need to be filled.
	2) Provision of a tailored and flexible learning offer – providing a learning offer which fills the specific skills gaps identified through the skills assessment. The offer could be training in literacy, numeracy or digital skills, or a wider set of skills, knowledge and competences, relevant for the labour market and active participation in society.
	3) Validation and recognition – assessing and certifying the knowledge, skills and competences, building on validation arrangements, and encourage their certification towards a formal qualification.

Member States were recommended to implement these three steps by:

·Identifying priority target groups 19 for the delivery of the Recommendation at national level;

·Ensuring effective coordination and partnership to implement the Recommendation, through clear responsibilities and by engaging stakeholders;

·Implementing outreach, guidance and support measures to raise awareness on the benefits of upskilling, provide guidance and/or mentoring services to support learners’ progression through all steps of the process, consider the design and implementation of support measures to adults or employers that address obstacles to participation, and support the initial training and Continuing Professional Development (CPD) of staff engaged in delivery of the Recommendation.

The European Commission was recommended to:

·Support the use of European funding programmes in the area of skills development;

·Facilitate cooperation, information sharing and mutual learning between Member States and promote the use of existing competence frameworks to support the implementation of the Recommendation;

·Support and carry out relevant research and analysis on adult learning and skills assessments.

Annex VI presents a graphical overview of the intervention logic, summarising how the recommended actions of the intervention were expected to deliver on the objectives and create results and impact in the form of more basic skills development among low-skilled adults, enabling employment and a more active participation in society.

2.2 Points of comparison

A first category of points of comparison relevant for the evaluation of the Recommendation concerns quantitative indicators.

Participation of adults in learning can help them to develop basic skills. In 2016, 10.3% of adults aged 25-64 participated in adult learning in the 4 weeks prior to the survey, with wide variation across countries. 20 Six countries had a participation rate below 5% (BG, EL, HR, PL, RO, SK) and seven countries had a rate above 15% (DK, EE, FI, FR, LU, NL, SE). By contrast, on average only 4.0% of low-qualified adults 21 participated in learning. This participation gap was recorded for all EU Member States for which data are available, although with considerable differences in participation levels: while 16 Member States had participation rates of the low qualified of under 5% (BE, CY, CZ, DE, EL, ES, HR, HU, IE, IT, LV, LT, MT, PL, PT, SI), participation was 19.7% in DK, 19.3% in SE and 12.9% in FI.

9.4% of unemployed adults had a recent learning experience in 2016- slightly below the rate for adults overall, in spite of the greater need for skills acquisition by many unemployed adults, and the greater availability of time for learning. 22 However, there was even greater variation across Member States in the learning participation of unemployed adults as compared to adults overall: Their participation ranged from 3% or less in SK, HU and HR to 20% or more in DK, FI and LU and 43% in SE.

Low levels of digital skills represent a particularly vital gap in an adults’ basic skills set, as basic digital skills are required both in the majority of workplaces and for full participation in civic life today. 44% of adults aged 25-64 in the EU27 lacked basic digital skills in 2016, ranging from less than 30% in LU, DK, NL, FI, SE and DE to 55% or more in RO, BG, CY, PL and IE. 23 A lack of basic digital skills was particularly wide-spread among the low-qualified: 77% of adults in this group lacked basic digital skills, and 90% or more in 8 Member States (BG, CY, RO, PL, HR, EL, SK, LT).

A source of more comprehensive information on adults basic skills is the OECD’s PIAAC: in the 20 Member States that participated in the first cycle of PIAAC between 2011 and 2017, around a quarter of adults lacked basic numeracy and/or literacy skills: 16% of 25–64-year-olds had both low levels literacy and numeracy skills, 4.4% have only low levels of literacy, and 7.3% had only low levels of numeracy skills. 24  

In 2016, 23.4% of EU adults aged 25-64 had a low educational attainment, defined as not having completed upper secondary schooling- ranging from less than 10% in LT, CZ, SK, PL, and LV to 40% or more in PT, MT, ES and IT. 25  In many Member States, nowadays more young people obtain an upper secondary qualification than they did in previous decades. Hence, the educational composition of the 25-34 year age group provides insights into the more recent conditions in national education systems: 16.8% of adults aged 25-34 had a low educational attainment in 2016, with less than 10% in PL, SI, HR, LT, SK, CZ, IE and FI, but still above 30% in ES, MT and PT.

Since a key objective of the Recommendation was to improve the employability of low-qualified adults through skills acquisition, labour market indicators provide further reference points: In 2016, 16.3% of low-qualified EU adults aged 25-64 were unemployed, about double the rate of all adults of 8.2%. 26 The employment rate provides a broader measure of labour market attachment, as it also captures how many adults have left the labour force altogether. In 2016, 53.3% of low-qualified adults aged 25-64 were employed, 19 percentage points below the overall employment rate of 72.3%. 27  In countries with a lower proportion of low qualified adults, their unemployment rate tended to be higher and their employment rate tended to be lower than in countries with higher proportions of low-qualified, pointing to specific and multiple labour market disadvantages the remaining low-qualified adults are confronted with in these countries.

A second category of points of comparison relevant for the evaluation concerns the policy measures to support adults basic skills development that were already in place at the time of the introduction of the Recommendation in 2016. On the basis of a mapping by national experts, validated by national authorities, the supporting study accompanying this evaluation identified 58 such policy measures across the EU-27 that were in place in 2016. 28    Figure provides an overview of the number of Member States in which measures implemented before 2016 covered the three steps of the Recommendation, and the degree of implementation.

Figure 1: Degree of implementation of the three steps of the Recommendation in 2016
(Number of Member States by degree of implementation)

Source: Section 2.2.2 of the supporting study.

This mapping finds that prior to the adoption of the Recommendation, for none of the three steps, the identified measures implemented the step fully or to a large extent in the majority of Member States. The least implemented step was validation and recognition (with no relevant identified measures in BG, HR, CZ, DE, HU, IE, IT, LV, LU, MT, PL, SK) 29 , followed by the provision of a tailored and flexible learning offer (with no relevant identified measures in BE, BG, CZ, IT, MT, ES) and skills assessment (with no relevant identified measures in BG, HR, CZ, LV, MT and PL). When also considering measures implementing the step to some extent, relevant measures existed in the majority of Member States.

The study supporting this evaluation also groups the Member States by their estimated need for implementing the Recommendation in 2016- separately for each step, and at an overall level (see Section 2.2.2 of the supporting study). The overall implementation need of a Member State was estimated on the basis of the policy measures already in place (the fewer measures already in place, the higher the remaining need for implementation), and on the basis of the share of low qualified adults and their adult learning participation (the worse a Member States score on these indicators, the higher the remaining need for implementation). The study adopted this “two pillar approach” to the overall implementation need in order to capture both the existence of relevant policy measures, and their effective outreach in terms of the share of adults reached by them as reflected in the quantitative indicators.

This mapping suggests that in 2016, 14 Member States were in the “high need for change” category (BE, BG, HR, CY, CZ, EL, ES, HU, IE, IT, MT, PL, RO, SK,), six Member States had a medium need for change (DE, LV, LT, LU, PT, SI), and seven Member States were in the “lower need for change” category (AT, DK, EE, FI, FR, NL, SE). 30 While this categorization is coarse (and also many Member States in the “lower need for change” group are at some distance from the best-performers in terms of the quantitative indicators considered), it helps to highlight in which Member States implementing the Recommendation was expected to bring more progress. 

On the basis of this description of the intervention and these points of comparison, it is possible to sketch what different degrees of success of the Recommendation over the evaluation period would look like 31 : in a low impact scenario, the Recommendation is not seen as a relevant or helpful reference point by national stakeholders, and there would be no clear progress in implementing the three recommended steps. In a moderate impact scenario, the Recommendation is seen as a helpful reference point for national reforms, and some progress in implementing the three steps have been taken- however, not yet in all Member States with medium or high need for change, and not yet at a sufficient scale in view of the objectives. In a high impact scenario, there is clear evidence for additional measures being implemented at scale in the Member States, resulting in an increasing participation of low-skilled adults in skills assessment, learning and validation opportunities.

3.How has the situation evolved over the evaluation period? 

3.1 Policy measures taken

The supporting study identified a total of 104 implementation measures in place in 2021, up from 58 in 2016 at the start of the evaluation period. 32  Figure 2 shows that there has been an increase in the number of measures that implement each of the three steps from the Recommendation fully or to a large extent since 2016. The increase was most pronounced in measures that fully or to a large extent implement validation and recognition (+4 Member States- BE, BG, CY, EL), followed by the provision of tailored and flexible learning (+3 Member States- DK, EE, PL) and skills assessment (+2 Member States- BE, PL).

Figure 2: Degree of implementation of the three steps from the Recommendation- evolution over the evaluation period- Number of Member States, 2016 vs. 2021

Source: Section 3.2 of the supporting study.

At the end of the evaluation period, all Member States had introduced skills assessment measures at least to some extent. One country implemented new measures improving their existing skills assessment process (BE) and in six Member States skills assessment mechanisms were introduced (BG, CZ, HR, LV, MT, PL).

All Member States except for three (CZ, ES, IT) have now in place measures for a stronger tailored provision of learning to low-skilled adults. Six Member States have either improved their existing systems by strengthening or reforming existing policies (DK, EE, PL) or have introduced new measures to integrate tailored provision of learning into their support for the low-skilled (BE, BG, MT).

In spite of the progress made since 2016, validation and recognition of skills acquired remains the least developed step of the Recommendation. Eight Member States have not yet implemented relevant measures to establish validation and recognition in their upskilling systems (CZ, HR, HU, IE, LU, MT, PL, SK). In other Member States, some progress is evident: five Member States have introduced systems for validation and recognition of skills acquired during the upskilling pathway (BE, BG, DE, IT, LV), two have improved their existing systems by reforming existing policies (CY, EL), and discussions have started in SK.

In summary, the supporting study identified improvements in 14 Member States compared to the situation in 2016. Member States deemed to have the higher need for implementation in 2016 were more likely to implement reforms over the subsequent years: progress was found in two of seven Member States with lowest need for change in 2016 (29%, namely DK and EE), two of six Member States with medium need for change (33%, namely DE and LV), and 10 of 14 Member States with greatest need for change (71%, namely BE, BG, CY, CZ, EL, HR, IT, MT, PL, SK). 33  

No data are available on the number of low-skilled adults reached by these implemented measures, but a 2019 stocktaking report 34 points to a limited scale. The report took stock of implementation measures reported by the Member States to the Commission by mid-2018, as indicated in paragraph 16 of the Recommendation. It concluded that “in the vast majority of cases, the measures outlined by Member States target only a few thousand individuals”, and that “taken as a whole, the measures proposed by Member States would not reach a significant proportion of the 61 million adults 35 in need of upskilling“ (p. 28).

3.2 Trends in relevant indicators

From the quantitative “points of comparison” introduced in Section 2.2., trends over the 5 year-period from 2016 to 2021 can be assessed for the participation in adult learning of low qualified adults, the share of adults with at least basic digital skills, the share of low-qualified adults, as well as their unemployment and employment rates. A caveat is that no data on trends in directly measured adult basic skills are available 36 , and not all implementation measures can be expected to lead to formal qualification. However, reinforced basic skills training not leading to a formal qualification would still be reflected in adult learning participation, and can also be expected to influence the other indicators indirectly.

Learning participation among low-qualified adults increased slightly over the evaluation period from 4.0 to 4.3% in the 4 weeks prior to the survey. 37  However, the learning participation of low qualified adults increased in only 13 of 22 Member States with available data for both periods (59%), most noticeably in FI, SE, NL and EE. This compares to a slightly higher absolute increase in adult learning participation among all EU adults, from 10.3% in 2016 to 10.8% in 2021. 38 Participation in learning among adults overall increased in 19 Member States (70%), with the most pronounced increases taking place in NL, SI, IE, MT, SE, ES and FI. 39  There are hence no signs of a systematic closing of the adult learning participation gaps between low qualified and other adults. Uncertainty for these comparisons of 2016 and 2021 data on adult learning participation levels comes from an improved measurement of adult learning in the Labour Force Survey from 2021 (possibly increasing measured participation), and from a possible continued effect of COVID-19 safety measures (possibly limiting participation). 40  

Among unemployed adults, there was a more pronounced increase in adult learning participation, from 9.4% in 2016 to 12.7% in 2021. 41 Participation increased in 20 of 27 Member States, with the most pronounced increases taking place in LU, EE, NL, IE and PT. Since unemployed adults do not receive support for adult learning from an employer, their participation is particularly sensitive to the degree of public support that is provided to them, notably by public employment services. However also in 2021, there was still great variation in adult learning participation rates of the unemployed across Member States, with participation ranging from 4% or less in HU, EL, CZ and HR to 30% or more in NL, DK and LU and 47% in SE.

The share of adults aged 25-64 with at least basic digital skills improved marginally, from 56% to 58% between 2016 and 2019. Increases could be observed in 20 of 27 Member States, and were most pronounced in IE, MT, EL, LT and PT. The 2021 value stood at 57%, consistent with a picture of limited overall progress, although the 2021 value is not perfectly comparable to previous years due to an update of the methodology for calculating the Digital Skills Indicator to reflect the Digital Competence Framework 2.0.. 42  Among low qualified adults, the share with at least basic digital skills slightly increased from 23% to 25% between 2016 and 2019, and hence remained far below average. 

The share of low qualified adults aged 25-64 decreased from 23.4% in 2016 to 20.7% in 2021, with decreases across all Member States except DE. 43  This corresponds to a decrease by about 7.4 million adults from 56.1 million in 2016 to 48.7 million low qualified adults across the EU in 2021. 44 Also in the age group of 25-34 years, the share of low qualified adults decreased from 16.8 to 14.8%, with decreases in all Member States except CZ, DE, PL, FI. The decreases also in this younger cohort suggest that the decrease in the broader 25-64 age group can be partly, but not exclusively, explained by the retirement of cohorts with lower educational attainments. However, the decrease may be also influenced by improvements in initial education and training systems leading to less early school leaving without upper secondary schooling, as opposed to Upskilling Pathways implementation measures.

Survey data do not suggest a clear increase in the rate at which low-qualified adults have completed a “second chance” upper secondary schooling pathway over the evaluation period. A first indicator is the rate at which low-qualified adults aged 25-29 complete such a pathway over a 5 year-period: in 2011, 18.4% of the 25-29 year olds had not yet completed upper secondary schooling. By 2016, this share had declined to 17.5% of the same cohort (then 30-34 years old), suggesting that an additional 0.9 % of that cohort had acquired upper secondary schooling 5 years later. Comparing the years 2016 and 2021, the corresponding share declined from 16.0% to 15.3%, and hence a comparable (yet slightly lower) 0.7% of adults had acquired upper secondary schooling over the five year-period. 45 A second indicator is the share of all adults with upper secondary education who have completed it aged 25 or above 46 : this share was 7.4% in 2016 and 7.6% in 2021, and hence also comparable (yet slightly higher) at the end of the evaluation period. In 2021, this share ranged from 2% or less in BG, EL, SK, HR, RO, CY and CZ to 24% or more in PT, DK, NL and FI. This highlights that at the end of the evaluation period, there was still a stark heterogeneity across Member States in the likelihood of still receiving upper secondary schooling through a “second chance” pathway after initial education age. 47

The labour market situation of low-qualified and other adults alike improved over the evaluation period, but without clear signs of a closing gap in labour market opportunities. 48  The unemployment rate of low qualified adults aged 25-64 decreased from 16.3% to 12.7% between 2016 and 2021, with decreases in 20 of 27 Member States (most pronounced in LT, EL, CY and CZ, with decreases by 7.5 percentage points or more). 49 The employment rate of low qualified adults increased from 53.3% to 55.6%, with increases in 23 Member States (most pronounced in CZ, LT, MT and PL). However, these trends reflect broader improvements in EU labour markets as opposed to a consistent decrease in the disadvantage of low-qualified adults: while the drop of the unemployment rate was more pronounced among low qualified adults than among all adults (where it decreased from 8.2% to 6.2%), the increase in employment rates was somewhat less pronounced compared to all adults (where it increased from 72.3% to 75.3%).

4.Evaluation findings 

4.1 To what extent was the intervention successful and why? (effectiveness, efficiency, coherence)

The analyses suggest that the intervention has overall been moderately successful: on the upside, Section 3 shows that some Member States have made progress in implementing the three steps from the Recommendation- and stakeholders credit it with stimulating developments at national level, view the Recommendation as internally coherent and estimate that implementation actions have been efficient in the sense that their benefits outweigh their costs. However, implementation actions taken up to 2021 tend to lack scale and integration across the three steps of the Recommendation, preventing them from fully reaching its objectives.

Effectiveness

The available evidence paints a picture of overall moderate effectiveness of the Recommendation: on the one hand, Section 3 shows that some Member States have made progress in implementing the three steps from the Recommendation, and there are moderate positive trends in relevant indicators. Some stakeholders view the Recommendation as a catalyser for renewed focus on support for adults to acquire basic skills 50 and consider the three step-approach as a useful reference point. This is also reflected in the public consultation responses summarized in Figure 3, which shows that a majority of respondents indicate a “fairly” or “very” large degree of implementation for all features of the Recommendation. On the other hand, the mapping conducted by the external supporting study does not find substantial improvements concerning the implementation of the three steps of the Recommendation over the evaluation period in 13 Member States 51 , including in 8 Member States 52  where it finds that there was medium or high need for change at the end of 2016. In some Member States, the lack of substantial improvements over the evaluation period is due to existence of advanced measures already in 2016. 53   Where Member States have taken implementation measures, these measures often lack scale and coordination and cannot be linked to improvements at a “macro” level (as reflected in representative surveys), even where they are successful at a “micro” level (as reflected in positive evaluations of specific upskilling projects). This is consistent with the finding of an often limited scale of implementation measures in the 2019 stocktaking report. 54  

Figure 3: To what extent have the following features of the Recommendation been implemented in your country/the EU?

Source: Public consultation, cf. Annex 6 of the supporting study.

Skills assessment offers of some sort are wide-spread across Member States 55 , but require more attention to staff professionalisation and capacity and a non-stigmatising integration with the other steps to realize their full potential. 46% of respondents to the survey of coordinating and implementing organisations 56 that was conducted as part of the supporting study indicated that skills assessments are used “not at all” or “to a small extent” by priority target groups. This is consistent with the 53% of respondents to the survey of organisations representing low-skilled adults 57 who held the view that skills assessments were used by the target group at most to a small extent. Obstacles to a more effective implementation of the skills assessment step include a lack of a formal and agreed process of how skills assessments should be undertaken, an associated lack of specific skills and available time of career counselling staff, and a lack of awareness of the benefits of skills assessments by the target group itself. 58  Targeted consultations pointed to a risk of stigmatization of potential beneficiaries where the skills assessment step is communicated as a “skills audit” to identify a lack of skills 59 : the skills assessment step can be communicated in a more positive and motivating way by emphasising existing skills to build upon, and presenting the skills assessment step as a pathway towards more effective support concerning the other two steps of the Recommendation.

Eight Member States have taken steps to improve the provision of tailored and flexible learning offers to low-skilled adults, but challenges remain due to insufficient funding to support the diverse target groups and barriers to access and take-up. 60  In the survey of coordinating and implementing organisations, “tailored learning offer” was the feature most often selected as “reformed or implemented after 2016”. 73% of respondents to this survey indicate that learning offers are now tailored to the needs of target groups to a large or fairly large extent, with similar levels of agreement in the survey of organisations representing low-skilled adults (69%) and in the public consultation 61 (60%). According to stakeholders, a first characteristic of effective approaches is their flexibility in the delivery modalities, combining a flexible timetable and enrolment schedule with hybrid provision online and offline, with a suitable combination of home-, classroom and workplace-based learning. A second characteristic is their modular nature, breaking down the learning content into smaller chunks that allow for visible progress during a self-paced and learner-led learning pathway. 62 Finally, they are backed by sufficient funding to operate at the scale that is needed to make tailored offers to the diverse target groups.

Implementing measures cover basic skills in addition to job-specific skills, and stakeholders point to a need for attention to put sufficient focus on basic skills to ensure accessibility of offers for the target groups of the Recommendation. For instance, the supporting study reports examples of implementation measures focusing on literacy in 20 Member States. 63  Stakeholders highlighted the importance of adults basic skills development as a precondition for labour market success and further learning of job-specific skills 64 : such skills are taken as granted by many employers, highlighting the specific responsibility of public support offers in this area. Also the 2019 stocktaking report of implementation measures concluded that a stronger embedding of basic skills provision into skills assessment and training offers targeting low-skilled adults remains a point of attention. 65

Eight Member States have taken steps to improve their skills validation and recognition provisions in relation to the implementation of the Recommendation 66 , but they often remain insufficiently integrated into the rest of the education and training sector. Provisions for the validation of prior learning are often characterized by low awareness and take-up among adults. A successful approach to skills validation and recognition depends on a high level of awareness and trust among individuals, education and training providers and employers. Where offers only exist at a project level or in specific corners of the adult learning system, insufficient trust and awareness often lead to low take-up, in particular among low-skilled adults. These conclusions are in line with the 2020 evaluation of the 2012 Council Recommendation on the validation of non-formal and informal learning, which calls for more support to individuals to promote take-up of existing offers and more strategic cooperation to build awareness and trust. 67  

Member States implementation measures identify a large number of priority target groups, but often do not succeed in effectively reaching all adults who would benefit from them. Around 45% implementation measures mapped by the supporting study identify their target group primarily on the basis of a low level of basic skills, and 40% primarily on the basis of the employment status. Mapped measures also target specific vulnerable groups such as migrants and refugees (AT, BE, DE, ES, IE, IT, LU, MT, PL, SE), people with disabilities (AT, BE, BG, DE, IE, IT, LV, PL, RO, SI), older workers (BE, BG, DE, LT, LU, IT, PL, RO, SI) or the long-term unemployed (AT, BG, FR, HU, IE, LT, RO). 68 However, stakeholders report that even though relevant target groups have been identified, challenges remain in effectively reaching them. 69 These include a lack of awareness of opportunities among the target groups, financial barriers to participation, and a lack of motivation of the target group, including due to negative previous experiences with learning and a perceived stigma of “being an adult that needs to return to school”. 70 Stakeholders also report that digitalisation has greatly increased the target group of the Recommendation in view of the prevalence of low digital skills 71 , rendering traditional definitions of priority targets groups incomplete. 72  

Career guidance and outreach efforts exist in all Member States, but with varying degrees of effectiveness. Guidance provided by a counsellor of the Public Employment Service is widely available across EU Member States, but suffers from low levels of awareness, may be perceived as intimidating 73 , and is not always available to low-skilled who are not unemployed. 74 In the survey of organisations representing low-skilled adults, 53% said that awareness amongst the target group was missing or low. In the survey of adult learners 75 , the greatest proportion of respondents said that they learned about their adult learning programme through friends (42%) and only around 10% cited national websites or a guidance provider, consistent with a limited effective outreach of organized career guidance and outreach efforts. The provision of outreach and guidance services to employers to encourage them to support employees has been somewhat inconsistent and piecemeal, having detrimental effect in particular on SMEs. Effective outreach examples are often characterized by integration with the broader system of support to adult learners while at the same time ensuring local level cooperation. 76 Reaching potential learners with complex needs arising from multiple vulnerabilities requires a holistic provision of support including psychological support and active outreach 77 , with associated funding for staff training and infrastructure. 78

While Member States have delivered implementation measures in cooperation with relevant stakeholders, their coordination often still suffers from a lack of clearly defined responsibilities. The stakeholders most commonly involved in implementing the Recommendation are public employment services, social partners, NGOs and local actors including public libraries 79 , but include other stakeholders such as private employment services. 80  In the public consultation, around half of respondents agreed that the implementation actions since 2016 have been delivered in cooperation with the relevant stakeholders to a “fairly large” or “large” extent. 81  The consultations point to stakeholder involvement as a success factor for implementation. 82  Experts consulted during the validation workshop and the interviews on the EU level expressed that there is room across the EU to involve social partners more in implementation, in particular in evaluation and monitoring of measures. Coordinating bodies for adult learning policies exist in all Member States except CZ, DK, SI, SK and SE 83 , and stakeholders report that the implementation of the Recommendation has in some cases improved the cooperation between ministries of education and labour. 84  However, existing arrangements often do not yet ensure effective coordination, with persisting coordination challenges in particular between the ministries of education and labour, and between national and regional authorities. 85 Unclear responsibilities create a risk that some target groups receive insufficient support, make it difficult for adults to navigate the landscape of available support, and create uncertainty for civil society stakeholders who are looking for long-term cooperation partners.

Reflecting the challenges with an effective coordination of the involved stakeholders, there is significant scope for improvement in the integration of the three steps into a comprehensive upskilling pathway. Stakeholders highlighted the importance of promoting smooth transitions between the three steps to encourage each individual step’s uptake and effectiveness: for instance, the skills assessment step aims to identify tailored learning offers, and tailored learning offers combined with validation of informally acquired skills can allow adults without completed upper secondary schooling to obtain a formal qualification. A joined-up approach between all three steps, reflected also at a strategic level, 86 is also crucial to ensure that adult learners feel supported and stay committed to their upskilling pathway, but is often not yet a reality. Some stakeholder point to a need for an individualized and holistic approach to support for the low-skilled adults furthest away from learning, integrating support concerning the three steps with social services and labour market programmes. 87

Consistent with the picture of a limited progress in terms of scale, integration and outreach of implementation measures, there is no evidence of an overall acceleration in positive trends in the relevant indicators since 2016, and there are no systematic links between such trends and implementation measures. For instance, the number of adults aged 25-64 without upper secondary education in the EU decreased by 13.2% between 2016 and 2021, from 56.1 million to 48.7 million adults- corresponding to a decrease at approximately the same pace as could be observed over the 5 years preceding the introduction of the Recommendation (2011-2016), when it decreased by 13.3% from 64.7 to 56.1 million adults. 88 This suggests that the decrease mostly happened because older cohorts (with lower educational attainment levels) have dropped out of this age group and because of less early school leaving from initial education, as opposed to implementation measures operating at a significant scale. The supporting study concludes that also at Member State level, greater implementation progress as identified through the mapping of policy measures in Section 3.1 has not consistently been associated with stronger improvements in relevant indicators. 89 However, a caveat is that these quantitative analyses are only able to capture short to medium run impacts of the Recommendation, whereas further possible indirect impacts through an influence on national policy discussions (cf. Section 4.2) would likely only materialize over the longer run.

Efficiency

No strong conclusions on efficiency can be drawn since Member States have not systematically collected data on the costs and benefits of implementation measures. However, stakeholders consider the implementation costs as proportionate to the implementation benefits: in the survey of coordinating and implementing organisations, 64% of respondents believed that the costs are proportionate to the benefits (15% to a great extent, 31% to a fairly large extent, and 19% to a small extent), 36% felt unable to comment, and no respondent felt that the costs were not proportionate to the benefits. Targeted stakeholder consultations corroborate the view that implementation actions have been cost effective, independently of the starting point of a Member State. This reflects low overall implementation costs in most Member States and comparatively large benefits- not only in terms of direct upskilling measures, but also in the form of increased awareness and stakeholder discussions on the issue of adults basic skills at national level. However, efficiency of implementation is perceived to vary with the degree to which clear responsibilities are assigned between implementing actors, the degree to which processes are in place to select quality training providers, involve specifically trained staff and cooperate with employers, as well as the degree to which monitoring and evaluation is used to improve delivery.

Implementation costs were mostly incurred by State, regional and municipal administrative bodies. Respondents to both the survey of coordinating and implementing organisations and the survey of organisations representing low-skilled adults indicated that incentives were provided to a great/large extent to education providers (41% and 45% respectively), to employers (38% and 37% respectively), and to individuals (31% and 33% respectively). 90 The survey of adult learners found limited costs for adults themselves: most programmes were offered free of charge for them, and a minority mentioned costs of usually less than 100€ for course fees or course supplies. However, stakeholder interviews pointed to more significant cost to learners related to validation processes.

EU funds, in particular the European Social Fund (ESF), have played a dominant role in funding the implementation of the Recommendation during the evaluation period. The ESF allocated €42 billion to investment priority 10.iii (enhancing equal access to lifelong learning) for the period 2016-2020, which includes implementation measures of the Recommendation in several Member States that aim to increase adult participation in learning, enhance the basic skills of low qualified adults, upskill workers to secure their professional paths, and help people getting back to work. 91  In the targeted interviews conducted for the supporting study, national stakeholders in most Member States 92  underlined the crucial role of the ESF in funding implementation of the Recommendation in their Member State, confirming the findings of the 2019 stocktaking report of planned implementation measures. 93 Additional EU funding sources supporting the implementation of the Recommendation include the European Regional Development Fund (ERDF), Erasmus+, the EU Programme for Employment and Social Innovation (EaSI), Horizon 2020 (the EU's research and innovation funding programme from 2014-2020) and the Structural Reform Support Programme (SRSP, succeeded by the Technical Support Instrument (TSI) in 2021). Figure 4 summarizes the responses to the public consultation concerning EU funding support: it confirms that several EU funding programmes have been available to support implementation, whereby agreement that the contribution has been “fairly” or “very” large is highest for the ESF, the ERDF and Erasmus+.

Figure 4: Contribution of EU funding programmes to Upskilling Pathways implementation

Source: Public consultation, cf. supporting study Annex 6.

Factors limiting efficiency of EU funding relate to a perceived high administrative burden in particular for smaller beneficiaries, and to the time-limited and project-based nature of EU funds which limits the time available for institutional learning to determine the most efficient ways of supporting the complex and heterogeneous target group of low-skilled adults. However, the 2021 evaluation of the 2014-2018 ESF support to education and training 94 concludes that in spite of these challenges, the overall benefits of ESF support over this period have outweighed its costs. 95 The Recovery and Resilience Facility (RRF) introduced in 2020 in response to the COVID-19 pandemic is a further significant source of EU funding for implementation measures. While disbursements only started in the end of the evaluation period and will continue until 2026, several national policymakers pointed to an important role of upcoming reforms and investments funded by the RRF for the implementation of the Recommendation. 96

Stakeholders report a range of benefits from implementation measures for individual learners, employers, and society at large. In the survey of coordinating and implementing organisations, approximately nine of ten respondents considered that the measures introduced in the last five years contributed to individuals gaining a qualification (94%), access to further education (93%) or to employment (92%), and basic literacy, numeracy and digital skills (89%). The positive assessment was shared in the survey of organisations representing low-skilled adults and in the survey of adult learners themselves, of which over three quarters noted that their expectations before participation had been met to a great or large extent. Concerning benefits for employers, respondents to the survey of coordinating and implementing organisations reported that implementation measures contributed to fewer skills shortages (76%), more motivated staff (73%), and higher levels of productivity (66%). At the level of society, most EU and national authority level stakeholders indicated that the Recommendation had increased the political salience of action to improve adult basic skills, and provided different stakeholders with a common reference framework on which discussions could be based. The various benefits of upskilling measures for adults are also backed by a significant body of research, including a 2017 European Centre for the Development of Vocational Training (Cedefop) study that explored benefits of reducing the share of low-skilled adults in the EU: the results of this study point to large fiscal benefits in terms of lower out-of-work social benefits expenditure and positive effects on tax revenues, increases in GDP and other benefits stemming from indirect impact channels via improved health and reduced crime. 97

Costs and benefits of the Recommendation varied across Member States depending on their starting point, but higher costs were generally also associated with higher benefits. In Member States that had in place measures largely in line with the Recommendation so that no major changes in policy or legislation were required, stakeholders confirmed that they incurred no or low additional cost as a result of the Recommendation. In Member States where adult learning systems were overall less in line with the type of provision put forward in the Recommendation initially and which took implementation steps (cf. Section 3.1), costs were higher. However, stakeholders from these Member States also felt that benefits were likely to be relatively large, so that implementation was similarly efficient. Across Member States, the COVID pandemic was perceived as having a negative effect on efficiency, by slowing down delivery without proportionate cost reductions. 98  

Key factors improving the efficiency of implementation measures identified by the stakeholders include clear responsibilities for and transparency about implementation measures, competitive and performance-based selection of training providers, availability of specifically trained staff and cooperation with employers. 99  Respondents in some Member States (including IE, MT and NL) highlighted that divided responsibilities for adult education and skills across government departments can reduce efficiency by leading to a co-existence of implementation measures with overlapping objectives, causing a duplication of efforts as well as confusion and insufficient take-up among potential beneficiaries. Relatedly, the Austrian case study found that uncertainties around the continuation of previously announced adult education priorities limit efficiency. 100  Other Member States (including FI, MT, SE) highlighted the importance of reducing the costs of publicly procured services to implement the Recommendation through an open application process in which potential adult education providers are required to set out their project costs, and of improving the quality of delivery by linking payments with outcomes. Several Member States (including LV and PL) highlighted the importance of specifically trained staff in the adult learning sector for an efficient implementation of the Recommendation. Stakeholders also mentioned cooperation with employers as a factor that can increase the efficiency of support delivery, by tailoring an individual’s upskilling pathway to job requirements.

Stakeholders identify the limited availability and use of rigorous monitoring & evaluation of implementation measures as an over-arching factor limiting efficiency of implementation. 101 Even where monitoring or evaluation measures are in place, there is often a lack of clear “feedback loops” that would allow such information to influence the design of the implementation measures. Monitoring and evaluation are key to improving the efficiency of implementation over time on the basis of “lessons learned” about the design of the three steps of the Recommendation in the specific Member State contexts. However, adequate arrangements can be resource intensive (especially the collection of data on participant outcomes such as employment rates) and complex (as they will often involve several stakeholders such as public authorities, education and training providers and employers). Hence, they require robust cooperation arrangements, a sufficient scale and sufficiently long planning horizon to make their set-up worthwhile- conditions that are often not met by the implementation measures taken in the Member States so far. Some stakeholders also called for an EU level monitoring framework for the Recommendation.

Coherence

Stakeholders consulted at both EU and national level consider the Recommendation to be internally coherent 102 , with consistent objectives, recommendations and expected impacts. The measures proposed in the Recommendation (three-step approach, coordination and partnership, outreach, guidance and support measures, follow-up and evaluation) are perceived to provide a comprehensive policy framework to guide implementation actions at national level. At the level of implementation, some stakeholders highlighted that it is important for step one (skills assessment) to emphasize existing skills to build upon as opposed to skills deficits, in order to avoid stigmatization and ensure coherence with the objective of motivating adults to continue their upskilling pathway. 103

Stakeholders also report a high level of coherence between the Recommendation and relevant policies and strategies at national and regional level. In the survey of coordinating and implementing organisations, 88% of respondents considered that the objectives, target groups and measures to implement adult learning measures have been coherent with education and training, employment and social policies at national/regional level to a fairly large (44%) or great (44%) extent. In the public consultation, more than half of the respondents considered that the measures set out in the Recommendation were to a fairly or very large extent coherent with their national policies (Figure 5). In several Member States (e.g., AT, DK, EE, FR, SE), existing national and regional policies for upskilling adults already aligned well with the Recommendation in 2016 and in other Member States, steps have been taken to increase alignment (cf. Section 3.1), though these steps have not always been very large (cf. Section on “Effectiveness).

Figure 5: Coherence of implementation measures with other education and training, employment and social policies

Source: Public consultation, cf. Annex V and supporting study Section 4.3 Q3.2.

Challenges to coherent implementation at national level are an insufficient integration of the three steps from the Recommendation into a single pathway and a lack of strategic coordination: while a Member State may have provisions relevant to all three steps of the Recommendation, these may follow separate logics and not be integrated in a user-friendly way. 104 Similarly, insufficient strategic coordination between implementation measures and other parts of the adult learning system with absence of a national adult learning strategy (e.g. in CZ, LV) can result in a situation where the available steps do not add up to an upskilling pathway as envisaged by the Recommendation. Conversely, stakeholders perceive better coherence in Member States where national skills strategies exist (as e.g. recently introduced in SK). Further factors perceived to increase coherence are clear responsibilities at ministerial level, and involving all key national stakeholders e.g. via the board of a national training agency (as in BG).

The Recommendation is also perceived to be coherent with other relevant EU policies. 105  The 2018 Council Recommendation on key competences for lifelong learning provides orientation on the basic skills that should be conveyed during the upskilling pathways. 106  A number of Council Recommendations focus on how to support distinct groups of adults (i.e. the 2013 and 2020 Council Recommendations on establishing and reinforcing the Youth Guarantee, and the 2016 Council Recommendation on the integration of the long-term unemployed into the labour market 107 ). Others address elements of the three step-approach recommended in this Recommendation  (i.e., the 2012 Council Recommendation on the validation of non-formal and informal learning, addressing steps one and three, and the 2022 Council Recommendations on micro-credentials, addressing step three, and individual learning accounts, addressing in particular step two 108 ). The present Recommendation outlines how Member States can build on these Recommendations and provide low-skilled adults with an integrated upskilling pathway. A point of attention mentioned by stakeholders is the need to pay attention to coherence among EU policies in the realms of labour market and education, mirroring the need for strategic coordination at national level, and to provide clear guidance to national stakeholders on the links between relevant EU policy and strategic documents created e.g. in different Council formations. 109

More generally, the Recommendation reflects an increasing recognition of continuing skills development throughout life as an individual right. The first principle of the 2017 European Pillar of Social Rights 110 recognizes the right to education, training and life-long learning as its first principle. On 25 June 2021, the European Council welcomed the EU headline targets of the European Pillar of Social Rights Action Plan and the Porto Declaration. 111 Leaders thereby supported the ambition that at least 60% of all adults should participate in training every year by 2030 and at least 80% should have at least basic digital skills. This is seen as necessary for employability and social cohesion in view of the accelerating labour market transformations. 112  While these commitments reflect that the current up- and reskilling challenge goes beyond basic skills and low-skilled adults, improving support for low-skilled adults remains essential for reaching these goals: low-skilled adults are consistently under-represented in adult learning participation within Member States, but show large participation differences across Member States (cf. Section 3.2)- pointing to the potential for progress through policy support that is tailored to their needs.

Finally, almost all stakeholders consulted at both EU and national level considered that there was good coherence between the Recommendation and relevant EU funding mechanisms. 113 They emphasised in particular the high degree of coherence with the European Social Fund (ESF, now: ESF+ 114 ), which allows funding for national and regional measures corresponding to all fields of the Recommendation, but also mentioned Erasmus+, the European Regional Development Fund (ERDF), the Structural Reform Support Programme (SRSP) (and its successor, the Technical Support Instrument (TSI)) and the EU Programme for Employment and Social Innovation (EaSI), as well as the more recent Just Transition Fund (JTF) and the Recovery and Resilience Facility (RRF).

4.2 How did the EU intervention make a difference? (EU added value)

The EU added value of the Recommendation is that in several Member States, it has increased the policy attention on low basic skills among adults, provided a common framework for national policy action, fostered stakeholder collaboration within and across Member States, and/or provided funding and technical support for policy experimentation. 115  Many Member States have taken action to implement one or several of the three steps (cf. Section 4.1, “Effectiveness”). In some Member States (including HR, EL and SK), the Recommendation served as an inspiration for reforms of the adult learning system. In others, it served as a benchmark against which to compare the existing support systems, and pilot or introduce targeted provisions concerning one of the three steps, most often skills validation (cf. Section 3.1). Stakeholders welcomed this flexibility of the Recommendation, though the frequent implementation of one step of the Recommendation in isolation from the other two may have contributed to the perception of insufficient efforts to ensure a smooth integration and improved permeability of the three steps. 116  Despite the good practices and progress identified, implementation measures often lacked scale and coordination (cf. Section 4.1, “Effectiveness”).

Stakeholders appreciated the role of the Recommendation in fostering knowledge sharing across Member States (informing reforms e.g. in EL, BG and CY). 117  Three mutual learning events organised by the European Commission brought together stakeholders from Member States who wanted to incorporate experiences from other Member States in their own national reforms. Mutual learning also took place through the national coordinators for adult learning, the adult learning working group and –in a more informal and “bottom up” manner- via the European Platform for Adult Learning and Education (EPALE). Technical assistance was provided via the Structural Reform Support Programme (SRSP) or its successor, the Technical Support Instrument (TSI), to BE, HR, CY, HU, IT, NL, PT and ES 118 , as well as indirectly by supporting the OECD in developing national skills strategies and diagnostic reports in several Member States (including BE, IT, LV, LT, NL, PL, PT, SK, SI, ES). 119  Cedefop supported Member State reforms by producing comparative research on their upskilling systems 120 , and by organising policy learning fora on the Recommendation. Competence frameworks such as DigComp 121 were promoted and available to support the implementation. The Recommendation fed into further initiatives to support upskilling also at EU level, as reflected for instance in a substantial number of country-specific recommendations 122 on this topic in the European Semester process over the evaluation period, the 2020 Council Conclusions on Reskilling and Upskilling 123 and the 2020 European Skills Agenda. 124  Finally, stakeholders highlighted that the Recommendation helped to direct EU funds from the various funding streams towards basic skills development projects for diverse target groups. As reflected in the 2019 stocktaking report 125 , stakeholders emphasise the key importance of EU funding for the implementation measures (cf. Section 4.1). This included three specific calls for actions on the Recommendation to support policy experimentation through the piloting of implementation measures under the EaSI programme. The implementation plans submitted by Member States in 2018 suggest that initially, this most often took the form of temporary or pilot projects, with unclear long-term impacts. 126 However, it is still unclear whether this will also be true for the skills investment and reform measures funded under the national Recovery and Resilience Facility as part of the Next Generation EU budget and implemented until the end of 2026. 127

Stakeholder views on the intervention’s influence on national policies are mixed, but point to a significant influence in some Member States. In the survey of coordinating and implementing organisations, 41% of respondents state that adult learning policy measures in their country/region during the past 5 years would have been implemented in the same way without the Recommendation to a fairly large or great extent, whereas 34% of respondents state this would have been the case not at all or to a small extent and 24% do not know. However, 46% of respondents state that the overall objectives of the Recommendation could have not at all or only to a small extent been achieved without EU intervention, as opposed to 32% who believe this to be the case to a fairly large or great extent, and 22% who do not know. 128  A substantial minority of 37% of respondents also indicated that the Recommendation helped adult learning measures to converge with other Member States to a fairly large or a great extent.  129  This picture is consistent with the mixed implementation progress outlined in Section 3.1, and the limited competences of the EU in the skill policy domain, where EU Recommendations are one among many factors influencing national policies.

4.3 Is the intervention still relevant?

Basic skills – literacy, numeracy and digital – are the foundation for any further learning and for full participation in the labour market and in society, and the review in Section 3.2 highlights that still far too many adults in the EU do not possess a functional level of these skills.

Since the adoption of the Recommendation in 2016, the digital transition has accelerated, partly due to the COVID-19 adaptation measures, and the EU has embarked on an ambitious “European Green Deal”, striving to become climate-neutral by 2050. Both trends increase the importance of basic and other transversal skills as a foundation to manage change at a personal level, and thrive on rapidly changing labour markets. 130  Demographic ageing is expected to accelerate in the coming decade as the “baby boomer” generation retires, increasing the pressure to use the full potential of everyone on the labour market today. Finally, COVID-19 and recent geo-political events have highlighted the importance of societal resilience for dealing with adverse external shocks and the resulting uncertainty- again with a crucial role for basic skills for managing change. 131  

Stakeholders are hence near unanimous in their view that the Recommendation’s objectives are still relevant 132 : in the public consultation, 87% of respondents agreed to this statement. Respondents to the survey of coordinating and implementing organisations also found that the objectives of the Recommendation were still relevant to the current socio-economic and policy context in their country/region: 95% of respondents said that this was true to a great or fairly large extent. The continued importance of the objectives is reflected in high-level EU targets on adult skills development for 2030 (cf. Section 4.1 on “Coherence”). Some stakeholders expressed appreciation that the Recommendation makes reference to a wider skills set next to numeracy, literacy and digital skills 133 , and highlighted that recent socio-economic trends highlight the need to include life skills such as digital safety awareness and environmental awareness in adults basic skills set. 134  

Stakeholders also consider the measures defined in the Recommendation to be still relevant for the objectives of the Recommendation 135 : 93% of the respondents to the survey of coordinating and implementing organisations agreed to this statement to a great of fairly large extent, and this was the case for 69% of respondents to the survey of organisations representing low-skilled adults. Moreover, 97% of respondents to the survey of coordinating and implementing organisations and 98% of respondents to the survey of organisations representing low-skilled adults agreed to a great or fairly large extent that the low-skilled are still in need of specifically targeted and tailored measures to help them gain skills. 

Stakeholders highlighted the continued relevance of the Recommendation’s “system building” approach, and the need to adapt education and training provision in view of the changing socio-economic context. While each of the three steps of the Recommendation is important, the steps need to be part of an integrated system 136 to meet the diverse needs of low-skilled adults. The Recommendation’s accompanying measures - including outreach, guidance, support measures and effective coordination –remain essential components of such an integrated system. They should involve providers of social services and active labour market programmes, in order to lower access thresholds for the “hardest to reach” adults in the target group. Moreover, stakeholders pointed to the need to rethink adult learning systems 137 in light of the digital and green transition’s challenges (e.g. a more rapid and wide-spread transformation of jobs) and opportunities (e.g. possibilities for digital integration of services, tailored online and blended learning), and the inadequacies of existing education infrastructure that have been exposed by the COVID-19 pandemic.

Stakeholders report that the various priority target groups of the Recommendation remain relevant (most frequently mentioning the long-term unemployed and inactive, migrants and older workers- cf. Section 4.1, “Effectiveness”), but highlight that implementation measures need to better reflect the individual nature of upskilling needs. Stakeholders also highlight that an exclusive focus on traditional priority target groups risks missing adults who would benefit from implementation measures, for instance adults with low basic digital skills. 138 Conversely, implementation measures need to recognize that traditional priority target groups are no monolithic blocks but heterogeneous, as e.g. some migrants are highly qualified while for other adults, multiple disadvantages intersect. 139 This is in line with the findings from a recent Cedefop opinion survey on the perceptions and circumstances of adults in Europe, which concludes that belonging to a specific group has surprisingly little predictive power at the level of individuals. 140  The Recommendation’s approach of responding to this challenge remains relevant: it emphasises the importance of an integrated and individualized delivery of the parts of the Recommendation (including individual skills assessment & tailored learning offer), as this approach to implementation allows to respond to heterogeneous and specific individual needs while avoiding access barriers based on ex ante-criteria defined at a group as opposed to an individual level.

EU level attention to implementation remains relevant: it has made a difference over the evaluation period by increasing policy attention for adult basic skills development, and fostering national reform and policy experimentation via funding, mutual learning and technical assistance (cf. Section 4.2). However, many Member States are only at the start of the process of building an integrated upskilling pathway as outlined by the Recommendation (cf. Section 4.1). Correspondingly, 87% of respondents to the public consultation agreed that EU policy guidance is still required to a fairly large or very large extent (Figure 6). Similarly, more than 80% of respondents to the survey of coordinating and implementing organisations indicated that the objectives of the Recommendation still require action and support at EU level to a fairly large or great extent. 141  

The continued relevance of the intervention is reflected also in the Commission decision on the European Year of Skills. 142 In her State of the European Union address announcing the proposal, President von der Leyen highlighted that “we need everyone on board” as Europe lacks not only high-end professionals but also “truck drivers, waiters and airport workers”. 143 The intervention’s objective of enhancing the employability of low-skilled adults through basic and further skills acquisition remains highly relevant in today’s context of wide-spread skills shortages that hold back Europe’s investment and growth, as clearly recognised by the newly launched Green Deal Industrial Plan 144  .

Figure 6: To what extent is EU policy guidance still required to provide opportunities for low-skilled adults to develop their skills?

Source: Public consultation, cf. Annex V and supporting study Section 4.5 Q5.3.

5.What are the conclusions and lessons learned?

5.1 Conclusions

The analysis in Sections 3 and 4 suggests that developments over the evaluation period are closest to the “moderate impact” scenario outlined at the end of Section 2: On the one hand, stakeholders highlight the continued relevance of the Recommendation, consider it to be coherent with other national and EU level policies, and deem implementation efforts taken so far to have been efficient. Moreover, EU action is perceived to create added value beyond what Member States could have achieved on their own. On the other hand, to date, the implementation of the Recommendation has not yet been fully effective, as steps have not yet been taken in all Member States even with a medium or high need for change and, where steps were taken, they are not yet at a sufficient scale that would enable reaching the objectives.

Some steps in the right direction have been taken. A first achievement is the stakeholder perception that the Recommendation has in some Member States reinvigorated national discussions on basic skills and on support for low-skilled adults in general. It has fostered cooperation among different stakeholders and acted as a useful benchmark for national policy discussions (Section 4.2). However, the full impact of this “agenda setting” role of the Recommendation cannot yet be ascertained, as national reform processes take time. For instance, the national “Recovery and Resilience Plans” aim to combine significant EU funding support from the Next Generation EU budget with durable national reforms, including of adult skills development measures, and will be implemented by Member States until 2026.

A second achievement is that a majority of Member States has expanded its knowledge basis on implementation measures over the evaluation period (Section 3.1), often in the form of EU-funded projects. In terms of the implementation of the three step approach, skills assessment offers of some sort are now wide-spread across Member States, and a number of Member States have taken steps to improve the provision of tailored and flexible learning and opportunities for skills validation and recognition since 2016. Future implementation efforts can draw on the “lessons learned” from these measures.

A third achievement is that the Recommendation has fostered mutual learning across EU Member States. Stakeholders appreciated the mutual learning and technical assistance provided over the implementation period, and “policy experimentation” was sometimes explicitly based on examples from other Member States (Section 4.2). 

However, significant challenges still remain. A first challenge is the uneven implementation progress among Member States, with large remaining discrepancies in upskilling opportunities available for low-skilled adults. The supporting study does not find substantial improvements concerning the implementation of the three steps of the Recommendation over the evaluation period in 13 Member States 145 , even including in 8 Member States with medium or high need for change in 2016 (Section 3.1). Relevant indicators point to the persistence of large differences in opportunities across Member States. For instance, the likelihood of still receiving upper secondary schooling through a “second chance” pathway after initial education age, or of participating in adult learning while being unemployed is uneven across Member States (Section 3.2).

A second challenge is the often still small scale and insufficient effective outreach 146  of implementation measures taken by the Member States which was already reflected in Member State’s planned implementation measures reported in 2018. 147  Also project-based EU funding as opposed to structural domestic funding, and no clear evidence in terms of impacts on relevant available indicators are evidence of significant implementation challenges (Section 4.1). Stakeholders highlight a resulting insufficient outreach to key target groups in view of the accelerating labour market transitions, in particular for adults with multiple vulnerabilities and those who are not part of traditional target groups (Section 4.3).

A third challenge is the often still insufficient integration of the three steps of the Recommendation into a comprehensive pathway, reflecting an insufficient degree of strategic coordination. Stakeholders highlighted the importance of promoting smooth transitions between the three steps to promote each individual step’s uptake and effectiveness and adult learners’ commitment to their upskilling pathway. However, the joint-up approach of the Recommendation is often not yet reflected in Member States’ implementation measures, which also results in limited “systemic learning” on the basis of monitoring and evaluation (Section 4.1).

5.2 Lessons learned

Recent socio-economic trends are increasing the importance of basic skills for active participation in the labour market and in society. However, support is not always sufficiently tailored to meet the needs of the individual. At the same time, participation in measures to support basic skills development runs a risk of being perceived by potential beneficiaries as stigmatising or not relevant. Moreover, recent labour market trends are broadening the priority target groups which could benefit from further strengthening their basic and transversal skills (Sections 4.1 and 4.3).

ØSuccess factors include opening up access to support to a broader range of adults with tailoring to specific needs on an individual basis, mainstreaming basic skills support across overall adult skills development strategies, and a communicative emphasis on future upskilling benefits as opposed to current skills deficits.

There is a need to devote sufficient policy attention and resources to effective outreach, including through structural funding of adequate infrastructure. Progress on adults’ basic skills includes successfully reaching the “hardest to reach”, who have often had negative experiences with school-based learning or face multiple and intersecting disadvantages, including a low level of digital skills.

ØSuccess factors include a broad partnership approach which combines the forces of different types of organisations, including public employment services and social partners but also those not primarily involved in education and training such as libraries and community or health or social inclusion centres, stimulating adults to engage in learning activities in their usual and trusted environments. They also include a public provision of adequate infrastructure to activate local initiatives that can start more learning pathways among low-skilled adults.

There is a need to move from good practice on a specific step of the Recommendation at small scale towards system-level improvements backed by increases in domestic funding. Over the evaluation period, the dominant approach to implementation has been based on pilot projects and EU funding- which has generated relevant evidence, but fallen short of the system-level improvements envisaged by the Recommendation.

ØSuccess factors include strategic cooperation at national level, characterized by clear responsibilities that allow for a smooth integration of the three steps into a transparent and user-friendly upskilling pathway with interlinked steps, follows a systematic approach to staff professionalisation and is responsive to results from monitoring and evaluation. This does not mean reducing regional or local initiatives, including by employers, but underpinning them with a clear and stable framework for cooperation.

Annex I. Procedural Information

Lead DG, Decide Planning/CWP references

The evaluation of the Recommendation was led by the Directorate-General for Employment, Inclusion and Social Affairs, with a Decide planning reference PLAN/2020/9155. This Staff Working Document is accompanying the Report to the Council 148 which has a Decide planning reference PLAN/2022/2233.

Organisation and timing

An evaluation roadmap published on 12.1.2021 explained the context, purpose and scope of the evaluation and informed stakeholders that an external evaluation study would be carried out, supported by a public consultation. The feedback to the roadmap was open from 12.11.2021 to 9.2.2021. The public consultation was open for 13 weeks from 16.12.2021 to 17.3 2022. Targeted consultations were carried out mainly within Q1 and Q2 2022. See Annex V for further details on the stakeholder consultations carried out.

An Inter-service Steering Group (ISG) was set up, coordinated by the Directorate-General for Employment, Inclusion and Social Affairs (represented by staff from the lead policy unit, other relevant policy units and the Better Regulation unit) and including members from Eurostat, the Secretariat-General, and the Directorate-Generals for Internal Market, Industry, Entrepreneurship a SMEs (GROW); Education, Youth, Sport and Culture (EAC); Economic and Financial Affairs (ECFIN); Migration and Home Affairs (HOME); Justice and Consumers (JUST); Regional and Urban Policy (REGIO); Research and Innovation (RTD); Health and Food Safety (SANTE); Structural Reforms Support (REFORM) and the Joint Research Centre (JRC). The ISG could take advantage of the technical assistance of an expert from Cedefop.

The ISG also acted as steering group for the external study supporting the evaluation, which was carried out by Ecorys. Contract VC/2021/0381 was awarded after reopening of competition (tender VT/2020/066) within the multiple framework contract VC/2017/0372). The contract VC/2021/0381 started on 8.7.2021. The draft final report of the external study was received on 29.4.2022, the inter-service steering group provided comments by 11.5.2022 and the contractor delivered the accepted revised final report on 21.9.2022, along with the agreed annexes (including an annex on the stakeholders’ consultation).

Exceptions to the better regulation guidelines

All Better Regulation requirements were fulfilled.

Consultation of the RSB (if applicable)

Not applicable.

Evidence, sources and quality

The main sources of information used in this evaluation are:

·Commission Staff Working Document supporting the Commission proposal underpinning the Recommendation in 2016 149 and the explanatory memorandum 150 of the Commission proposal

·Commission Staff Working Document taking stock of the implementation measures reported by the Member States in 2018 151

·Study supporting the evaluation 152 of the Council Recommendation of 19 December 2016 on Upskilling Pathways: New Opportunities for Adults, carried out by Ecorys

·Public consultation- with 60 responses, cf. Annex V

·Targeted consultations- an exchange of views took place in dedicated meetings with the Directors General for Vocational Training (DGVT), the Advisory Committee on Vocational Training (ACVT), the European Network of Public Employment Services (PES Network), the National coordinators for adult learning and the European Qualifications Framework (EQF) advisory group, cf. Annex V

·Eurostat data

See Annex II for additional information on the methodological approach taken. To ensure quality, the ISG provided feedback throughout all steps of the evaluation process. Chronological overview of the evidence collection process:

Date	Type of activity
17.12.2020	ISG meeting to discuss the Tender specifications
12.1.2021 – 9.2.2021	Publication of the roadmap and feedback
18.3.2021	Launch Request for services
16.4.2021	Offers received
8.7.2021	Signature of Contract
23.7.2021	ISG meeting: kick-off meeting with external contractor
10.9.2021	ISG Meeting: Inception external evaluation report and discussion about the consultation strategy and public consultation
16.12.2021 17.3.2022	Public consultation
14.12.2021	ISG Meeting: Interim external evaluation report
Q1-Q2 2022	Targeted consultations
29.4.2022	Draft final report
10.5.2022	ISG meeting: draft Final external evaluation report
20.6.2022	Revised draft Final external report
14.9.2022	ISG meeting on the draft Staff Working Document
31.10.2022	Receipt of final version of external report supporting the evaluation

Annex II. Methodology and Analytical models used

Overall approach to the evaluation

The evaluation follows the requirements set out in the Better Regulation Guidelines and Toolbox. 153 This includes looking into how EU Member States have responded to the Recommendation and assessing its implementation against the five key evaluation criteria of relevance, effectiveness, efficiency, coherence and EU added value presented in the below Table. The evaluation work including the external study and the evaluation report are structured around describing the baseline and the evolution of relevant policy measures and indicators over the evaluation period until November 2021. 

Overview of evaluation criteria

Evaluation criteria	Overview of key points
Effectiveness	The assessment of effectiveness has been multi-faceted, covering the extent to which the Recommendation was effectively implemented (i.e. to meet objectives and expected results), but also how it was implemented in terms of the extent to which the implementation of the Recommendation facilitated access for low-skilled adults to different upskilling opportunities from the baseline situation (in 2016), and the degree to which relevant priority targets were effectively targeted and reached was examined. Focus was given to exploring the extent to which the recommended three-step approach (skills assessment, tailored and flexible learning offer, validation and recognition) together with support via outreach and guidance services, have been implemented in Member States, and their effectiveness. In addition, it has been explored whether and to what degree relevant national, regional and local stakeholders were involved in the coordination and delivery of the implementation of the Recommendation in different Member States. At Member State level, the extent to which measures implemented in relation to Upskilling Pathways have been monitored and evaluated, and whether the results have contributed to ongoing improvements to their design and delivery was assessed. At EU level, it was examined whether mutual learning has been facilitated, and any relevant outcomes. Finally, contribution analysis was used to assess the degree to which the Recommendation contributed to observable changes in Member States in relation to the skills levels of low-skilled adults and developments in lifelong learning opportunities. The contextual dimension has been key in exploring the specific changes in their particular delivery contexts.
Efficiency	The examination of efficiency has explored the degree to which the objectives of the Recommendation have been achieved at optimal cost for different relevant stakeholders, and the factors which have contributed to this, including a consideration of the use of different sources of EU funding to fund actions. Addressing the efficiency criterion has involved identifying and using appropriate benchmarks to respond to the different evaluation questions. Quantitative data has been used as far as possible, but supported by qualitative data from the case studies, existing (national/regional) evaluation reports and data generated from the targeted and public consultations. The analysis has comprised an assessment of the costs and benefits associated with the implementation of measures linked to the Recommendation for different stakeholders (including adult learners themselves) and has aimed to explore the factors which influenced efficiency of implementation, ascertaining whether observed benefits could have been achieved at a lower cost. It has also explored the use of financial incentives to encourage participation in upskilling pathways. In addition, it has assessed which EU funding programmes (e.g. ESF, Erasmus+) contributed to achieving the objectives of the Recommendation, and the degree of cost-effectiveness of this use of funding sources. Bringing together the results of the different analyses, it has provided an assessment of the overall degree to which the costs of the implementation of actions related to the Recommendation have been proportionate to the benefits to individuals (adult learners), the economy and wider society.
Relevance	Assessment under this criterion has focused on examining relevance at several levels and from several perspectives. Firstly, the degree to which the objectives set out in the Recommendation are still relevant both to the current socio-economic context and to the current EU policy context was assessed. Secondly, and in order to achieve these objectives, the extent to which the measures defined in the Recommendation are also still relevant was assessed. Thirdly, the assessment has explored the extent to which the measures defined in the Recommendation are still relevant to the needs of both the target groups and economic sectors. Finally, it was examined whether the target groups specified in the Recommendation and those defined by different Member States remain relevant.
Coherence	Examining coherence implies the need to assess linkages, synergies and potential duplication with related EU, national and regional policies, instruments, initiatives and recommendations (policy and programme coherence). The assessment of coherence has focused on complementarity and coherence from two principal dimensions – ‘internal’ and ‘external’ coherence. In terms of ‘internal’ coherence, the coherence between objectives, target groups and measures of the Recommendation was examined. The assessment of ‘external’ coherence has explored the complementarity and coherence of the implementation of the Recommendation with: (a) relevant (education, training, employment and social) policies at national and regional levels; (b) other related EU level policies (e.g. the Youth Guarantee, the Long-Term Unemployment Recommendation, etc.); and (c) relevant EU funding mechanisms (e.g. ESF, EaSI, Erasmus+, etc.). Evidence to address the coherence criterion has been drawn principally from the mapping exercise and desk-based research, but also on other sources, notably the stakeholder insights gained from consultation and detailed Member State case studies.
EU added value	As specified in the Better Regulations Toolbox (tool #47), assessing EU added value involves looking for changes which can reasonably be argued are due to the EU intervention, over and above what could have been expected from national actions by the Member States. Building on the evidence gathered and analysis carried out to assess the other criteria, the added value of the Upskilling Pathways with a particular focus on the degree to which it supported a common approach (in terms of objectives, design, target group, coordination and support measures) to providing upskilling pathways for low-skilled adults was assessed. It was also assessed whether the objectives of the Recommendation could have been achieved solely by the action of the Member States, as well as the degree to which they still require EU level support and action. Detailed evidence from the case studies has been a key way of assessing the relative importance in particular contexts, alongside evidence available from results of the public and targeted consultations, as well as the mapping exercise.

Source: Supporting study.

Methodology and data sources

The evaluation is based on a mixed-method data collection approach, combining qualitative and quantitative research methods.

This included:

·Literature review and scoping interviews

oDuring the initial phase, the contractor undertook a preliminary literature review on upskilling pathways. This laid the foundations for the subsequent tasks of the study.

oScoping interviews were also conducted to gain further insights into the Recommendation.

·Quantitative data mapping

oThe contractor mapped the quantitative datasets relevant to the study. Specific focus was given to datasets available at least from the year 2015, to establish the baseline for the evaluation, and until the year 2021, to also allow for a comparison over time. The contractor gathered data on indicators linked to adult skills and education level, the participation of adults in adult learning, employment and unemployment rates, as well as further indicators related to economic structure and economic growth.

·Mapping of key policy measures

oTo better understand the relevant initiatives in place both at the baseline and since the adoption of the Recommendation, a mapping was conducted starting with a systematic review of EU and national level sources. On this basis, the research team developed a longlist of relevant upskilling pathways measures in each Member State, over the period considered for the evaluation.

·Targeted consultations

oInterviews with key stakeholders at national and EU level. 

oTargeted survey with organisations involved in the coordination and implementation of the Recommendation, organisations representing low-skilled adults and adults participating in initiatives implementing the Recommendation.

oExchanges of views in dedicated meetings organised by the Commission. Meetings with the Directors General for Vocational Training (DGVT), the Advisory Committee on Vocational Training (ACVT), the European Network of Public Employment Services (PES Network), the National coordinators for adult learning and the European Qualifications Framework (EQF) advisory group.

oOnline expert meeting to discuss initial study findings and exchange on topics related to the Upskilling Pathways implementation.

oValidation workshop to discuss preliminary study fundings, draft conclusions and recommendations.

·Public consultation

oThe online public consultation was launched by the European Commission on 16 December 2021 and ran until 17 March 2022. The questionnaire used to implement the survey was distributed using the online EU Survey tool.

·Case studies

oTo gain deeper insight into the actions taken in response to the Recommendation on the national/regional/local level(s) the contractor carried out ten case studies across ten Member States.

See Annex V for more information on the consultations, and Annex 1 of the supporting study for further information on the methodology and data sources.

Research limitations

It should be noted that a robust counterfactual impact evaluation of the Recommendation is not possible due to data limitations and the multitude of factors other than the Recommendation which may have influenced trends in relevant indicators. Moreover, impacts of any structural changes in national skills policies influenced by the Recommendation may take longer to unfold.

Where relevant, this evaluation discusses alternative explanations for the observed trends, and presents stakeholders views to support their interpretation. Finally, responses to the public consultation or targeted consultations need not be representative for the relevant target populations or all EU Member States. Where possible, the evaluation triangulates results from several information sources to increase the robustness of its conclusions. The below Table provides a more extensive overview over the key limitations of the study supporting the evaluation, and outlines the mitigation measures taken to increase the robustness of the conclusions.

Key limitations of the supporting study

Limitation	Explanation	Mitigation measures
Lack of clearly identifiable policies that were implemented in response to the Recommendation	Member States’ adult learning systems differ greatly in their organisation, scope and infrastructure and the same is true for the overall support provided for upskilling. In many Member States, actions which correspond to the Recommendation existed prior to its adoption in 2016. There is therefore no ‘easy access’ single repository (at EU or Member State level) of actions implemented by Member States in response to the Recommendation after the 2018 reporting of implementation measures by Member States, analysed in the 2019 stocktaking report. 154	To establish a list of relevant initiatives to assess and analyse for the evaluation, in particular for the mapping task, the initiatives that collectively are considered by national authorities of each Member State to be the most relevant and representative of their country’s response to the Recommendation based on key EU policy documents on adult learning and the Recommendation were identified. Then the Member State representatives were consulted on the completeness of this sample of initiatives and asked to validate the sample. Representatives were able to remove or add other initiatives in order to come to a validated list of initiatives that best represent a Member States’ response to the Recommendation.
Limitations of the mapping methodology	The analysis of the situation in 2016 and the evolution since then is based on a sample of measures and therefore cannot be seen as a fully comprehensive analysis of all relevant policies in place.	It was ensured that the major initiatives that have been put in place in response to the Recommendation or that are related to the Recommendation are captured in the analysis, this was verified with national authorities to ensure completeness and have further cross-checked with findings from the interviews and views of national experts. Whilst this does not provide the full overview of all measures related to upskilling pathways, it has enabled the assessment of the expected impact of the Recommendation in 2016 and the degree of implementation since then.
Limited data that directly links to the Recommendation and can be used for monitoring its results.	As a result of the lack of clearly identifiable policies implemented in response to the Recommendation (explained above) there is also a lack of data that is directly linked to the Recommendation. The implementation of the Recommendation is not monitored uniformly or systematically in each Member State meaning that a comprehensive monitoring data is not widely available.	The evaluation methodology has allowed for identification of existing studies and evaluations of measures that Member States deem representative of their UP response. Along with the rich data gathered from consultation activities, the surveys and the case studies were used to understand and analyse the results of upskilling initiatives linked to the Recommendation, in order to assess its effectiveness in particular.
Lack of data on specific target groups, sectors and on specific skills	The low-skilled adults targeted by the Recommendation encompass a wide range of different groups, including for example migrants, older adults, low-skilled workers, unemployed adults and adults at risk of or experiencing poverty. However, granular and comparable data on both the size of and the impact of upskilling measures on specific target groups is not readily available. In particular, data on participation in education and training for subgroups such as migrants, those at risk of poverty is not available. Furthermore, data on specific skills’ levels is also limited with the latest comparative data on numeracy and literacy skills being the OECD PIAAC survey from 2017.	Data available for foreign-born populations, the unemployed, long-term unemployed and inactive as well as data disaggregated by gender where relevant were examined. Beneficiaries of upskilling pathways and organisations representing or working with low-skilled individuals were targeted through the surveys conducted which has allowed for understanding how the Recommendation has impacted specific target groups, as far as possible. In addition to this, all quantitative data has been triangulated with primary data collected through the consultation activities.
Lack of causal impact evaluation of policies due to data availability	To perform rigorous causal analysis of Upskilling Pathways initiatives and their effectiveness, it is crucial to have data on the basic units targeted by the policy (i.e., the so-called target group, low-skilled adults) and people belonging to target groups and economic sectors participating in Upskilling Pathways initiatives. Similar data need to be available for the comparison group, meaning access to microdata (i.e., data at the level of individuals, households or businesses or communities). As this has not been available, causal impact evaluation has not been fully undertaken for the evaluation.	Despite this limitation, other ways were explored to try and assess the causal impact of the Recommendation. It was formally tested how key outcomes/indicators (have changed since the Recommendation was published and, drawing on the mapping exercise, the extent to which any changes are associated with different levels of adoption/implementation of the Recommendation, through an interrupted time series analysis, included in Annex 9 of the supporting study. The case studies also include evidence from evaluation reports/findings of specific interventions (aligned to the Recommendation) which were subject to a robust counterfactual impact evaluation (CIE). Whilst these findings must be treated with caution (as they do not account for other factors) they provide a useful insight that has been triangulated with other data sources.
Capturing the long-term impacts of participation in upskilling pathways	The full benefits of the participation of individuals (and organisations) in upskilling activities cannot be captured in the short-term, where the focus is on measuring immediate results. Assessing the full outcomes of participation ideally requires longitudinal research involving follow-up after several years to assess the long-term impact of participation e.g., in terms of labour market outcomes, access to further education and training, health and wellbeing, etc. The full range of benefits for individuals and organisations of the 2016 Recommendation will not yet be apparent in some cases.	Several important impacts from the initial five-year period of implementation, have been captured including through the case studies where existing evaluations of specific measures have been analysed and drawn from to provide sound estimates for the expected long-term effects of completed upskilling pathways in adulthood).
Capturing the views of beneficiaries including adults with low levels of skills, competences and knowledge	There are clear challenges with consulting beneficiaries of upskilling measures, not least in identifying and gaining access due to data privacy (GDPR) and similar issues (national/regional authorities and upskilling providers cannot provide – and sometimes do not hold – contact details for previous participants).	To capture the views of adult learners as far as possible, it was decided to work with representative organisations of relevant target groups such as EU and national level NGOs working closely with the target groups. The surveys targeting adult learners and organisations representing the low-skilled have allowed for gaining insights into the views of beneficiaries.
Lack of representativeness of the public consultation results	The public consultation received 60 responses. The recoding of variables to disaggregate the analysis and allow differentiation between all response options was therefore not possible	The relevant differences between sub-groups based on disaggregation by type of organisation and groups of respondents were analysed where this was possible. However, when results are disaggregated, inferences should be made with caution with regards to the results. Results from the public consultation have been triangulated with all other data sources and have been primarily used to illustrate or confirm findings that have emerged from more robust data collection activities.
Impact of external shocks	Data and evidence on adult learning (as other domains) from 2020 have been heavily impacted by the COVID-19 pandemic.	In terms of measuring progress over time, data for 2020 has been examined through the lens of the pandemic, considering any major changes in outcomes as outliers in response to a major external shock. The impact of the pandemic on the implementation of upskilling pathways was explicitly addressed in the evaluation sub questions and by examining data from 2019 and 2021 to identify any trends in the indicators that may have been halted due to the onset of the pandemic.

Source: Supporting study.

Annex III. Details on answers to the evaluation questions (by criterion)

Answers to the evaluation questions

The tables provide an overview of key findings for each evaluation question under each evaluation criteria (effectiveness, efficiency, relevance, coherence and EU added value). The source of the information is the supporting study (see section 4 of the supporting study for further information).

Evaluation question	Answer
Effectiveness 155
Q1.1 To what extent have the Member States facilitated access to upskilling pathways for adults with low level of skills, knowledge and competences to: a) acquire a minimum level of literacy, numeracy and digital competence; and/or b) acquire a wider set of skills, knowledge and competences, relevant for the labour market and active participation in society by making progress towards a qualification at EQF level 3 or 4?	The evidence shows that Member States have made some efforts to facilitate access to upskilling pathways since the Recommendation was adopted. 10 Member States out of the 14 that had a high need for change in 2016 – based on their existing adult learning provision and the size and needs of the target group – implemented change in response to the Recommendation. However, of the 13 Member States where no change in response to the Recommendation was identified (AT, ES, FI, FR, HU, IE, LT, LU, PT, RO, SI, NL, SE), four Member States (HU, IE, RO, ES) had high need for change and three had medium need for change in 2016 (LT, LU, SI), showing that even in countries where change to upskilling pathways in line with the Recommendation would have been needed, change has not always occurred. The indicators relevant to upskilling pathways also show moderate progress: since 2016, there has been an overall improvement in the educational attainment level of adults across the EU, with the share of low-qualified decreasing in all Member States (except Germany) in that time and the EU average falling from 23.4% to 20.7%. However, there is limited evidence which shows that this improvement is the result of upskilling pathways implemented in response to the Recommendation. Long-term trends show that higher educational attainment levels are due to long-standing improvements of initial education. The direct macro effects of the measures taken in response to the Recommendation will only become fully apparent in the longer term, given that individual upskilling journeys usually take a number of years to complete. Nevertheless, on the micro level, progress on the objectives of the Recommendation is evident: results from evaluations conducted of specific measures/programmes linked to the Recommendation show that the qualification levels of adults that participated in these measures increased, with a positive effect on their employability and employment also recorded. Participation in upskilling measures introduced since the Recommendation is held by stakeholders as having contributed to an improvement in various areas that are linked to social participation and inclusion, including gaining more confidence and improved participation in society overall. Particularly in countries that had high need for change in 2016, stakeholders confirmed that the Recommendation has contributed to highlighting the challenge of the low-skilled, revealing issues with current learning provision and setting out a way forward for improving the upskilling provision for the target group. This discrepancy between progress on the macro-level due to the Recommendation (limited) and progress on the micro-level due to the Recommendation (more evident) cannot be explained fully by the results of the supporting study. However, it may suggest that measures to facilitate access to upskilling pathways are either taking place on too small a scale to be reflected on the macro-level or have not been implemented for a long enough period of time for their results to be reflected in the quantitative data trends.
Q1.2(a) To what extent have Member States identified priority target groups and economic sectors for the delivery of upskilling pathways? Were the most in need and disadvantaged groups properly targeted (e.g., persons with migrant background, Roma, persons with disabilities, persons with a minority racial or ethnic background, other vulnerable groups, etc.)?	All Member States have identified priority target groups for upskilling pathways. However, Member States have remained broad in their definition of the priority target groups of upskilling measures: around 45% of the measures mapped in-depth identify the low-skilled as their target, 40% specify the target group in terms of employment status (with a balanced split between the unemployed and employed), and 25% of measures specifically refer to the inactive as a target group. The majority of mapped measures cover all age groups. Vulnerable groups are targeted by Member States’ initiatives within these broader groups. Long-term unemployed, older workers, people with disabilities, nationals with a migrant background or third country nationals are the groups most often targeted. Evidence shows that Member States have generally targeted those most in need of support, though some groups emerged as requiring further attention, including employed workers with low skills, older workers, women, and refugees or newly arrived migrants. However, there is evidence from the study that even though the right target groups have been identified, challenges remain in effectively reaching them. These include lack of awareness of opportunities amongst the group itself, financial barriers to participation, and stigma/motivation of the target group.
Q1.2(b) Which were the outputs and results for specific groups (in both qualitative and numerical terms, including the profile of the target groups covered as well as the number of participants from each group -at EU and MS level-?)	Evidence that is available on the outputs and results for specific groups shows that participating in upskilling measures has a positive impact on different groups. Evaluations of specific upskilling pathways measures find that participants are more likely to progress into employment or further education after participation, with some evidence showing that this positive effect is higher for those groups that are in more vulnerable situations.
Q1.2(c) Which methodologies/criteria were used by Member States for identifying the different target groups and relevant economic sectors? Which were most/least effective, and why?	Educational attainment level or employment status are the key criteria for identifying target groups. Sectoral needs are addressed in the design of upskilling pathways though this has not been done across all Member States systematically. Where it is done, it is carried out either on a yearly basis based on skills forecasting/ demands, as demand arises, with specific measures developed to address skills shortages, or through identification of regional needs.
Q1.3. To what extent have Member States applied the three steps approach in the design of upskilling pathways? How effective was this approach?	According to the study, the majority of Member States (15) have made changes in implementation of the three steps either by adopting new measures or reforming existing ones, though the degree of changes varies Progress is noticeable in all steps of the Recommendation but measures for validation and recognition of prior learning have seen the biggest increase since 2016, though they remain challenging to implement. However, there is less evidence of clear efforts to ensure a smooth integration and improved permeability of the three steps (and accompanying measures) into one comprehensive pathway. Skills assessment: Seven Member States have made changes to skills assessment approaches (BE, BG, CZ, HR, LV, MT, PL) The limited number of Member States that have made changes in this step can be explained by the fact that it was the step that was most implemented before the Recommendation was adopted. However, evidence from the interviews, case studies and the surveys shows that meaningful skills assessments are not always implemented or used even in cases where they are in place. Challenges include lack of specific skills of staff, lack of awareness of the benefits of skills assessments and lack of know-how by the target group itself and lack of a formal process underlining how a skills assessment should be undertaken Tailored learning offers: Six Member States have made changes in their provision of tailored learning offers since the Recommendation was adopted (BE, BG, DK, EE, MT, PL ) Out of these countries, three had high need for change in 2016 in this area of the Recommendation (BE, BG, MT). Nonetheless, it is clear that the provision of a tailored learning offer is relatively widespread now across EU Member States. Barriers remaining include the difficulty in identifying/adapting to diverse training needs of disadvantaged adults, insufficient funding for tailored programmes, lack of information about tailored programmes and insufficient offers in rural areas. Validation: Eight Member States have made changes in their upskilling approaches since the Recommendation to better implement the step of validation (BE, BG, CY, DE, EL, IT, LV, SK). Six of these Member States had a high need for change in this step (BG, DE, IT, LV, IT, SK). Even though there have been efforts to improve the system of validation, establishing fully functioning validation systems remains a challenge across all Member States, linked to key obstacles including the challenge of integrating the validation system into the adult learning system as a whole and the lack of take-up of validation amongst both the low-skilled and employers.
Q1.4(a) To what extent have Member States delivered upskilling pathways in cooperation with relevant stakeholders, in particular social partners and local, regional and national economic actors? Were relevant public, private and third sector actors in education and training, employment and other relevant policy areas engaged?	The evidence gathered in study has shown that Member States have delivered upskilling pathways in cooperation with relevant stakeholders to a high degree although there are differences across Member States and challenges remain in effective cooperation. Stakeholders most commonly involved in upskilling pathways are social partners, public employment services, local actors and NGOs. Social partners are engaged in different aspects of upskilling pathways, with higher levels of involvement in countries that have a strong tradition of social dialogue. There is however room across the EU to involve social partners more in upskilling pathways, in particular in evaluation and monitoring of upskilling measures. Public employment services often collaborate with social partners and other national authorities in the design of upskilling measures. There is also clear involvement of actors at the local and the regional levels in many upskilling pathways measures. NGOs in particular most often play a role in identifying target groups, particularly vulnerable individuals, and in delivering outreach activities. There is room, however, to more systematically involve these local actors – that are closer to the target group – in the design and implementation of upskilling measures.
Q1.4(b) How have Member States ensured effective coordination of stakeholders at national and regional level to implement the Recommendation? Which arrangements have been put in place and which were the most promising in terms of good practice? Were there any bad practices observed?	Member States have ensured coordination of stakeholders at national and regional level to implement the Recommendation through coordinating bodies for adult learning policies and measures. These exist in the vast majority of Member States, with only five Member States indicating that they currently do not have such a coordinating body in place (CZ, DK, SI, SK, SE). However, stakeholders consulted outlined that even if such cooperation processes are established on paper, cooperation is not always fully effective in practice. Challenges that have been identified include cumbersome administrative processes and a lack of clearly defined responsibilities and roles between the partners which is particularly the case in this field of policy given that adult learning often sits between the remit of education and labour market authorities.
Q1.5 To which extent have Member States provided outreach and guidance services to support the implementation of upskilling pathways? Which arrangements have been put in place and which are the most promising in terms of good practice? Were there any bad practices observed? Which stakeholders have been involved? To what extent have Member States provided support to learners or indirect support to employers for upskilling their employees? To what extent has the support to learners and employers addressed obstacles to participation? To what extent have Member States provided initial training and continuous professional development to staff engaged in the delivery of upskilling pathways (in particular to teaching professionals)?	There is widespread evidence of outreach to potential learners in upskilling pathways measures in all 27 Member States, but with varying degrees of effectiveness in implementation. The vast majority of measures mapped that are relevant to upskilling pathways offer some form of outreach to potential learners. Promising examples have several factors in common including local level cooperation, sufficient funding to ensure comprehensive outreach, and outreach that is embedded into the broader system of support to adult learners. However, more holistic types of support including psychological support are less common and more tailored outreach is needed, recognising in particular the heterogeneity of the target group of low-skilled adults, and the intersectional nature of their vulnerabilities. For this to occur, more funding and human resource capacity needs to be dedicated to outreach, whilst local actors that are closer to the target group need to be more systematically involved in outreach efforts. Guidance that is provided in the context of PES and that involves direct contact with a counsellor is available in all EU Member States to unemployed individuals 156 . However, services are not always available to the low-skilled that are in employment, whilst the low level of awareness of the offer of support also poses an obstacle to accessing guidance services. Evidence shows that the provision of outreach and guidance services to employers has been somewhat inconsistent and piecemeal. Whilst many Member States engage with employers in some way in the design of upskilling pathways, systematic provision of guidance to encourage employers to support employees is limited. This lack of guidance has a more detrimental effect on SMEs than larger companies. Evidence of providing training and continuous professional development to staff involved in delivering adult learning measures was identified in the majority of Member States, though the degree of effectiveness of such support and the level of tailoring to the needs of the target group vary substantially. In some Member States, training is provided but it is not tailored to working with adult learners; in others Member States, a great focus is given to ensuring continuous professional development of staff involved in adult learning provision on a strategic level: in some Member States, there is a strong focus on CPD of staff, but only on a project-basis, with limited evidence of systematic provision.
Q1.6(a) To what extent has the Recommendation contributed to the observed changes in Member States, including changes with regards to levels of literacy, numeracy and digital skills among the target population?	The degree to which the decrease in the share of low-qualified across the EU can be attributed to the Recommendation is very limited. The overall improvement in the educational attainment level of adults across the EU can only to a small extent be considered the outcome of upskilling pathways activities, given that it is likely to be due to long-standing improvements of initial education. This is reflected also in the fact that the downward trend in the share of low-qualified across the EU has been visible since 2011 (see section 3.2.1 of the supporting study). The interrupted time series analysis (annex 9 of the supporting study) has further confirmed this, only revealing one country (Belgium) where such a link between implementation of the Recommendation and change in the share of the low-qualified could be determined. However, it is clear from the consultations with stakeholders undertaken for this study, that the Recommendation has contributed to changes in an indirect way by acting as a catalyst for a renewed policy focus on the target group of low-skilled adults. In Member States that had high need for change in particular, the Recommendation is viewed consistently by stakeholders as having made a contribution to highlighting the challenge, revealing issues with current provision, setting out a way forward for improving the upskilling provision for the target group of low-skilled adults and stimulating cooperation between actors on this topic that did not previously exist. This confirms the finding outlined in Q1.1 of a discrepancy between progress on the macro-level due to the Recommendation (limited) and progress on the micro-level due to the Recommendation (more evident). Measures to facilitate access to upskilling pathways are either taking place on too small a scale to be reflected on the macro-level or have not been implemented for a long enough period of time for their results to be reflected in the quantitative data trends, but some progress on the micro-level in achieving the objectives of the Recommendation is evident.
Q1.6(b)To what extent has the Commission, with the support of the Advisory Committee on Vocational Training, been successful in following up on the implementation of the Recommendation, e.g., through exchanges such as mutual learning?	The three mutual learning events funded by the EaSI programme and organised by the European Commission on upskilling pathways have been beneficial in terms of supporting the sharing of best practices. EU funding has also been identified as one of the principal contributions of the Recommendation with the ESF in particular supporting changes in Member States, particular those with less developed adult learning systems. Projects supported by the Employment and Social Innovation programme (EaSI), funded under the three specific calls for actions have enabled the piloting of approaches to upskilling and encouraged experimentation. The Commission’s publication of a stocktaking report on implementation of the Recommendation in 2019 157 also allowed for an understanding of progress in Member States’ implementation of the Recommendation.
Q1.7 To what extent have Member States evaluated and monitored measures related to upskilling pathways and used the results of their evaluation in informing the design and delivery of upskilling pathways?	Whilst measures related to upskilling pathways are monitored by Member States and some have been the subject of evaluations, the degree to which the results have impacted the design and delivery of the measures is limited. The involvement of a wide range of actors in the upskilling pathways measures (institutions from the labour market and education) means that cooperation needs to be very robust to allow for such data to be tracked, analysed and used to influence programmes. The lack of a systematic data collection process on the implementation of the Recommendation on the EU level was also identified as challenging, making it difficult to assess and compare progress EU-wide.

Evaluation question	Answer
Efficiency 158
Q2.1 What were the costs associated with the implementation of the Recommendation for the different stakeholders (including adult learners) at various levels (national, regional, local)? To what extent have Member States encouraged participation in upskilling pathways through financial incentives?	Costs for the state and municipal administrative bodies include adjustment costs from designing, delivering, and monitoring upskilling measures often occurred at the national level and, for countries with devolved education systems, at the regional or local administration level. Costs also arose to the state and municipal administrative bodies from providing financial incentives to encourage the provision and take up of adult education. Evidence shows that incentives to education providers are most common, followed by incentives to employers and then incentive to learners. Finally, should participating in learning mean that learners become eligible for new welfare payments or support there may also be an increased cost to the State or municipal administrative body. Evidence shows that learners can be expected to face some costs as a result of engaging in upskilling measures, including costs for travel, accommodation, and any equipment such as textbooks (if not reimbursed via the learning provider) and costs in the form of forgone earnings. Overall though the evidence available suggests that these costs for learners are likely to be relatively low.
Q2.2 Which EU programmes have contributed to achieving the objectives of the Recommendation? Was the use of funding from different EU programmes cost-effective?	During the 2016-2020 period, a range of different EU programmes and funding mechanisms have provided funding to support the implementation of the Recommendation. By far the largest source of EU funding was the European Social Fund (ESF), with over EUR 42 billion allocated to investment priority 10.iii (enhancing equal access to lifelong learning) for the period 2016-2020. Although the amounts were much lower, funding was also available under strands of Erasmus+ and EaSI, including specific calls for proposals related to the implementation of upskilling pathways. Funding was also available to support the implementation of the Recommendation under the European Regional Development Fund (ERDF), Horizon 2020 and the Structural Reform Support Programme (SRSP). Under the new Multiannual Financial Framework (MFF) 2021-2027, there are more specific references to upskilling and re-skilling in funding mechanisms, for example ESF+, the Technical Support Instrument (TSI) and Recovery and Resilience Facility (RRF), which is likely to be an impact of the awareness generated by the Recommendation. From 2021, other funding mechanisms supporting upskilling pathways include Erasmus+, EaSI (under ESF+), Horizon Europe, the ERDF and the Just Transition Fund (JTF). Stakeholders in certain Member States (e.g., AT, DE, DK) highlighted however that some upskilling pathways measures are funded solely using national or regional funds. Although it is impossible to provide a full assessment within the scope of this study, the scale of EU funding available, in particular via relevant priorities of the ESF, together with the range of activities targeted by different funding sources, suggests that significant resources were available to meet the needs targeted by the Recommendation. Nonetheless, there is a risk that an over-reliance on EU funding can hinder the large-scale impact of the measures, with project-based measures introduced, as opposed to large-scale reform. This is particularly the case for adult learning, also because national funding for this policy area is often included within broader budget lines for education investments as a whole and can thus remain scattered and insufficient. The supporting study has also found that the use of funding from EU programmes has been cost-effective. The use of existing funds (as opposed to the creation of a new fund or budget line) has allowed the use of existing structures (e.g., ESF and ERDF Managing Authorities, Erasmus+ National Agencies, etc.) and administrative mechanisms (e.g. IT tools, forms, reporting, etc.) to distribute and manage the funds which has led to economies of scale. The main concerns expressed about cost-effectiveness, in particular in relation to the ESF and ERDF, relate to the administrative burden, particularly for smaller beneficiaries. Evidence from the Commission’s ESF evaluation however concludes that, for most stakeholders, the benefits outweigh the costs.
Q2.3 What are the benefits associated with the implementation of the Recommendation for the different stakeholders (including adult learners), at various levels (national, regional, local)?	Adult learning measures introduced in the last five years have contributed to a range of benefits for individual learners, employers, and society, at both the national, regional, and local levels. The main benefits for learners are increased learning opportunities, allowing both unemployed adults and employees to gain and adapt their skills and/or gain a qualification to become more productive. At the employer level, the adult learning measures introduced in the last 5 years have contributed to additional funding for training, fewer skills shortages, and more motivated staff as well as higher levels of productivity. At the level of society, upskilling pathways measures implemented since the Recommendation have likely contributed to increasing income tax and VAT as a result of increased wages and consumer spending and to lowering the cost of welfare payments, though this is challenging to quantify at this stage of implementation. Other benefits highlighted by stakeholders include awareness raising of the importance of prioritising the target group of the low-skilled, and enabling a common framework on which to base discussions and cooperation on adult learning across different stakeholders.
Q2.4 What factors influenced the efficiency and how did they do so? Could benefits have been achieved at a lower cost?	The study has found that the Recommendation overall has been efficient. Many countries already had adult education programmes running in some form prior to the Recommendation, so were able to adapt their current education provision supporting efficiency. The fact that the low-skilled have been prioritised at EU level has also been identified as supporting efficiency as this EU focus has meant that other key stakeholders, including education providers and institutions, have been increasingly engaged in offering courses that are in line with the needs of adult learners. There is also some evidence to show that, through raising awareness among employers that adult learning contributes to fostering productivity, competitiveness, and innovation, employers have become more engaged with adult learning. Covid-19 is noted as having a negative effect on efficiency through slowing down delivery. This meant fewer adults undertook training than originally planned, reducing the scale of the benefits seen by wider society. Other challenges to delivery have also impacted efficiency including the lack of specifically trained staff in the training sector and the difficulty of reaching the target group which meant that more funding to outreach activities, along with better integration of NGOs with existing links to the target groups, was required to bring these people on board. Finally, overlaps identified between other measures focused on skills and adult learning was also noted as limiting efficiency as it has at times caused some confusion among potential learners on where to go to access training and who was eligible. Overall, based on the data available, it is unlikely that benefits of the Recommendation could have been achieved at a lower cost. The evidence has shown that positive relationship between costs and benefits appeared to be mainly due to respondents experiencing low implementation costs and large benefits in terms of raising awareness and stimulating discussions with key stakeholders on the issue of adult education.
Q2.5 Are there significant differences in costs/benefits between the Member States? If yes, what factors may have caused that?	The evidence suggests that the scale of both costs and benefits seen is influenced by the extent to which Member States pre-existing adult education measures and planned direction of travel aligned already with the Recommendation at the time of its adoption. In countries that were already operating measures largely in line to the Recommendation, meaning no major changes in policy or legislation were required, stakeholders confirmed that they incurred no additional cost as a result of the Recommendation, or that these costs were very low. In countries where adult learning systems were overall less in line with the type of provision put forward in the Recommendation costs were higher. However, stakeholders from these countries also felt that benefits were likely to be relatively large, mainly due to the Recommendation increasing awareness of the importance of adult education, providing a framework to improve the provision of adult education, and stimulating cooperation between stakeholders to improve the system.
Q2.6. To what extent are the costs of the actions suggested by the Recommendation proportionate to the benefits brought to individuals, economy and society?	Evidence indicates that the costs and benefits are proportionate. This is due to the belief in the critical nature of the Recommendation in raising awareness of the importance of, and availability of, adult education, along with the Recommendation’s role in focusing political priorities on adult education. This is seen as a large-scale benefit, due to established links in published literature between participation in education and resulting benefits for the individual in the form of increased wages and/or moving into stable employment. By extension, society may then see benefits in the form of reduced welfare payments and increased tax revenue.

Evaluation question	Answer
Relevance 159
Q3.1 To what extent are the objectives defined in the Council Recommendation still relevant to the current socio-economic and EU policy context	The supporting study confirms that the key objectives of the Recommendation continue to be highly relevant to the current EU socio-economic context and the EU policy context. Not only have challenges which were faced across Member States when the Recommendation was adopted in 2016 persisted, but new challenges – in particular those resulting from the impact of the Covid-19 crisis, the ongoing transformation of jobs and work, and the acceleration of the green and digital transition – have in fact heightened the relevance of the Recommendation’s objectives, in particular the need for increasing basic skills. Low-skilled adults in particular are highly vulnerable to the evolution of the socio-economic context, including the acceleration of the need for digital skills, and require increasing policy attention. Despite variation across the Member States, the share of adults with low educational attainment remains high, and participation in adult learning is significantly below the target (60% of all adults participating in training every year) set at the EU Social Summit in Porto. The objectives set out in the Recommendation remain highly relevant to the EU policy context, in particular priorities set out in relation to the overall strategic agenda, the recovery from the pandemic, adult learning and social policy. The objectives have increased relevance in light of the 2030 headline target for adult learning, with a particular need to focus on increasing the participation of low-skilled adults in training in order to reach the goal.
Q3.2 To what extent are the measures defined in the Recommendation still relevant to achieve the objectives?	The measures defined in the recommendation are still very relevant in order to achieve its objectives. The three-step approach remains a very useful framework to guide both policy and implementation of upskilling measures, and each step remains relevant in light of needs and also the ongoing development of upskilling measures in different member states. The accompanying measures - including outreach, guidance, support measures and effective coordination – also remain essential components of approaches to upskilling, in particular when targeting low-skilled adults. The research highlighted however the need to ensure that the different measures set out in the recommendation form part of an integrated whole, with clear permeability and pathways for low-skilled adults between each step, with outreach to attract participants and wrap-around support and throughout the entire process. The fact that low-skilled adults remain a vulnerable group that is further away from the education system – and often the labour market – as whole (see section 3.1), means that such an integrated three-step approach remains necessary for effectively supporting this group through learning pathways that can increase their skills.
Q3.3 To what extent are the measures defined in the Recommendation still relevant to the needs of the target groups/economic sectors (e.g., tourism; textile, construction, mining etc.)?	The measures defined in the Recommendation are still very relevant to the needs of the priority target groups across the EU. In line with the findings noted above, the Covid-19 pandemic has heightened the relevance of the measures defined in the Recommendation to the needs of the target groups, due to higher levels of vulnerability to the impacts of the pandemic. This includes, for example, older low-skilled adults and low-skilled adults working in sectors strongly affected by the crisis. Adult learners themselves, however, highlighted areas in which the relevance of the measures could be further increased for them, including: more ICT courses (computer and smartphone), support for wider aspects of their life, help in combining learning with job seeking, more practical exercises, other types of learning (e.g., accounting, financial management) and more support with transport or work-related issues. The measures are still also relevant to the needs of different economic sectors, in particular as they can be implemented flexibly to adapt to differing and evolving needs. There was clear evidence of regional and sectoral needs being addressed by specifically targeted upskilling initiatives as required based on skills forecasting and analysis. Organisations representing low-skilled adults also noted that there was a need to take into account the geographic (rural/urban) context and training needs that are specific to certain regions.
Q3.4 To what extent are the target groups addressed by the Recommendation and those defined by the Member States still relevant?	The evidence confirms that the broad target group of the Recommendation remains highly relevant across Member States. Low-qualified adults remain further from the labour market than those with higher educational attainment levels as seen in Section 3 of the supporting study. The focus on this target group has become even more relevant in recent years due to the evolving socio-economic conditions, particularly due to the pandemic and rapidly accelerating digital transition, and demographic factors. In terms of specific priority groups within the broader group of low-skilled adults, the flexible framework provided in the Recommendation supports its continued relevance as it allows Member States to identify their own priority groups in light of national, regional or local needs. The most frequently selected priority groups were the long-term unemployed and the inactive. The study showed that other priority groups however also remained relevant, including people with disabilities, older workers, nationals with a migrant background and third country nationals.

Evaluation question	Answer
Coherence 160
Q4.1 To what extent have the objectives, target groups and measures to implement upskilling pathways as defined in this Council Recommendation been coherent with education and training, employment and social policies at national and regional level?	There is overall a good level of coherence between the objectives, target groups and measures defined in the Recommendation and relevant policies, legal frameworks (where they exist) and strategies at national and regional level. Evidence from the supporting study shows that the nature of coherence with national and regional policies falls into three main categories. Firstly, in several Member States (e.g., AT, DK, EE, FR, HU, IE, SE), existing national and regional policies, as well as the legal frameworks (where relevant), for upskilling adults already aligned well with the Council Recommendation when it was published. Secondly, in another set of Member States (e.g., BG, CY, EL, HR, SK), the publication of the Recommendation led to national and regional policies targeting low-skilled adults, as well as in some cases (e.g., EL, HR) the legal framework governing VET, being adjusted to ensure coherence. Thirdly, some Member States (e.g., PL) have introduced new policies to ensure coherence with the Recommendation; although these national and regional policies are often broader (e.g., the Polish Integrated Skills Strategy does not only focus on low-skilled adults), they incorporate the key lines of the Recommendation and have provided important impetus to the policy focus in this field. Even though overall coherence with national and regional policies is good, the case studies have shown that there are areas on which the Recommendation focuses which could be further developed or prioritised in the policies in specific Member States (e.g., training of trainers (FR) or validation of learning (AT, EE)). It is also important to note that, in some countries, while all three steps of the Recommendation can be found in the national offer of training and support for low-skilled adults, they are not necessarily combined in a single programme (please see examples Q1.3 in the effectiveness section in the supporting study). Challenges and obstacles to aligning national and regional policies with the Recommendation include: a focus on other key policy priorities, the time needed to progressively align national policies with the Recommendation, resistance to from certain stakeholders who fear that the existing VET/dual training systems could be undermined), the ongoing absence of a specific national adult learning strategy or system, frequent changes in government slowing down the implementation of new measures, ensuring a tailored approach for low-skilled adults without creating a parallel system or fragmentation, lack of sufficient coordination at national level and the negative impact of the Covid-19 pandemic on adult learning. Factors which have facilitated coherence with national and regional policies include: ensuring that all key national stakeholders have representatives on the board of the national training agency and implementing joint measures, integrating the objectives, target groups and measures into the broader government's programme at the inter-institutional level, setting up an upskilling pathways working group chaired the by the Ministry for Education or the Ministry of Employment, the existence of specific national policies or strategies for upskilling low-skilled adults and the availability of specific EU funding for upskilling measures.
Q4.2(a) To what extent have the objectives, target groups and measures to implement upskilling pathways as defined in this Council Recommendation been coherent with other related EU level policies?	The objectives, target groups and measures of the Recommendation have a good degree of coherence overall with other related EU level policies in the fields of training and employment, adult learning, equality strategies and other fields (e.g., recovery or digital strategies). These other EU policies are complementary to the Recommendation and do not duplicate it, with limited potential for overlap. The analysis shows a general trend of an increasing specific focus and recognition of the importance of upskilling under other relevant EU policies over time, most likely linked to the influence of the Recommendation. Despite the overall positive assessment of coherence with other EU policies, potential areas for improvement include: creating better linkages and synergies with policy fields outside the sphere of employment (e.g. education, social policy, health, environment); developing a more comprehensive, systematic and strategic approach to gender and equality mainstreaming within the Recommendation; and providing more guidance to national and regional stakeholders about how the multiple different EU policies, strategies, and recommendations in the fields of training, adult learning, skills/competences, and labour market integration work together.
Q4.2(b) To what extent have the objectives, target groups and measures to implement upskilling pathways as defined in this Council Recommendation been coherent with EU funding mechanisms?	The Council Recommendation has a high degree of coherence with relevant EU funding mechanisms, in particular the European Social Fund (ESF(+)), Erasmus+, Employment and Social Innovation (EaSI) programme, the European Regional Development Fund (ERDF), the Just Transition Fund (JTF), the Structural Reform Support Programme (SRSP), the Technical Support Instrument (TSI) and the Recovery and Resilience Facility (RRF). Upskilling measures which are coherent with the aims of the Recommendation have been funded under these different EU funds. To further enhance coherence with EU funding, clearer signposting could be provided to national and regional stakeholders of the EU funds available to support upskilling pathways. An overview of the specific measures of the funding instruments which allow support for the implementation of upskilling pathways is set out in in Q.2.2 (see Efficiency section in the supporting study). The analysis shows that there is an increase in the specific focus on the implementation of upskilling measures in EU funding mechanisms over time, for example with increased and more specific funding possibilities under the new Erasmus+ and ESF+ programmes compared to previous programming periods.
Q4.3 To what extent have objectives, target groups and measures been internally coherent?	The Recommendation shows a very good degree of internal coherence. There is clear coherence between the general, specific and operational objectives, inputs, activities and expected outputs, results and impacts in the intervention logic. The measures proposed in the Recommendation (three-step approach, coordination and partnership, outreach, guidance and support measures, follow-up and evaluation) provide a comprehensive and complementary policy framework which is internally coherent.

Evaluation question	Answer
EU added value 161
Q5.1 What has been the added value of this Council Recommendation in promoting, informing and supporting a common approach to offering upskilling pathways to low-skilled/qualified adults, in terms of their objectives, design, target group and coordination and support measures?	The Recommendation generated EU added value in four main areas. It informed the design and a common framework for national upskilling policies including a three-step approach. It promoted the focus on low-skilled/qualified adults as a key target group for upskilling measures and raised awareness about multiple vulnerable groups among the low-skilled/qualified population. It provided an impetus for more structured coordination, knowledge sharing and mutual learning activities among key stakeholders, contributing to a more joined-up approach, synergies in searching for solutions, sharing good practice and lessons learnt and increased peer support for tackling labour market needs across the EU. Finally, it offered financial support by leveraging funding at EU and national/regional level for delivering upskilling measures in the Member States. The supporting study also identified some ways through which EU added value of the Recommendation could be increased. These include: Improving visibility of the Recommendation at the national level by strengthening the linkages between the national and regional support measures and the Recommendation; further acknowledging the diversity of the target group and encouraging implementation of measures targeting sub-groups of low-skilled; continuing to tailor and adapt the Recommendation to the specific policy context and institutional frameworks of each country; and continuing to provide supervision and support at EU level with room for increasing mutual learning and knowledge-sharing.
Q5.2 Could the objectives of the Recommendation have been achieved sufficiently by the Member States acting alone?	Evidence from the study indicates that the objectives of the Recommendation could not have been achieved to the same extent without the EU level intervention. Despite education being a national competence, there were changes or reforms in national approaches following the Recommendation’s adoption and the mutual learning activities that followed. Countries expanded the implementation of existing upskilling measures to low-skilled adults and strengthened their focus on the three-step approach (see Section 3 of the supporting study for further analysis).
Q5.3 To what extent do the objectives and measures addressed by the Recommendation continue to require action/support at EU level?	The objectives addressed by the Recommendation continue to require action and support at EU level. Although some progress has been achieved regarding the implementation of the Recommendation, some countries are only at the beginning of their reform process, meaning that knowledge-sharing, guidance and collaboration will still be valuable at EU level. Securing financial resources, support in building new and stronger partnerships among Member States by broadening existing mutual learning activities to a wider group of stakeholders, providing a framework for monitoring and evaluating the implementation of upskilling measures and EU action support in improving outreach measures are areas that would bring key added value moving forward.

Annex IV. Overview of benefits and costs and, where relevant, Table on simplification and burden reduction]

The Table provides information on the resources used to implement the Recommendation and the changes generated (i.e. an assessment of costs and benefits) as identified by the evaluation. Source of the information is the supporting study. See annex 11 of the supporting study.

Cost	Cost description	Type (one-off or recurrent)	Citizens/consumers		Businesses		Administrations
			Quantitative	Comment	Quantitative	Comment	Quantitative	Comment
Direct compliance costs
Adjustment costs: Implementation costs	Activities undertaken to enable familiarisation with the Recommendations	One-off	Not available		Not available		Not available	National and regional ministries and government bodies responsible for education/employment policies provided internal staff and stakeholder training to provide information on the Recommendations. No monetary estimates provided.
Adjustment costs: Direct labour costs	Developing and/or rewriting existing qualifications frameworks and standards, and (re)designing modular courses.	One-off (per course)	Not available		Not available		Not available	For policymakers and organisations representing low-skilled adults at the EU level, these costs are typically limited to one or two staff members working across adult education rather than focusing purely on UP measures, so are estimated to be low (no monetary estimates provided).
Adjustment costs: Equipment costs	Modernising and improving learning spaces	One-off	Not available		Not available		Not available	Approximately four fifths of survey respondents noted that they had incurred infrastructure costs to some extent. Only two Member States spoke of this cost during interviews, noting they experienced costs in modernising and improvement learning spaces. No monetary estimates provided.
Other	Advertising/marketing costs to raise awareness of courses among citizens	Recurrent	Not available		Not available		Not available	Advertising/marketing costs to raise awareness of courses among citizens of learning opportunities. No monetary estimates provided.
	Financial incentives	Recurrent	Not available		Not available		Not available	Approximately a third of interviewees and survey respondents mentioned providing some form of financial incentive to individual learners, employees and/ or training providers. Bulgaria highlighted that vouchers for the employed to undertake training had been introduced as a result of the Recommendations, with a total project cost of €15.3million over 2018 to 2021.
Indirect costs
Indirect costs to individual learners	Expenses (travel and accommodation costs), course supplies such as textbooks, reduction in leaners income due to a reducing their paid work hours to completing the training.	One-off (per course per individual)	Not available	The survey of adult learners found 59% of leaners faced training cost. Estimates provided range from €10 to over €700 per course.	Not available		Not available
Direct economic benefits
Improved welfare	Upskilling of population/reduction in skill shortages	Recurrent	Not available	Gaining a qualification, moving into further education, gaining basic literacy, numeracy and digital skills, improved soft skills. No monetary estimates provided.	Not available	Improvement in skills and knowledge of employees and job applicants. No monetary estimates provided.	Not available	Increase in the share of the population with the skills needed for green and digital transitions. No monetary estimates provided.
	Increase in employment & productivity	Recurrent	Not available	Moving into employment, getting a promotion. No monetary estimates provided.	Not available	Increased in employee productivity. No monetary estimates provided.	Not available
Improved market efficiency	Increased learning opportunities and information on opportunities available	Recurrent	Not available	Increased number of adult education courses available. No monetary estimates provided.	Not available		Not available	Increased awareness of citizens and employers of adult education courses and the benefits of life-long learning. No monetary estimates provided.
Indirect benefits
Wider macroeconomic benefits	Reduced welfare payments and increased tax revenue	Recurrent	Not available		Not available		Not available	Reduced welfare payments and increased tax revenue following on from upskilling the population and an increase in employment. No monetary estimates provided.
Wider welfare benefits	Improved individuals’ participation in society	Recurrent	Not available	More integrated into society. No monetary estimates provided.	Not available		Not available
	Improved individuals health and wellbeing	Recurrent	Not available	Improvement to individuals’ health and wellbeing. No monetary estimates provided.	Not available		Not available	Reduced public health spending. No monetary estimates provided.
	Reduction in crime	Recurrent	Not available	Drop in crime rates. Noted as a possible future benefit by one Member State. No monetary estimates provided.	Not available		Not available	Reduction in the financial cost associated with crime (e.g. prisons and enforcement).
	Improving cohesion of society and equal opportunities	Recurrent	Not available	Improving cohesion of society and equal opportunities. No monetary estimates provided.	Not available		Not available

Annex V. Stakeholders consultation - Synopsis report

Outline of the consultation strategy

This document provides a synopsis of the stakeholder consultations conducted for the evaluation of the Council Recommendation of 19 December 2016 on Upskilling Pathways: New Opportunities for Adults. The consultation strategy was developed on the basis of the Evaluation Roadmap. 162 The Roadmap indicates that the aim of the consultation activities is to gather the views and opinions of the wide variety of stakeholders involved in policy making and implementation of the types of measures outlined in the Recommendation, to inform the evaluation work. The consultation activities addressed the effectiveness, efficiency, coherence, relevance and EU added value of the Recommendation and the progress made towards raising the levels of literacy, numeracy and digital competence amongst low-qualified/low-skilled adults.

A wide range of stakeholders were involved in the consultation process, including national and regional authorities responsible for adult learning/training, vocational education and training, lifelong learning, skills or employment policies; social and economic partners; education and training providers; organisations representing the low-skilled; individuals benefitting from upskilling pathways interventions; as well as other relevant stakeholders at European, national, regional level; and the general public.

To adequately reach these stakeholders, different consultation activities and methods 163 were used:

·A public consultation, including position papers submitted by organisations responding to the public consultation;

·Targeted consultations, including:

oInterviews with stakeholders on the EU and national level

oCase studies of 10 countries 164 which also included consultations with key stakeholders in the form of in-depth interviews

oThree internet-based surveys targeting coordinating and implementing organisations of upskilling measures, organisations representing/working with the low-skilled and adult learners themselves;

·Other targeted consultations including exchange of views with the Directors General for Vocational Training (DGVT) and the  Advisory Committee on Vocational Training (ACVT), exchange of views with the European Network of Public Employment Services, consultation of the National coordinators for adult learning and the EQF advisory group). 

The consultations were implemented as foreseen.

Overview of consultation activities

The evaluation roadmap was open for consultation for four weeks between 12 January 2021 and 9 February 2021, with the aim of gathering the views of relevant stakeholders on the Commission's understanding of the problem and possible solutions and to share any relevant information that they may have on the evaluation of the Recommendation.

The main aim of the public consultation was to provide an open channel for all interested stakeholders to provide their input. The consultation took the form of an e-survey with closed and open questions, addressing the key evaluation criteria and tailored to the different stakeholder groups. The public consultation was open for 13 weeks from 16 December 2021 to 17 March 2022.

The targeted consultations aimed to gather more detailed input from stakeholders involved in the implementation of the Recommendation in different capacities. The targeted consultations consisted of six main sub-activities as described in the following sections.

Targeted interviews with EU and national level stakeholders focused on two key stakeholder groups: EU level stakeholders directly and indirectly linked to the education, training, and adult learning policies; and national level stakeholders responsible for adult learning, training, vocational education and training, skills policies (these included primarily Ministries of Employment and/or Social Affairs and Ministries of Education, as well as other stakeholders such as national agencies responsible for education and training, adult learning or vocational education and training that are not direct Ministries). The targeted semi-structured interviews were conducted from October 2021 to February 2022.

A selection of ten case studies were conducted as part of the evaluation research. As part of the case studies, additional interviews were conducted with representatives involved in the practical implementation, operations, and monitoring of national approaches to the implementation of the Council Recommendation. The case study interviews were conducted from March to April 2022.

Three internet-based surveys were conducted. The first survey targeted coordinating and implementing organisations, with a focus on national and regional authorities in charge at operational level of upskilling initiatives. The second survey targeted organisations representing potential target groups of the Recommendation and working with low-skilled adults. The third survey aimed to gather the views of beneficiaries of upskilling measures namely adults with low levels of skills, knowledge and competences and organisations representing them.

An expert meeting was held online on 24 February 2022 to gather the views of selected consultees on preliminary findings on all evaluation criteria, and provide examples of approaches, good practices and lessons learnt. Consultees included thematic experts, as well as representatives from national authorities with overall responsibility for the implementation of upskilling pathways, and those in charge of implementing upskilling pathways on the ground.

A validation workshop was held on 1 June 2022 as a final consultation activity to validate the findings of the evaluation study and for selected consultees to share their views on the findings and lessons learned. Selected consultees included representatives of national authorities in charge of the Recommendation, representatives from social partners and EU level civil society organisations and experts.

Lastly, other targeted consultations included an exchange of views in dedicated meetings with the Directors General for Vocational Training (DGVT), the Advisory Committee on Vocational Training (ACVT), the European Network of Public Employment Services (PES Network), the national coordinators for adult learning and the EQF advisory group.

Summary and reflection on challenges

The consultation methods were implemented as planned, in line with the agreed consultation strategy. The targeted interviews posed some challenges, such as delays due to the unresponsiveness of a number of national level stakeholders. Moreover, some EU level stakeholders felt that they did not have the relevant expertise to contribute to the study but provided contacts of their national partners. There were some challenges in securing answers to the surveys, as there was a geographical imbalance in the responses, with certain countries being over-represented. The impact of this on the consultation strategy was minimal, however, as the other targeted consultations conducted, and the interviews in particular, allowed the research team to still gather in-depth views of a range of stakeholders across the EU27.

Information on consultation activities and stakeholder groups consulted

All relevant stakeholders as identified in the evaluation roadmap were consulted through the consultation activities, as set out in Error! Reference source not found. .

Table 1 - Overview of stakeholders consulted in all consultation activities

Type of stakeholder	Public consultation	Evaluation roadmap	Targeted interviews	Surveys	Expert meeting	Validation workshop	Other targeted consultations
Ministries and institutions responsible for education and training policy	X	X	X	X	X	X	X
Ministries and institutions responsible for employment policies	X	X	X	X	X	X	X
Other relevant public institutions responsible for adult learning, training, VET and skills policies	X	X	X	X		X	X
Social and economic partners at national and EU level	X	X	X	X		X	X
Education and training providers	X	X	X	X			X
EU level institutions/ policymakers	X	X	X	X			X
Organisations representing low-skilled adults and target groups	X	X	X	X
Beneficiaries	X	X	X	X
Research /academia and other international organisations	X	X	X			X
Citizens	X	X

Evaluation Roadmap

The total number of respondents to the consultation on the evaluation roadmap was 14. These included six responses from non-governmental organisations (NGOs), five responses from economic and social partners (four trade unions and one business association), one from EU citizens, and two from other types of stakeholders. 165  

Public consultation

The total number of respondents to the public consultation was 60. The data on respondents showed a somewhat unbalanced geographical distribution. A total of eight responses referred to Belgium as their country of origin (including four EU level organisations and one international organisation), followed by seven to Lithuania, five to both Italy and Germany, and three to Ireland and Poland. Not all EU Member States were represented among the respondents, and a total of four responses were received by respondents from non-EU countries.

In terms of the type of respondents, the public consultation targeted the following main groups:

·Group A – organisations: Public authorities, NGOs, companies or business organisations, and trade unions;

·Group B – individuals: either EU or non-EU citizens;

·Group C – others: respondents representing other types of groups than those listed in Group A or B.

Out of the 60 respondents, the stakeholders with the largest number of responses were public authorities (n=13, 22%) and individuals who are EU-citizens (n=13, 22%), followed by non-governmental organisations (10, 17%) and those who indicated that they belonged to a group not listed as a main stakeholder group in the questionnaire (i.e. Other) (n=10, 17%). The remaining respondents represented company/business organisations (n=6, 10%), trade unions (n=3, 5%), business associations (n=3; 5%) and non-EU citizens (n=2, 3%). Figure 1 presents the breakdown of respondents by main stakeholder groups.

Figure 1 - Breakdown of respondents by main stakeholder groups

Respondents were asked to further specify the capacity in which they are replying to the public consultation by indicating which sub-group of stakeholders they belonged to from the list provided. In total, 56 respondents (93%) gave their answer to this question, Figure 2 presents the breakdown of respondents by sub-group of stakeholders and shows that the largest group of respondents belonged to public authorities or bodies which are involved in adult learning/training, vocational education and training, lifelong learning, skills and/or employment policies (12 out of 56, or 21%).

Figure 2 – Breakdown of respondents by sub-group of stakeholders

Ten position papers were also submitted in response to the public consultation outlining different organisations’ views on the Recommendation/.

Targeted interviews with EU and national stakeholders

A total of 123 interviews were undertaken. 14 interviews of EU level stakeholders and 109 interviews of national level stakeholders were undertaken.

Table 2 – Targeted interviews completed

EU level
DG EAC - Unit B2 (2 interviews)
DG EMPL - Unit G1
Eurochambers
All-Digital
European Federation of Education Employers
European Training Foundation
European Association for the Education of Adults (2 interviews)
European Association of Regional & Local Authorities for Lifelong Learning
ERGO Network
European Forum for Technical and Vocational Education and Training
European Basic Skills Network
Eurofound
Total EU level interviews	14
National level
Type of stakeholder	Countries covered	Number of interviews conducted
National and regional ministries and government bodies responsible for adult learning/training	AT, BE, BG, CY, CZ, DK, EE, EL, ES, FI, FR, DE, HR, HU, IE, IT, LV, LT, LU, MT, NL, PL, PT, RO, SI, SE	41
National and regional ministries and government bodies responsible for employment policies	BE, BG, CZ, EE, EL, ES, FI, FR, HR, HU, IE, IT, LV, LT, RO, SK	22
Other national and regional public authorities or bodies	BE, CY, DK, EL, ES, FI, HU, IE, LV, LU, MT, PL, RO, SK, SE	35
Social partners, economic actors, and EU fora on skills	CZ, FR, IT, MT, PT	9
Representatives of education and training providers	HR, IT	2
Total of national level interviews		109

Case studies

48 interviews were conducted for the case studies, with stakeholders from the following categories in each of the case study countries:

·Ministries, governmental departments, or intermediaries responsible for implementing the initiatives;

·Regional or local authorities that are engaged with upskilling pathways, adult learning or lifelong learning strategies;

·Relevant implementing organisations of selected initiatives/schemes;

·NGOs, think tanks; research centres, academics or other organisations that engage with public policies in the areas of skills and adult learning

·social partners (trade unions and employers);

·Public employment services

·VET centres and education centres involved in the practical implementation of upskilling initiatives

Internet-based surveys

A total of 108 responses were received for the survey targeting organisations either coordinating the implementation of upskilling measures or implementing upskilling measures. The data showed a somewhat unbalanced geographical distribution in the responses. While respondents represented all Member States with the exception of Hungary, a total of 21% (n=23) of respondents selected Bulgaria as the country in which their organisation was active. This was followed by Italy (n=12, 11%), Lithuania (n=9, 8%) and Portugal (n=6, 6%). Furthermore, 5% of respondents selected Belgium (n=5) and Germany (n=5), whilst 6 countries received 4 responses each (Cyprus, Denmark, Greece, Ireland, Portugal, and Romania), accounting for 4% of the total number of responses. The remaining 14 countries had 3 or less responses.

A total of 180 responses were received for the survey targeting representative organisations of low-skilled adults. 129 (72%) responses were submitted by organisations based in Portugal. The remaining responses were submitted by organisations based in Poland (n=9, 18%), Austria (n=7, 14%), Bulgaria (n=7, 14%), Germany (n=5, 10%), Slovenia (n=4, 8%), Belgium (n=3, 6%), Greece (n=2, 4%), Italy (n=2, 4%), Romania (n=2, 4%), Sweden (n=2, 4%).

A total of 119 responses were received for the survey of adults participating in upskilling initiatives. However, more than half of the responses were submitted from Portugal (n=41, 40%) and Poland (n=33, 32%), and Italy (n=17, 17%). The remaining 11 responses were submitted from Slovenia (n=6, 6%), Romania (n=2, 2%), and Austria (n=1, 1%), Bulgaria (n=1, 1%), and Estonia (n=1, 1%),

Expert meeting

The online expert meeting was attended by a total of 40 participants, 25 of these were representatives from 12 Member States (Austria, Cyprus, Greece, Finland, Italy, Ireland, Latvia, Netherlands, Poland, Portugal, Romania, Spain). The remaining 15 were representatives from the European Commission and the research team.

Validation workshop

The online validation workshop was attended by a total of 32 participants. Of these, seven were representatives of EU level organisations, while 15 were representatives from 14 Member States (Belgium, Czechia, Denmark, Finland, Greece, Ireland, Italy, Latvia, Luxembourg, Malta, Poland, Slovenia, Spain, Sweden). The remaining 10 were representatives from the European Commission and the research team.

Other targeted consultations

The following additional online meetings were organised as part of the targeted consultations:

·EQF Advisory Group consultation on Upskilling Pathways: this meeting took place on 25 January 2022 and was attended by 37 participants;

·PES Network and National Coordinators for Adult learning consultation on Upskilling Pathways: this meeting took place on 4 February 2022 and was attended by 34 participants;

·Upskilling Pathways Consultation webinar with ACVT/DGVT members: this meeting took place on 15 February 2022 and was attended by 41 participants.

Methodology for data processing

The feedback on the evaluation roadmap was conducted by carrying out a qualitative analysis of the responses to identify common trends and relevant insights.

For the public consultation, the analysis of results was carried out using both quantitative and qualitative methods. The quantitative data analysis included analysis of frequency distribution for each of the variables related to the closed-ended questions, and cross-tabulations between specific variables and characteristics of respondents and between specific variables, though the low number of responses has limited the possibilities for this. For the qualitative data analysis, information was classified by related variable (number of question) and analysed to identify additional information and trends.

For the targeted interviews with EU and national level stakeholders, the write-ups from the interviews were collected and exported into analytical grids, broken down by the different questions and by the respective evaluation criteria. The research team used the analytical grids to carry out an in-depth analysis of the data to inform the relevant sections and annexes of the interim and final reports.

Information gathered from the case studies interviews was used in the analysis of each country case study report. The case study reports were used to inform the analysis carried out by the research team for all sections of the final report.

The surveys were analysed using both quantitative (to analyse the frequencies of the closed answers) and qualitative methods (for the open questions, to analyse complex concepts and substantiate and interpret the quantitative data with relevant insights). Survey findings were used by the research team for all sections of the final report.

For the expert meeting and validation workshop, the outcomes of the discussions were collected in meeting reports drafted by the research team. These were used by the research team for all sections of the final report.

Overview of results of consultation activities

Effectiveness

Consulted stakeholders largely agreed that Member States have facilitated access to upskilling opportunities since the adoption of the Upskilling Pathways in 2016. 65% (66 out of 101) of survey respondents from organisations coordinating and implementing upskilling initiatives agreed that adults with low levels of skills have had greater access to adult learning measures over the past five years to a great or a fairly large extent and 65% (31 out of 48) of respondents to the public consultation stated that they consider the Recommendation as very or mostly effective in helping low-skilled adults take part in adult learning measures. However, stakeholders across different groups highlighted that differences remain across Member States. This mixed picture is reflected in the interviews, where national authorities consulted emphasised that the situation is not uniform across the EU27. This view is confirmed by the survey of implementing and coordinating organisations, with just over half of respondents (21 out of 41, 51%) stating that the Recommendation has contributed only to a small extent to changes in levels of literacy, numeracy and digital skills. Moreover, both the responses to the Roadmap and the feedback from the EU level stakeholders and national authorities participating in the validation workshop highlighted the need to ensure that the measures to implement the Recommendation are adequate and high quality.

There was large agreement amongst stakeholders (national authorities, PES, trade unions, EU level stakeholders) consulted that Member States have identified a broad target group for upskilling pathways, with most measures targeting the ‘low-skilled’ in general. In the survey of implementing and coordinating organisations, 77 out of 108 respondents (71%) identified adults with low levels of basic skills as the priority target group of adult learning measures. Nonetheless, the surveys and interviews (national authorities, PES, trade unions, EU level stakeholders) show that Member States have made some efforts to prioritise the most in need within this broader group. This is clear in the results of the survey of implementing and coordinating organisations, where the long-term unemployed (67% or 72 out of 108 respondents) and economically inactive people (47% or 51 out of 108 respondents) were the second and third most selected target groups of adult learning measures. The case studies also confirm this, for example by highlighting that a number of Member States have prioritised vulnerable groups in their provision of upskilling measures. Despite this generally effective identification of the target groups most in need, the study (e.g., responses to the Roadmap, surveys) has also identified some groups that have not been adequately prioritised, including older workers, which were identified by 24% (26 out of 108) of respondents to the survey of implementing and coordinating organisations as the group that should have been targeted by upskilling measures since 2016, but was not.

Stakeholders from implementing organisations in particular identified key success factors in reaching target groups including: informal outreach by establishing personal relations with target groups and involving actors outside public authorities; individual and tailored approach to training offers; strong cooperation between actors. However, some barriers were also identified including financial obstacles; stigmatisation of the target group and its impact on motivation; global pressures of digitalisation and automation which have resulted in an increased need for upskilling beyond the low-skilled.

The consultation activities show that Member States have made efforts to apply or improve the three-step approach in the design of upskilling pathways since 2016:

·Consultations with national authorities and implementing organisations through the interviews, case studies and surveys confirm that the skills assessment is the most common step, with 60% (29 out of 48) of respondents to the public consultation indicating it had been implemented to a very large or fairly large extent in their country or in the EU, and 67% (52 out of 79) of respondents to the survey of coordinating and implementing organisations stating that skills assessment measures are accessible to priority target groups.

·Tailored learning offers are also a relatively common step, with the 73% (80 out of 108) of survey and 65% (31 out of 48) of public consultation respondents across all groups indicating that learning offers are tailored to the needs of target groups to a large or fairly large extent.

·Validation is the step that is the least implemented in Member States, with 40% (19 out of 48) of respondents to the public indicating it had been implemented to a small extent or not at all, as establishing fully functioning validation systems remains a challenge regardless of the strength of existing adult learning provisions.

Consultees across groups agreed that different stakeholders are involved in the implementation of the Recommendation - from social partners, to PES, and local level actors (e.g., NGOs) - and that structures for cooperation (e.g. central working groups or councils) exist in the majority (22) of Member States. The involvement of stakeholders was also identified as an important aspect of upskilling measures by respondents that submitted a position paper in the context of the public consultation. However, results from the public consultation suggest that cooperation is not always fully effective: while the validation and recognition of skills acquired is seen as being delivered to a fairly large or very large extent in cooperation with relevant stakeholders by 51% (23 out of 45) of respondents, the percentage of respondents selecting indicating that cooperation only happens ‘to a small extent’ or ‘not at all’ is only marginally different (40% or 18 out of 45). Replies to the Roadmap, as well as views expressed in the surveys, interviews, expert meeting and validation workshop further highlight that the involvement of local actors should be improved, particularly to strengthen guidance and outreach, and that social dialogue can play a key role in the design, implementation and monitoring of upskilling measures.

Despite variations in the level of implementation and impact across the EU, stakeholders (national authorities, PES, trade unions, EU level stakeholders) agree that the Recommendation has acted as a catalyst to accelerate progress in upskilling, by bringing increased attention to adult, stimulating cooperation between relevant actors, raising awareness on the challenges faced by the low-skilled. However, EU level stakeholders providing responses to the Roadmap and written contributions following the validation workshop highlighted the lack of systematic data on the upskilling measures implemented as a critical gap that does not allow to identify the direct impact of the Recommendation on the target group and on the situation of the low-skilled in Europe as a whole.

Efficiency

Stakeholder consultations identified a wide range of costs associated with the Recommendation:

·Adjustment/administrative costs: The survey of coordinating and implementing organisations found that 82% (55 out of 67) of the respondents had incurred adjustment/administrative costs to implement adult learning measures over the last five years. Qualitative evidence from the interviews and the public consultation also identified staffing costs as a common expenditure linked to designing, delivering, and monitoring the Recommendation. Case studies further highlighted costs associated with organising meetings/information days to inform other stakeholders of the Upskilling Pathways measures.

·Financial incentives: Financial incentives (e.g., to education and training institutions; employers; learners) were mentioned both by respondents to the surveys and the public consultation, as well as by interviewees particularly national authorities, as a common cost.

·Costs incurred by learners: The consultations showed a difference in opinion with the regards to costs incurred by learners as a result of engaging in upskilling measures (e.g., travel and accommodation costs; education and training material; forgone earnings to take part in training). 58% (60 out of 102) of respondents to the survey of adult learners indicated that they did not incur in any cost, whilst interviewed national authority stakeholders highlighted that costs to learners can be high.

Adult learning measures introduced in the last five years have been identified by stakeholders as contributing to a range of benefits for society, employers, and individual learners. In terms of benefits for the learners, according to the survey of coordinating and implementing organisations, approximately nine in ten respondents considered that the adult learning measures introduced in the last five years contributed to individuals gaining a qualification (94%, 96 out of 102), gaining access to further education (93%, 95 out of 102), gaining access to employment (92%, 94 out of 102), and gaining basic literacy, numeracy and digital skills (89%, 91 out of 102). In addition, the vast majority of respondents to both the survey of coordinating and implementing organisations and the survey of organisations representing low-skilled adults noted that the measures contributed to individuals gaining more confidence (82%, 88 out of 102, and 86%, 44 out of 51 respectively), improved individuals’ participation in society (77%, 83 out of 102, and 94%, 48 out of 51 respectively), and improved individuals health and wellbeing (73%, 74 out of 102, and 76%, 39 out of 51 respectively). Respondents to the survey of adult learners highlighted a similar range of benefits for adult learners, additionally finding that leaners benefited from being able to change careers, from gaining a higher paid job, and from being able to better support others in their learning.

At societal level, consultees, in particular EU level stakeholders identified greater attention for education provision for low-skilled adults as a political priority as one of the main benefits of the Recommendation. According to national authorities consulted through the interviews and the case studies, the Recommendation not only provided a common framework, but also pushed authorities to give more attention to adult education. This, moreover, was perceived by consultees as having positive ripple effects in the medium to long term.

While most consulted stakeholders, due to lack of data, could not identify factors that influenced efficiency or provide views on whether benefits could have been achieved at a lower cost, the overall consensus was that the Recommendation has contributed to improving efficiency of adult learning measures. However, it is clear from the outcomes of the interviews and surveys that the Covid-19 pandemic had a negative impact on efficiency, as it slowed down delivery efforts. This was mentioned by stakeholders from all stakeholder categories.

Relevance

Consultations showed a strong agreement across stakeholder categories that the key objectives of the Recommendation continue to be highly relevant to the current EU socio-economic context, with 87% of the respondents to the public consultation (41 of 47) indicating that providing learning opportunities and further guidance support to low-skilled adults is still very relevant today. 95% (39 of 41) of the respondents to the survey of coordinating and implementing organisations also agreed that the objectives of the Recommendation were still relevant to the current socio-economic and policy context in their country or region.

Respondents to both the public consultation and the consultation on the evaluation roadmap stressed the relevance of the Recommendation in responding to the impact of the COVID-19 pandemic, as well as the digital and green transformation, particularly in relation to supporting vulnerable groups. The impact of the Covid-19 pandemic on the relevance of the Recommendation was also highlighted in replies to the Roadmap, in the position papers submitted in response to the public consultation and was confirmed by the survey of coordinating and implementing organisations, with 55% (n=21) of respondents noting that the pandemic had increased the relevance of the measures defined in the Recommendation to the needs of the target group.

Consulted stakeholders (EU level stakeholders, national authorities responsible for education and employment policies, social partners) also agreed as a whole that the three-step approach in the Recommendation provides a useful structure and guidance for Member States to adopt measures addressing specific needs at the national or regional level. National authorities responsible for upskilling that participated in the expert group agreed that developing skills assessments is still a very relevant measure for achieving the goals of the Recommendation. Moreover, consulted stakeholders identified ensuring validation and recognition of skills and competences as a key area of relevance to support individuals in the recognition of their skills and employers in understanding how these match their needs. National authorities responsible for upskilling participating in the validation workshop highlighted the three-step approach represents a key strength of the Recommendation, and that, for the future, efforts should focus on fostering the functional integration of all three steps.

Lastly, the outcomes of the targeted consultations highlighted that the priority target groups set out in the Recommendation were still considered to be relevant. This was further confirmed by the outcomes of the public consultation, to which all target groups of the Recommendation listed were considered to be very relevant, with respondents identifying adults with low levels of basic skills (33 respondents out of 45, 73%), older workers above 55 years old (32 respondents out of 45, 71%), and long-term unemployed adults (32 respondents out of 45, 71%) as the most relevant sub-groups. However, consulted stakeholders across all stakeholder categories emphasised the need to pay greater attention to the heterogeneity of the priority groups by ensuring that measures are better tailored to the needs of specific sub-groups of the low-skilled where needed, and that more efforts are made to strengthen outreach towards these groups and increase awareness of the need for upskilling.

Coherence

Overall stakeholders (EU level stakeholders, national authorities responsible for education and employment policies, social partners) expressed the view that there is a good level of coherence between the objectives, target groups and measures defined in the Recommendation and relevant policies and strategies national and regional level. The survey of coordinating and implementing organisations shows that a very high share of respondents (88% or 36 out of 41) considered that the objectives, target groups and measures to implement adult learning measures have been coherent with education and training, employment and social policies at national/regional level. Stakeholders shared the opinion that the level of coherence tends to be much higher in Member States where specific national policies or strategies for upskilling low-skilled adults exist.

Despite the overall positive feedback, stakeholders from national authorities in particular underlined that aligning national and regional policies with the Recommendation can still present challenges, which vary depending on the specific national context. In particular, EU stakeholders consulted in the validation workshop and the case study interviews identified a number of areas for improvement including: a more specific focus on diversity, gender equality and training of trainers in relation to upskilling; the validation of non-formal and informal learning; and the integration of all three steps of the Recommendation into a single programme.

Both EU and national level stakeholders (national authorities responsible for education and employment policies, social partners) consulted shared the view that the Recommendation has a good degree of coherence with other related EU policies and that there is no or very limited duplication or overlap, and good complementarity. However, there was a difference in views between EU level stakeholders and national organisations implementing upskilling activities: 30% (12 out of 40) of respondents to the survey of coordinating and implementing organisations did not know or did not wish to provide an answer when asked about complementarity and synergies between the Recommendation and other EU policies. ‘Do not know’ was also the most common response to the question in the public consultation about the complementarity and synergies between the Recommendation and other selected EU policies.

Lastly, consulted stakeholders across all categories largely agreed that the Recommendation has a high level of coherence with relevant EU funding mechanisms, albeit to varying degrees depending on the type of instrument. In responses to the public consultation and the survey of coordinating and implementing organisations, the European Social Fund (57%, 25 out of 44, and 79%, 30 out of 38 respectively) and Erasmus+ (46%, 21 out of 46, and 55%, 21 out of 38 respectively) received the largest number of positive responses with regards to their contribution to achieving the objectives of the Recommendation.

EU added value

The perception of EU added value resulting from the Recommendation was positive overall among the range of stakeholders consulted.

The public consultation, the surveys and the interviews show that the Recommendation has likely contributed to an increased provision of validation and recognition of skills. Stakeholders across all consultation activities recognised that the Recommendation fostered cooperation between key actors, by highlighting the importance of involving them at all levels and in different stages of policy design and implementation. Views of organisations implementing upskilling activities were less positive, however, in relation to the extent to which the Recommendation has led to greater convergence between approaches adopted at Member State level: less than half (37% or 14 out of 38) of respondents to the survey of coordinating and implementing organisations indicated that the Recommendation has led to greater similarities across Member States.

There was clear consensus amongst all stakeholders (EU level stakeholders, national authorities responsible for education and employment policies, social partners) consulted that the key added value of the Recommendation was increasing awareness of the importance of recognising the heterogeneity of the target group and their different skills needs, and by encouraging knowledge-sharing and mutual learning across Member States. Moreover, consulted stakeholders agreed that EU funding provides significant EU added value, as it ensures the continuation of existing upskilling measures.

The majority of interviewed stakeholders across all categories (EU level stakeholders, national authorities responsible for education and employment policies, social partners) generally agreed that that the objectives of the Recommendation could not have been achieved to the same extent without the EU level intervention. This view was confirmed by respondents to the survey of coordinating and implementing organisations, as well as by stakeholders from national authorities interviewed. However, the outcomes of the stakeholder consultations identified some key areas where additional EU action/support will be needed, and that would increase EU added value. These include building new and stronger partnerships among Member States by involving key actors in existing mutual learning activities; providing a framework for monitoring and evaluating the implementation of upskilling measures; improving outreach measures to target vulnerable groups more effectively; and further tailoring and adapting the Recommendation to the specific policy context and institutional frameworks of each country.

Annex VI. Intervention Logic

The Figure provides information on the intervention logic of the Recommendation (source is the supporting study). The intervention logic helped to identify the supporting evidence needed throughout the evaluation process. See also section 2.1 of the supporting study.

Annex VII. Starting Points and Relevant Trends in Indicators and Policy Measures by Member State: Overview Tables

The Tables in this Annex summarize the analysis from the external supporting study that supports Sections 2.2 and 3: Table VII.1 provides an overview of the indicators and existing policy measures by Member State in 2016 that underpin the “need for change” categorization described in Section 3.1. The categorizations in each of the five columns of this Table are aggregated into an overall classification as described in Section 2.2.2 of the supporting study.

Table VII.2 summarizes the supporting study’s assessment of the evolution between 2016 and 2021: In the three right columns, the colour of a cell indicates the level of implementation of the step at the end of the evaluation period in 2021, and the symbol indicates the change in this level over the evaluation period. A ‘+’ sign indicates that the research team found a substantial positive changes in relation to the level of implementation of the step since 2016. Note that the absence of a ‘+’ sign in a Member States does not mean that no relevant measures have been taken, but that the research team of the external supporting study did not consider them as substantial relative to what was already in place in 2016. See the supporting study further details (including an overview of the underlying policy measures in Annex 3).

Table VII.1: Starting point in 2016 by Member State, grouped by need for change estimated by the supporting study

	Country	Low-qualified 25–64-year-olds – 2016 (%)	Participation of low-qualified in education and training 25–64-year-olds - 2016 (%)	Skills assessment	Provision of tailored & flexible learning	Validation and recognition
Lower need	Austria	15.5 (MN)	5.1 (LN)	LN	LN	LN
	Denmark	19.7 (MN)	19.7 (LN)	LN	MN	LN
	Estonia	11.4 (LN)	5.0 (LN)	LN	MN	MN
	Finland	11.9 (LN)	12.9 (LN)	LN	MN	MN
	France	21.9 (MN)	7.5 (LN)	LN	LN	LN
	Netherlands	22.9 (MN)	9.1 (LN)	LN	LN	LN
	Sweden	15.0 (MN)	19.3 (LN)	LN	LN	MN
Medium need	Germany	13.5 (MN)	3.7 (MN)	MN	LN	HN
	Latvia	9.3 (LN)	3.0 (MN)	HN	MN	HN
	Lithuania	5.4 (LN)	2.4 (MN)	MN	MN	MN
	Luxembourg	21.6 (MN)	6.9 (LN)	LN	LN	HN
	Portugal	53.1 (HN)	4.1 (MN)	LN	LN	LN
	Slovenia	12.7 (MN)	2.4 (MN)	LN	MN	LN
High need	BelgiumFR	24.9 (HN)	2.8 (MN)	MN	HN	MN
	BelgiumNL			MN	HN	MN
	Bulgaria	17.7 (MN)	0.5 (HN) 166	HN	HN	HN
	Croatia	17.3 (MN)	0.4 (HN)	HN	MN	HN
	Cyprus	20.5 (MN)	1.5 (HN)	MN	MN	MN
	Czechia	6.6 (LN)	2.3 (MN)	HN	HN	HN
	Greece	28.2 (HN)	0.7 (HN)	MN	MN	MN
	Hungary	16.6 (MN)	2.7 (MN)	MN	MN	HN
	Ireland	18.6 (MN)	2.3 (MN)	LN	LN	HN
	Italy	39.9 (HN)	2.3 (MN)	LN	HN	HN
	Malta	50.6 (HN)	2.8 (MN)	HN	HN	HN
	Poland	8.7 (LN)	0.9 (HN)	HN	MN	HN
	Romania	23.3 (HN)	0.3 (HN) 167	MN	MN	MN
	Slovakia	8.1 (LN)	n/a	MN	MN	HN
	Spain	41.7 (HN)	3.4 (MN)	LN	HN	LN

LN = Lower need for change

MN = Medium need for change

HN = High need for change

Source: Supporting study. For an explanation of the methodology used to come to these categorisations, please see Section 2.2.2 and Annex 1 of the supporting study.

Table VII.2: Estimated change between 2016-2021, grouped by need for change in 2016 as estimated by the supporting study

Overall need for change	Country	% change - Low-qualified 25–64-year-olds, 2016 to 2021	% change - Participation of low-qualified in adult learning 2016 - 2021	Skills assessment	Provision of tailored & flexible learning	Validation and recognition
Lower need	Austria	-9.0	13.7	=	=	=
	Denmark	-9.1	-24.4	=	+	=
	Estonia	-7.9	72.0	=	+	=
	Finland	-6.7	57.4	=	=	=
	France	-18.7	-44.0	=	=	=
	Netherlands	-15.3	51.6	=	=	=
	Sweden	-16.7	37.3	=	=	=
Medium need	Germany	12.6	-10.8	=	=	+
	Latvia	-16.1	13.3 168	+	=	+
	Lithuania	-5.6	12.5	=	=	=
	Luxembourg	-8.8	-1.4	=	=	=
	Portugal	-23.7	0.0	=	=	=
	Slovenia	-31.5	41.7	=	=	=
High need	BelgiumFR	-25.7	42.9	+	+	+
	BelgiumNL			+	+	+
	Bulgaria	-6.2	20.0 169	+	+	+
	Croatia	-26.0	-25.0	+	=	=
	Cyprus	-24.9	-6.7	=	=	+
	Czechia	-15.2	-26.1	+	=	=
	Greece	-28.4	-42.9	=	=	+
	Hungary	-17.5	0.0	=	=	=
	Ireland	-32.8	73.9	=	=	=
	Italy	-6.5	4.3	=	=	+
	Malta	-24.1	42.9	+	+	=
	Poland	-21.8	22.2	+	+	=
	Romania	-18.5	266.7 170	=	=	=
	Slovakia	-17.3	n/a	=	=	+
	Spain	-13.4	52.9	=	=	=

Source: supporting study. An ‘=’ sign indicates that the external supporting study did not identify substantial changes to the level of implementation of the step relative to the situation in 2016; ‘+’ sign indicates that positive changes in relation to the level of implementation of the step have been identified since 2016. See Section 2.2.2 and Annex 3 of the supporting study for further details, including a more disaggregated assessment of the positive changes.

(1)    Council Recommendation of 19 December 2016 on Upskilling Pathways: New Opportunities for Adults , OJ C 484/1 of 24.12.2016.

(2)       European Pillar of Social Rights, jointly signed by the European Parliament, the Council and the Commission on 17 November 2017, at the Social Summit for Fair Jobs and Growth in Gothenburg, Sweden. Principle 1, Education, training and life-long learning: “Everyone has the right to quality and inclusive education, training and life-long learning in order to maintain and acquire skills that enable them to participate fully in society and manage successfully transitions in the labour market”. Principle 4, Active support to employment, states that everybody has “the right to timely and tailor-made assistance to improve employment or self-employment prospects”.

(3)       European Skills Agenda for sustainable competitiveness, social fairness and resilience .

(4)       European Council Conclusions , 24-25 June 2021, following up on the Porto Declaration  of 8 May 2021. See also the European Pillar of Social Rights Action Plan.

(5)      See Report from the Commission to the Council COM(2023) 439 final.

(6)   Ecorys and 3S, Study supporting the evaluation of the Council Recommendation of 19 December 2016 on Upskilling Pathways: New Opportunities for Adults .

(7) For most but not all Member States at national level, cf. Annex 1 of the supporting study.

(8) The mapping of policy measures done by the external supporting study considers measures taken until November 2021. See Annex 3 of the supporting study .

(9) E.g. collaborative economy models, contract based work.

(10)  In 2016, data were available for Rounds 1 and 2 of Cycle I, cf. Survey of Adult Skills (PIAAC) - PIAAC, the OECD's programme of assessment and analysis of adult skills .

(11) Since the benefits of basic skills investment are wide-spread, there is a risk of underinvestment: see for instance Leuven (2005), The economics of private sector training: A survey of the literature , Journal of Economic Surveys.

(12) Cf. Burdett and Smith (2002), The low skill trap , European Economic Review.

(13)  Commission Communication of 10 June 2016 on A new skills agenda for Europe - Working together to strengthen human capital, employability and competitiveness , COM/2016/0381 final .

(14)  Cf. in particular the Council Recommendation of 28 June 2011 on policies to reduce early school leaving and the Council Recommendation of 22 April 2013 on establishing a Youth Guarantee , then replaced by the Council Recommendation of 30 October 2020 on A Bridge to Jobs – Reinforcing the Youth Guarantee .

(15)      For information on the EQF levels, cf. the EQF web page .

(16)      Council Recommendation of 20 December 2012 on the validation of non-formal and informal learning .

(17) Council Recommendation of 15 February 2016 on the integration of the long-term unemployed into the labour market .

(18) UN Resolution adopted by the General Assembly on 25 September 2015 on Transforming our world: the 2030 Agenda for Sustainable Development .

(19) The situation across Member States is varied, and the group of low-skilled adults may be comprised of different subgroups (such as but not limited to employed, unemployed, economically inactive, migrants, disadvantaged groups, persons in rural areas) depending on national circumstances.

(20) The rate was 9.4% for men and 11.1% for women. See Eurostat: online data code: TRNG_LFSE_01 .

(21)  Adults with low level of educational attainment (i.e. ISCED levels 0-2).

(22)  The rate was 8.1% for men and 10.8% for women. See Eurostat: online data code TRNG_LFSE_02 .

(23) See Eurostat, online data code: ISOC_SK_DSKL_I . Individuals whose digital skills could not be assessed because they did not use the internet in the past 3 months are assumed to lack basic digital skills.

(24) See Eurydice (2021), Adult education and training in Europe Building inclusive pathways to skills and qualifications. These results will become a relevant point of comparison in 2024, when the results from the second cycle of PIAAC will be released.

(25) See Eurostat, online data code: EDAT_LFSE_03 , and Figure 2 in Section 2.2.1 of the supporting study for an illustration of how the share of low qualified adults in the EU Member States varies across age groups.

(26) See Eurostat, online data code: LFSA_URGAE .

(27) See Eurostat, online data code: LFSA_ERGAED .

(28) See Section 2.2.2 of the supporting study for the summary info on policy measures in place in 2016, and Annex 3 of that study for the mapping of policy measures by Member State underpinning this analysis.

(29) The mapping by the supporting study does not include all validation and recognition measures existing in a Member States, but only those that the Member States recognised as implementing the Recommendation (see Annex 3 of the supporting study for more details on this mapping). Hence, the scope of the mapping is different than the scope of the Cedefop validation inventory and the 2020 evaluation of the Council Recommendation on validation ( SWD(2020) 121 final ).

(30) See Annex VII for an overview table, and the supporting study for more details.

(31) For the degree of success of the Recommendation see section 5.1 Conclusions.

(32) See Section 3.2 of the supporting study for the summary info on policy measures implemented over the evaluation period, and Annex 3 of that study for the mapping of policy measures by Member State underpinning this analysis.

(33) Cf. the overview in Section 4.1, Table 8 of the supporting study.

(34)   SWD(2019) 89 final .

(35) At the time of the 2019 stocktaking report, data for EU28 (including UK) were used.

(36) Since results from PIAAC cycle 2 will become available only in 2024, cf. Section 2.2. The Eurostat´s basic digital skills indicator has been revised for the data collected in 2021 and published in 2022, and uses individuals’ digital activities in five (instead of previously four) domains as a proxy for their digital skills, cf. the Digital Economy and Society Index (DESI) 2022.

(37) See Eurostat, online data code: TRNG_LFS_02 .

(38) See Eurostat, online data code: TRNG_LFS_02 .

(39) This refers to participation in the last four weeks, as measured every year through the Labour Force Survey. For its target that at least 60% of EU adults should participate in adult learning every year by 2030, the European Pillar of Social Rights Action Plan refers to a 12 month as opposed to a 4 week reference period. This will allow a more comprehensive measurement of adult learning participation. However, the first data using this indicator after 2016 will only be published in 2023.

(40) See the Education and Training Monitor 2022 for more details on participation of adults in learning, and its revised measurement in the LFS from 2021 onwards.

(41) See Eurostat, online data code: TRNG_LFSE_02 .

(42)  See ISOC_SK_DSKL_I for data up to 2019, and ISOC_SK_DSKL_I21 for the 2021 data. The digital skills indicator is a composite indicator based on selected activities performed by individuals on the internet in the specific areas: until 2019, these included information, communication, problem solving and software and from 2021 onwards an additional area, safety, was added. More information about the methodology is available in the Digital Skills Indicator Metadata. For a more comprehensive assessment of EU progress on digital skills, see the Digital Economy and Society Index (DESI) 2022.

(43) See Eurostat, online data code:e EDAT_LFSE_03 .

(44) See Eurostat, online data code: EDAT_LFS_9901 .

(45) See Eurostat, online data code: EDAT_LFS_9903 and EDAT_LFSE_03 .

(46) This indicator is arguably more comprehensive, yet less sensitive to policy change over a short time period, as the reference population is larger (all adults with upper secondary education as the highest EQF level).

(47) Source: Eurostat special extract from the Labour Force Survey. See the Eurydice reports from 2015 and 2021 for a more extensive analysis of these data for the years 2013 and 2019. A low number of this indicator appears most worrisome in conjunction with a high share of low-qualified adults aged 25 and above: this amounts to a high share of adults who have “missed the train” while they were in initial education age combined with limited opportunities to still “get on the train” as an adult.

(48)

 See Eurostat, online data code: LFSA_URGAE and LFSA_ERGAED .

(49) Reflecting mostly transitions to employment as opposed exits from the labour force, as the activity rate among low-qualified adults increased from 71.1 to 73.3% over this period- cf. Eurostat, online data code: LFSA_ARGAEDCOB .

(50) For instance, according to the position paper by the European Association for the Education of Adults (EAEA),“Upskilling Pathways has prompted national strategies to focus on basic skills and validation of skills acquired through non-formal and informal learning, moving the attention of learning programmes on learners with low(er) basic skills”.

(51)  AT, ES, FI, FR, HU, IE, LT, LU, NL, PT, RO and SE, cf. Section 3.1 and Annex VII.

(52)  ES, HU, IE, LT, LU, PT, RO and SI, cf. Annex VII.

(53)  Member States such as AT, FR, NL rather continued on the good course already set by 2016. 

(54)   SWD(2019) 89 final .

(55) Skills assessment offers now exist in all Member States at least to some extent according to the supporting study, cf. Section 3.1 and Section 4.1 of the supporting study.

(56)  A total of 108 responses from national respondents from all Member States except HU were collected as part of this survey, conducted as part of the external supporting study. They include respondents working for ministries or other government bodies responsible for adult learning/training, vocational education and training, lifelong learning or skills policies, or for education and training providers. See Annex 1 of the supporting study for details. 

(57) This survey was conducted as part of the external supporting study and collected 51 responses from organisations representing various target groups of the Recommendation. See Annex 1 of the supporting study for details.

(58) In its contribution to the public consultation, the European Training Foundation (ETF) highlighted the importance of sufficient capacity notably of Public Employment Services (PES), which should be guided to enhance the assessment tools for an effective skills audit for low skilled adults.

(59) Voiced e.g. by participants from EAEA and PL during the validation workshop on the findings of the supporting study.

(60) BE, BG, CY, DK, EE, MT, PL and SI as per the supporting study, cf. Section 3.1. For further analysis of the barriers to access and take-up, also see the impact assessment accompanying the Commission proposal on individual learning accounts, SWD(2021) 369 final.

(61) 60 respondents participated in the public consultation between 16 December 2021 and 17 March 2022. See Annex V for a summary, and the “Have your say” page of this consultation for further information. Position papers received during the public consultation are also quoted throughout this report.

(62) For instance, the public consultation position paper by Eurodiaconia highlights the importance of “positive flexibility” in terms of both delivery modalities and pace of learning. The position paper by “Literacy 100” also highlights the importance of such flexibility for the target group of the Recommendation: “A standard curriculum governed by tests does not have the flexibility to respond to such a variety of individual needs. In this system, there are learners who will simply be unable to keep up in lessons and will therefore fall by the wayside.”

(63) NL, IE, DE, EL, MT, PT, LV, BE, ES, LV, HU, HR, AT, IE, PL, BG, LU, SK, SI, IT.

(64) For instance, EAEA and the World Employment Confederation Europe (WEC - Europe) in their public consultation position papers.

(65)   SWD(2019) 89 final , p. 29.

(66)  BE, BG, CY, DE, EL, IT, LV and SK as per the supporting study, cf. Section 3.1. Note that the mapping by the supporting study does not include all validation and recognition measures existing in a Member States, but only those that the Member States recognised as implementing the Recommendation (see Annex 3 of the supporting study for more details on this mapping).

(67)   SWD(2020) 121 final . This was reflected also during the targeted consultations, with e.g. an IE representative to the EQF advisory group stating that “it takes time to build the infrastructure. Looking forward, there should be a way to ensure the person-centred part of the Recommendation becomes stronger”.

(68) Cf. supporting study Section 4.1, Q1.2(a).

(69) E.g. the EAEA position paper argues that “Upskilling Pathways do not always seem to reach the most vulnerable target groups, for instance women who are inactive in the labour market and older workers”.

(70) E.g. the position paper by “Literacy 100” points out that “Education in adulthood can be a daunting prospect for adults who have low levels of personal resilience and self-esteem. It is not uncommon for people who have been homeless to describe overwhelmingly negative childhood memories of bullying, inadequate teaching support, and poor understanding of their difficulties at both school and home.”

(71) Cf. supporting study Section 4.1, Q1.2(a), and Section 3.2.

(72) Cf. also recent analysis by the Commission’s Joint Research Centre on this topic: Centeno C., Karpinski, Z., Urzi Brancati, C., Supporting policies addressing the digital skills gap – Identifying priority groups in the context of employment, EUR 31045 EN, Publications Office of the European Union, Luxembourg, 2022, ISBN 978-92-76-51319-3, doi:10.2760/07196, JRC128561 The JRC analysis mentions e.g. the need to include NEETs, individuals with medium-level formal education, those living in rural areas, and individuals inactive in the labour market, as well as those employed in semi-skilled and low-skilled occupations.

(73) Cf. position paper by Eurodiaconia, which states that “our members stress, many times these services are seen as intimidating or untrustworthy by some people. To be accessible and inclusive, PES need to be provided in a safe space for everyone to feel protected from bias and any form of discrimination”.

(74)  Cf. Position paper by EAEA, noting that “guidance measures are not sufficiently available everywhere; however, they would be in a central position to address target groups with low skills”.

(75) This survey was conducted as part of the external supporting study, and collected responses from 102 adult learners. See Annex 1 of the supporting study for details.

(76) Including to support service provision also in rural areas, cf. position paper by ETF.

(77) Cf. position papers by “Literacy 100” and Eurodiaconia.

(78) Cf. the European Basic Skills Network (EBSN) position paper, and see supporting study Section 4.1, Q1.5 for examples. The EAEA position paper concludes that “Public investment in infrastructure and the training of adult learning and education staff is at the forefront.”

(79) The position paper by “Public libraries 2030” points out that “public libraries play an important role in the non-formal education sector as they are often the first organisations providing assistance and support to vulnerable groups with low literacy skills”.

(80) The WEC-Europe position paper mentions successful public-private partnerships in FR, BE and NL.

(81) See Section 4.1, Q1.4(b), Figure 12 of the supporting study.

(82) E.g. the EAEA position paper argues that „those countries that succeed in involving other stakeholders, e.g. labour market services, but also social partners, are more successful in upskilling. EAEA members argue that Upskilling Pathways requires a holistic approach that needs not only the education sector but also the active engagement of employers, social services, labour market services, etc. in order to reach key target groups and make progress in the implementation of the strategy“.

(83)  Supporting study Section 4.1, Q1.4(b).

(84) EAEA position paper.

(85) Cf. e.g. the ILO position paper, emphasising the importance of “a whole-of-government approach to effective governance”.

(86) The EAEA position paper highlights the importance of embedding the measures into a broader strategic framework: “What emerges is that those countries that have integrated Upskilling Pathways into a broader lifelong learning strategy are most likely to reach the target groups and build innovative upskilling programmes, for instance Slovakia, Finland and Austria.”

(87)  See e.g. position papers by Eurodiaconia and “Literacy 100”.

(88) Also see Annex 9 of the supporting study, which does not find trend breaks for a larger number of indicators relevant to the Recommendation (including participation in adult learning, unemployment or inactivity rates). Except for basic digital skills, no “before-after” comparisons of direct measures of adult’s basic skills are available for this evaluation, since the results from the second cycle of PIAAC will only become available in 2024 (cf. Section 2.2).

(89) See the “interrupted time series analysis” of the supporting study, presented in Section 4.1 Q1.1 and in Annex 9.

(90) Education and training providers may be public or private, and may receive funding to cover staff, infrastructure or awareness raising costs, allowing them to offer their services to adults for free or at a reduced price. Incentives to employers include e.g. reimbursements for training costs or for salary costs while an employee participates in training. Incentives to individuals can take the form of scholarships/study grants to cover the costs of living while studying, or support for travel or other expenses related to participation in upskilling measures. See supporting study Section 4.2 Q2.1.

(91) For instance in BG, HU, LT and PL. See Section 4.2 of the supporting study for additional details, and examples for EU support from the different programmes.

(92) Including AT, BE, BG, CY, DK, EE, ES, FI, FR, GR, HU, IE, IT, LT, LU, LV, MT, PL, PT, RO, SE, SI, SK.

(93)   SWD(2019) 89 final .

(94)   SWD(2021) 10 final.

(95)   Cf. Section 4.2 of the supporting study.

(96)  Including BE, CY, EL, ES, FR, GR, IE, LU- cf. supporting study Section 4.2 Q2.2 and EAEA position paper.

(97)  Cedefop (2017), Investing in skills pays off: the economic and social cost of low-skilled adults in the EU . For a literature review, also see Annex 10 of the impact assessment accompanying the proposal for a Council Recommendation on individual learning accounts, SWD(2021) 369 final .

(98) Supporting study Section 4.2 Q2.4.

(99) Cf. supporting study Section 4.2 Q2.4.

(100)  Cf. supporting study Annex 4.

(101) Cf. the ETF position paper, concluding that “Stronger monitoring and evaluation procedures and wider in-country and cross-country mutual learning would help a more structural orientation of education and training polices and actions to serve the most deprived groups”.

(102) Cf. Section 4.4 of the supporting study.

(103) Cf. supporting study Section 5.2. This point was also made by participants from EAEA and PL during the validation workshop on the findings of the supporting study.

(104) For instance in DE, cf. supporting study Section 4.4 Q4.1.

(105) Supporting study Section 4.4 Q4.2(a).

(106)   Council Recommendation of 22 May 2018 on key competences for lifelong learning.

(107)   Council Recommendation of 22 April 2013 on establishing a Youth Guarantee ; Council Recommendation of 30 October 2020 on A Bridge to Jobs – Reinforcing the Youth Guarantee ; Council Recommendation of 15 February 2016 on the integration of the long-term unemployed into the labour market .

(108)  Council Recommendation of 20 December 2012 on the validation of non-formal and informal learning . Council Recommendations of 16 June 2022 on individual learning accounts and micro-credentials for lifelong learning and employability .

(109) Supporting study Section 4.4 Q4.2(a).

(110)   European Pillar of Social Rights.

(111)   European Council Conclusions , 24-25 June 2021. This followed up on the Porto Declaration  of 8 May 2021. See also the European Pillar of Social Rights Action Plan.

(112)   Council Conclusions of 8 June 2020 on reskilling and upskilling as a basis for increasing sustainability and employability , in the context of supporting economic recovery and social cohesion.

(113) Supporting study Section 4.4 Q4.2(b).

(114) For the 2021-2027 programming period, the ESF+ has a budget of €99.3 billion and will continue to support the implementation of the EU’s skills policies including by supporting adults with low skills and basic skills acquisition.

(115) Supporting study, Section 4.5 Q5.1.

(116) Supporting study, Section 4.1 Q1.3.

(117) Supporting study, Section 4.5 Q5.1.

(118) Supporting study Section 4.2 Q2.2.

(119)   National Skills Strategies - OECD .

(120)  See for instance the 2020 reports on “Empowering adults through upskilling and reskilling pathways”, volume 1 and volume 2 .

(121)   Several reports were published by the Joint Research Centre to support the use of the competence frameworks.

(122)  Country-specific recommendations e.g. for 2019 and 2020 .

(123)   Council Conclusions of 8 June 2020  on reskilling and upskilling as a basis for increasing sustainability and employability, in the context of supporting economic recovery and social cohesion.

(124) 2020 European Skills Agenda.

(125)   SWD(2019) 89 final, p. 19 :  “With very few exceptions, all countries have indicated that much of the activity to support low-skilled adults is co-funded through the European Social Fund.”

(126) The 2019 stocktaking report of implementation measures concludes that “the majority of implementation measures seem to be primarily driven by access to EU funds and tied in with their planning periods. Hence, many initiatives are running until 2020 or at best to 2022, and there are no obvious plans on how to sustain them beyond this date” ( SWD(2019) 89 final, p. 20).

(127)   Recovery and Resilience Scoreboard.

(128) Supporting study, Section 4.5 Q5.1. In the targeted consultation with the EQF advisory group, a representative from SK indicated that “Basic skills were not reflected in national policies in SK and were not discussed as an issue. The Recommendation helped put this topic on the table. (…) A strategy on lifelong learning (LLL) was approved in November 2021 and basic skills development is included in a set of concrete measures”.

(129) Supporting study, Section 4.5 Q5.2.

(130) This view was also frequently expressed by stakeholders in the consultations. For instance, the WEC-Europe position paper concluded that “careers today are no longer linear but take numerous twists and turns, which makes upskilling pathways even more important”.

(131) See for instance “ Adult learning and COVID-19: challenges and opportunities ” from the ET 2020 working group on adult learning, and the position paper submitted by Wallonia.

(132) Supporting study, Section 4.3 Q3.1. Cf. also the position paper by WEC-Europe, concluding that “EU guidance is required to a fairly large extent to promote the exchange of good practices and promote mutual learning and bench-learning”.

(133) The position paper of Wallonia mentions “soft skills, transversal skills (communication, problem solving, teamwork, emotional intelligence) and entrepreneurial skills based on creativity, critical thinking and problem solving, but also on taking initiatives, perseverance and collaboration.”

(134) Position paper by the European Basic Skills Network: “To build individual and social resilience it is imperative that all adults have access to quality provision of a dynamic set of skills. Basic Skills continue to be the core focus, but they need to be acquired in a context that acknowledges and develops Life Skills.” The ILO position paper calls for “renewed emphasis on the development of core skills, including social and emotional skills, cognitive skills, basic digital skills and relevant skills related to environmentally sustainable economies, as well as science, technology, engineering and mathematics (STEM) skills, in education and training, including TVET”. Also the Eurodiaconia position paper highlights the importance of transversal skills to “continue learning, adapt to change and deal with uncertainty during labour market transitions”.

(135) Supporting study, Section 4.3 Q3.1.

(136)

Positive examples concerning the integration of the three steps identified by the supporting study include PT, DK, SE (already at the start of the evaluation period) and –at a project level- PL, cf. Section 4.1 Q1.3.

(137) In the targeted consultations with ACVT and DGVT, for instance DE highlighted that “many qualified adults are in need of updating their skills and qualifications to match the new skills needed on the labour market. While low-qualified adults remain a priority target group, it is also important to take into account this additional target group in future implementation of the Recommendation”.

(138)  In the targeted consultations, the European Trade Union Confederation (ETUC) stated that it considers the Recommendation as a crucial instrument, but that a challenge is that “the Recommendation did not include provisions for universal support and encouraged Member States to define the target group. In some instances, this has led to certain groups being left out”.

(139) Supporting study, Section 4.3 Q3.4.

(140) See Cedefop (2021), More perceptions: opinion survey on adult learning and continuing vocational education and training in Europe- Volume 2: Views of adults in Europe .

(141) Supporting study, Section 4.5 Q5.2.

(142)

  Decision 2023/936 of the European Parliament and of the Council on a European Year of Skills .

(143)   State of the Union address 2022.

(144) Communication from the Commission to the European Parliament, the Council, the European Economic and Social Committee and the Committee of Regions, A Green Deal Industrial Plan for the Net-Zero Age, COM(2023) 62 final

(145) This also includes some Member States which continued on the good course that had already been set by 2016. 

(146)  Understood here in terms of the share of adults in need of support with basic skills development who receive support. Effective outreach may be insufficient either because of low formal coverage of/eligibility for support measures, or because of low awareness and take-up among target groups.

(147) SWD(2019) 89 final .

(148) COM(2023) 439.

(149)   https://eur-lex.europa.eu/resource.html?uri=cellar:cd0fa1ca-2ee9-11e6-b497-01aa75ed71a1.0001.02/DOC_4&format=PDF  

(150)   https://eur-lex.europa.eu/legal-content/EN/TXT/PDF/?uri=CELEX:52016DC0382&rid=9

(151)   SWD(2019) 89 final .

(152)   https://ec.europa.eu/social/main.jsp?catId=738&langId=en&pubId=8516&furtherPubs=yes

(153)   Better regulation: guidelines and toolbox.

(154)   SWD(2019) 89 final .

(155)  Supporting study, 4.1 Effectiveness.

(156)

  https://op.europa.eu/en/publication-detail/-/publication/827fcd9c-1a8c-11ec-b4fe-01aa75ed71a1/language-en/format-PDF/source-232128473  

(157)

  https://ec.europa.eu/info/sites/default/files/file_import/implementation-report-upskilling-pathways_en.pdf .

(158)  Supporting study, 4.1 Eficiency.

(159)  Supporting study, 4.3 Relevance.

(160) Supporting study 4.4 Coherence.

(161)  Supporting study, 4.5 EU added value.

(162)  European Commission, Improving adults' basic skills (“Upskilling pathways recommendation”) – evaluation: https://ec.europa.eu/info/law/better-regulation/have-your-say/initiatives/12637-Improving-adults-basic-skills-Upskilling-pathways-recommendation%E2%80%9D-evaluation_en  

(163) See also the supporting study e.g. annexes 1, 5, 6, 7 and 8.

(164)  Austria, Bulgaria, Estonia, France, Germany, Hungary, Ireland, Netherlands, Spain, Sweden.

(165)  European Commission, Improving adults' basic skills (“Upskilling pathways recommendation”) – evaluation: https://ec.europa.eu/info/law/better-regulation/have-your-say/initiatives/12637-Improving-adults-basic-skills-Upskilling-pathways-recommendation%E2%80%9D-evaluation/feedback_en?p_id=18724489  

(166)

Data from 2017 was used to replace missing 2016 data.

(167)

Data from 2015 was used to replace missing 2016 data.

(168)

Data for 2019 was used to replace missing 2021 data.

(169)

Data for 2017 was used to replace missing 2016 data.

(170)

Data for 2015 was used to replace missing 2016 data.