A NEW METHODOLOGY FOR MEASURING THIRD MISSION ACTIVITIES OF UNIVERSITIES

Abstract

European universities have traditionally recognised two main missions as their duty: teaching and academic research. Recently, a Third Mission, frequently underdeveloped by universities, has emerged to include in their missions those activities that facilitate their engagement with society and industry. While the first two missions of the universities have been widely studied and frequently measured, Third Mission is yet to be well examined and approaches to its measurement developed. The objective of the project European Indicators and Ranking Methodology for University Third Mission (E3M) coordinated by Universidad Politécnica de Valencia and involving eight partner universities from seven different European countries, is to create a ranking methodology for measuring Third Mission activities of universities. This three years project (2009-2012) is funded by the European Commission under the Lifelong Learning Programme. The measurement of the three missions of the universities has been generally based on indicators. In line with this, the methodology used by the E3M project to evaluate the contributions of universities to society consists, among other activities, of the identification, definition and selection of the best set of indicators that, in a possible future ranking, would be the basis to evaluate the excellence of institutions in this field.

Full text

A NEW METHODOLOGY FOR MEASURING THIRD MISSION
ACTIVITIES OF UNIVERSITIES
A. Carrión1, V.R. García-Gutiérrez1, M.C. Bas1, J.M. Carot1
1Universidad Politécnica de Valencia (SPAIN)
acarrion@cq.upv.es, vegargu@upvnet.upv.es, maibacer@doctor.upv.es, jcarot@cq.upv.es
Abstract
European universities have traditionally recognised two main missions as their duty: teaching and
academic research. Recently, a Third Mission, frequently underdeveloped by universities, has
emerged to include in their missions those activities that facilitate their engagement with society and
industry. While the first two missions of the universities have been widely studied and frequently
measured, Third Mission is yet to be well examined and approaches to its measurement developed.
The objective of the project European Indicators and Ranking Methodology for University Third
Mission (E3M) coordinated by Universidad Politécnica de Valencia and involving eight partner
universities from seven different European countries, is to create a ranking methodology for measuring
Third Mission activities of universities. This three years project (2009-2012) is funded by the European
Commission under the Lifelong Learning Programme.
The measurement of the three missions of the universities has been generally based on indicators. In
line with this, the methodology used by the E3M project to evaluate the contributions of universities to
society consists, among other activities, of the identification, definition and selection of the best set of
indicators that, in a possible future ranking, would be the basis to evaluate the excellence of
institutions in this field.
Keywords: Universities, Third Mission, indicators, ranking methodology
1 INTRODUCTION
According to Martin & Etzkowitz, 2000 [1] the Third Mission of universities is not a new phenomenon.
During the twentieth century the contribution to the economy and society co-existed within universities
with teaching and academic research. It is also considered that since their inception, universities have
always made contributions, directly and indirectly, to the society in general, not only in academic
fields. But it is nowadays when Third Mission contributions are seen as essential and they deserve of
specific policies and resources to ensure their effective functioning (Molas-Gallart et al, 2002) [2].
The notion of Third Mission is ambiguous, and can be taken very differently depending upon the
configuration of university activities, upon its embedding in its geographical territory and upon the
country’s institutional framework (Laredo, 2007) [3]. There are two perspectives or approaches to
define the University Third Mission. The first one is the “Triple Helix” model of university-industry-
government relations (Leydesdorff & Etzkowitz, 1996) [4]. This approach is in fact, an actualization of
the Sabato Triangle model developed by Sabato & Botana, 1968 [5] which is well known in Latin
America but less known and recognized in Europe and USA. The second approach is described by
Molas-Gallart et al., 2002 [2] who define this Third Mission as: “all activities concerned with the
generation, use, application and exploitation of knowledge and other university capabilities outside
academic environments”.
From this point of view, the E3M project aims to define the best way for measuring the effort and
results concerning Third Mission activities in universities by offering a relevant set of indicators and a
cohesive methodology to assess this Third Mission.
2 METHODOLOGY
As mentioned before, not all third mission activities need to be exactly the same in all universities, so
there is always some diversity. But the spirit of the mission is the same as for any other part of the
academy. Basing on this idea, the E3M project started by structuring all third mission activities into
three groups or dimensions, which were considered by the project research team to be the most

representative of the Third Mission as a whole. In this way, the three dimensions in which the project
characterised third mission activities are (Fig.1): Continuing Education (CE), Technology Transfer &
Innovation (TTI) and Social Engagement (SE).
Continuing
Educat ion
Technology
Transfer &
Innovat ion
Social
Engagement
Third
Mission
Fig.1 Three dimensions under which the project examines the Third Mission
The term CE is referred to “all learning activities undertaking throughout life, with the aim of improving
knowledge, skills and competences with a personal, civic, social and/or employment related
perspective” (European Commission, 2001) [6].
The concept TTI is related to the movement of an idea, tacit knowledge, know-how, technical
knowledge, intellectual property, discovery or invention resulting from research carried out at
universities (in cooperation with external partners or not) into a non-academic environment where it
can lead to social and commercial benefits at local, regional, national or global levels.
The term SE is connected to the role of the universities to engage with its civic, cultural, industrial and
business communities and the main activities that the university organises aimed to society at large or
to specific sectors of the society to enrich them on the cultural or developmental field.
Once these dimensions were developed and described, different processes or activities were
identified, as conforming the way in which the three dimensions are implemented. Their identification
was made through a process map (Fig. 2) which set the basis for the analysis and development of the
future set of indicators. Every single process was connected to a collection of different activities with
inputs and outputs and with specific objectives as well.
CE TTI SE
P1
P2
P3
P1
P2
P3
P1
P2
P3
.
.
.
.
.
.
.
.
.
i1 i2 i3 i4 i1 i2 i3 i4 i1 i2 i3 i4
Fig. 2 Tool used to get the first collection of indicators taking the three dimensions as a starting point

The processes do not tried to be in any sense a standard of what they should do, although the project
team considered that they followed a very reasonable structure. Probably different processes would
produce a different indicators set, but in any case they will not vary significantly from those that are
defined in Fig. 3 to 5.
CE2: Implementat ion of
Continuing Education
Activities
CE3: Informat ion
and Advertising CE5: Financial
Management
CE4: Application and
Admission
Management
CE6: Teaching and Learning
CE7: Quality
Evaluation
CE1: Analysis of the
Demand and Curriculum
Design
CE0: Institutional
Involvement in Continuing
Education
CE8: Final Assessment
and Follow-up
TTI0: Institutional Involvement in Technology Transfer & Innovation
Specific
Processes Generic
Processes
TTI1:
Licensing of
university
patents to
companies
TTI2:
Formation of
start-ups &
spin-offs
companies
TTI3: Non-
patent &
software
innovations in
public domain-
creative
commons &
social
innovation
TTI4:
Problem
solving
cooperation
in R&D
TTI5: Public
space –
sharing
space/facilities/
equipment/
services/
networking
TTI6: People
– mobility
and
education
Fig. 3 Continuing Education processes Fig. 4 Technology Transfer & Innovation processes
SE0: Institutional Involvement in Social
Engagement
SE1: Non-discipline volunteering
SE2: Expert advisory engagement
SE3: Services
and facilities to
community
SE4: Educational
outreach/collaboration
and widening
participation
Local National International
Individual
Institutional
Fig. 5 Social Engagement processes
Specific processes for the three dimensions were identified with one common process for all
dimensions (named as process 0) that embraced the Institutional Involvement.
After this common structure with the processes for the three dimensions was established, a set of
primary indicators was defined for the three dimensions. Thus, a wide collection of indicators was
developed as result of a literature review and the work done by the research group.
For producing a list of the most relevant indicators having as a basis the initial set, an expert’s
judgment following the Delphi technique was needed. The Delphi technique is an established method
for obtaining consensus. Specifically, in the E3M project implied that an expert panel compound by
twenty experts from very different areas in Europe and USA was consulted in three rounds with
different and specific purposes, but with the common objective to reach a consensus and get a
reduced and more refined list of indicators.

The Delphi procedure was carried out in the following seven stages (Fig. 6 and 7): 1. Implementation
of the first round questionnaire; 2. Analysis of first round responses; 3. Implementation of the second
round questionnaire. 4. Analysis of the second round responses; 5. Implementation of the third round
questionnaire; 6. Analysis of the third round responses; 7. Final report.
Analysis
Second
round TTI
Second
round SE
Second
round CE
Analysis
Third Round
CE+TTI+SE Final
Report
Expert
selection
First
round CE
First
round TTI
First round
SE
Fig. 6 Three rounds of the Delphi consultation process
2010
Deadlinefor
presentation
of expert
proposals
Invi tatio n em ai l to
expertsfor
participation Delphi final report
Invi tation to p articip ate in
the1st Delphi round sent
out with the
questionnaire
Final list
of experts
selected
Invitationto participatein
the2nd Delph i round sent
out with the
questionnaire
Invi tation to p articip ate in
the3rd Delphi round sent
out with the
questionnaire
May June July Aug Sept Oct Nov Dec Jan Feb March April
2011
Analysisof 1st
round
responses
Analysisof 2nd
round
responses
Analysisof 3rd
round
responses
Fig. 7 Planning and schedule of the consultation process
During this consultation procedure, experts were asked through an online survey to rate the indicators
in terms of relevance, validity, reliability, feasibility and comparability using a rating scale from 1 to 7
from the least to the most important. They also provided a feedback on the processes identified as
well as a global vision about the full set of indicators. All experts’ opinion about indicator’s definition
and characteristics were incorporated to produce a final set. As a result, a final and homogeneous
group of indicators was produced in the three areas of the Third Mission. As shown in Table 6, a total
of 54 indicators were obtained for CE, TTI and SE. They all intend to satisfy the need of incorporating
Third Mission indicators in the evaluation of the university activities.
Table 6 Summary of first results obtained
Dimensions Processes Indicators
CE 9 18
TTI 7 20
SE 5 16
Ʃ 21 54
3 RESULTS
Below is presented the final list of indicators identified for each dimension and its processes.
- Final set of Continuing Education indicators:
CE0-i1: CE is included in the mission of the HEI
CE0-i2: CE is included in the policy and/or the strategy of the HEI
CE0-i3: Existence of an institutional plan for CE in the HEI

CE0-i4: Existence of quality assurance procedure for CE activities
CE1-i1: Total number of CE programmes active in that year for implementation
CE1-i2: Number of CE programmes delivered which have a major award under higher education
system
CE1-i3: Number of partnership with public and private business CE programmes delivered in that year
CE1-i4: Percentage of international CE programmes delivered in that year
CE1-i5: Percentage of funded CE training projects delivered in that year
CE1-i6: Total number of the ECTS credits of the delivered CE programmes
CE4-i1: Number of ECTS credits enrolled
CE4-i2: Number of registrations in CE programmes in that year
CE4-i4: Percentage of CE ECTS enrolled referred to the total ECTS enrolled
CE6-i1: Percentage of qualifications issued referred to total CE registrations
CE7-i1: Students satisfaction
CE7-i2: Key stakeholder satisfaction
CE7-i3: Completion rate for all programmes (in average)
- Final set of Technology Transfer & Innovation indicators:
TTI0-i1: TTI is included in the mission of the HEI
TTI0-i2: TTI is included in the policy and/or strategy of the HEI
TTI0-i3: Existence of an institutional action plan for TTI in the HEI
TTI1-i1: Number of licenses, options and assignments (active and executed, exclusive and non-
exclusive) to start-ups or spin-offs and existing companies
TTI1-i2: Total budget coming from revenues from commercialisation of HEI knowledge
TTI2-i1: Number of start-ups and spin-offs established
TTI3-i1: Number of creative commons and social innovation projects that HEI employees are involved
in
TTI4-i2: Number of R&D sponsored agreements, contracts and collaborative projects with non-
academic partners
TTI4-i3: Percentage of HEI budget coming from income of R&D sponsored contracts and collaborative
projects with non-academic partners
TTI4-i4: Number of consultancy contracts
TTI4-i5: Percentage of postgraduate students and postdoctoral researchers directly funded or co-
funded by public and private businesses
TTI5-i1: Number of created (co-funded) or shared laboratories and buildings
TTI6-i2: Number of companies participating in continuous professional development courses (CPD)
TTI6-i3: Number of HEI employees with temporary positions outside of academia
TTI6-i4: Number of non-academic employees with temporary positions at HEIs
TTI6-i5: Number of postgraduate theses or projects with non-academic co-supervisors
TTI6-i7: Number of joint publications with non-academic authors
TTI6-i8: Number of academic staff participating in professional bodies, networks, organizations and
boards
TTI6-i9: Number of external organizations or individuals participating at advisory, steering, validation,
review boards to HEIs, institutes, centres or taught programmes
TTI6-i10: Number of prestigious innovation prizes awarded by business and public sector associations
or funding agencies (national and international)
- Final set of Social Engagement indicators:
SE0-i1: SE is included in the mission of the HEI
SE0-i2: SE is included in the policy and/or strategy of the HEI
SE0-i3: Existence of an institutional action plan for se in the HEI
SE0-i4: Budgetary assignment to se
SE2-i1: Percentage of academics involved in volunteering advisory
SE3-i1: Number of events open to community/public
SE3-i2: Number of research initiatives with direct impact on the community
SE3-i4: Number/cost of staff/student hours made available to deliver services and facilities to
community
SE3-i5: Number of people attending/using facilities
SE4-i1: Number of projects related to educational outreach
SE4-i2: Number of faculty staff and students involved in educational outreach activity
SE4-i4: Percentage of HEI budget used for educational outreach

SE4-i5: Number of community participants in educational outreach activity
SE4-i7: Number of activities specifically targeting disadvantaged students /community groups
SE4-i9: Number of community representative on HE boards or committees
SE4-i11: Amount of grants/donations/contracts arising from engaged partnerships
For each of the indicators, a template was completed including a detailed definition, purpose of the
indicator, time frame to data catching, type of information source and formula.
To complete the analysis of the set of indicators six case studies were implemented. The objective of
the case studies was to check the opinion of these six universities or HEIs about the indicators
selected through a confrontation with institutional representatives of the Third Mission activities in the
three fields, but also to detail best practices of each of the universities. The cases studies took place
at the following institutions in Europe: Universidad Politécnica de Valencia (Spain), Politecnico di
Torino (Italy), University of Cambridge (United Kingdom), University of Turku (Finland), Dublin Institute
of Technology (Ireland) and Széchenyi István University (Hungary). The case studies also provided a
large debate about the possible improvements of the three set of indicators proposed to the visited
university.
To conclude, a global web for European Third Mission providers is currently being built to provide
access and allow European institutions to enter data regarding their activities in the Third Mission.
This global web will be a useful tool to benchmark institutions and compare their own indicators and
services across Europe.
4 CONCLUSIONS
- The methodology used along the project has proved to be efficient enough to integrate knowledge
and experience and to define a homogeneous set of significant indicators that entirely cover the
definition of third mission.
- The indicators developed, are a good starting point towards developing a more comprehensive
framework for measuring and evaluating Third Mission activities and embedding them in the higher
education policy.
- The identification and collection of these best set of indicators can be the basis to assess excellence
of universities, and an excellent benchmarking tool for institutions.
- The project offers an innovative approach on the concept of the ranking methodologies to evaluate
the Third Mission activities of universities.
REFERENCES
[1] Etzkowitz, H., Webster, A., Gebhardt, C. & Terra, B. (2000). The Future of the University and
the University of the Future: evolution of ivory tower to entrepreneurial paradigm. Research
Policy 29: 313-330.
[2] Molas-Gallart, J., Salter, A., Patel, P., Scott, A. & Duran, X. (2002). Measuring Third Stream
Activities. Final Report to the Russell Group Universities. SPRU, University of Sussex.
[3] Laredo, P. (2007). Revisiting the Third Mission of Universities. Toward a Renewd
Categorization of University Activities? Higher Education Policy, 20, 441-56
[4] Leydesdorff, L. & Etzkowitz, H. 1996. ‘Emergence of a Triple Helix of University-Industry-
Government Relations’, Science and Public Policy 23: 279-86.
[5] Sabato, J. & Botana, N. (1968). La Ciencia y la Tecnología en el Desarrollo Futuro de América
Latina. Revista de la Integración, INTAL, Buenos Aires, año 1, nº3, pp. 15-36.
[6] European Commission (2001). Making a European area of lifelong learning a reality.
Communication from the Commission.



