Towards innovation measurement in the software industry

Abstract

In today's highly competitive business environments with shortened product and technology life cycle, it is critical for software industry to continuously innovate. This goal can be achieved by developing a better understanding and control of the activities and determinants of innovation. Innovation measurement initiatives assess innovation capability, output and performance to help develop such an understanding. This study explores various aspects relevant to innovation measurement ranging from definitions, measurement frameworks and metrics that have been proposed in literature and used in practice. A systematic literature review followed by an online questionnaire and interviews with practitioners and academics were employed to identify a comprehensive definition of innovation that can be used in software industry. The metrics for the evaluation of determinants, inputs, outputs and performance were also aggregated and categorised. Based on these findings, a conceptual model of the key measurable elements of innovation was constructed from the findings of the systematic review. The model was further refined after feedback from academia and industry through interviews.

Full text

Towards Innovation Measurement in the Software Industry
Henry Edison
a
, Nauman bin Ali
a
, Richard Torkar
a,b
a
Blekinge Institute of Technology, Karlskrona, Sweden
b
Chalmers University of Technology and Gothenburg University, Gothenburg, Sweden
Abstract
In today’s highly competitive business environments with shortened product and technology life cycle, it is critical
for software industry to continuously innovate. This goal can be achieved by developing a better understanding
and control of the activities and determinants of innovation. Innovation measurement initiatives assess innovation
capability, output and performance to help develop such an understanding. This study explores various aspects relevant
to innovation measurement ranging from definitions, measurement frameworks and metrics that have been proposed
in literature and used in practice. A systematic literature review followed by an online questionnaire and interviews
with practitioners and academics were employed to identify a comprehensive definition of innovation that can be
used in software industry. The metrics for the evaluation of determinants, inputs, outputs and performance were
also aggregated and categorised. Based on these findings, a conceptual model of the key measurable elements of
innovation was constructed from the findings of the systematic review. The model was further refined after feedback
from academia and industry through interviews.
Keywords: Innovation; measurement; systematic literature review; metrics; empirical study
1. Introduction
The modern business context is characterised by high competition driven by deregulation, empowered customers,
emerging technology, globalisation of the economy, uncertain economic circumstances and rapid product develop-
ment (Romijn and Albaladejo, 2002; Akman and Yilmaz, 2008; Gumusluo
ˇ
glu and Ilsev, 2009). With shortened
product and technology life-cycles, the software industry is particularly a↵ected because of its knowledge-intensive
and technology-driven nature (Romijn and Albaladejo, 2002). In this context, innovation is considered a key success
factor, central to increasing economic output and productivity (Akman and Yilmaz, 2008). Innovation is the ability
to dictate and modify the ‘rules of the game’ that enables organisations to gain entry to new markets and challenge
established market leaders (Brown and Eisenhardt, 1995).
In the past, management has focused on cost, lead time reduction and quality improvement for competitiveness in
the market (Rejeb et al., 2008). However, in today’s competitive business environment, quality is a necessity but it is
not sufficient (Romijn and Albaladejo, 2002; Rejeb et al., 2008). Organisations must continuously innovate, develop
new processes and deliver novel products (in this study, we use the term ‘products’ for both goods and services), to
achieve and sustain a competitive advantage (Muller et al., 2005; Rejeb et al., 2008).
Innovation helps organisations to gain large market share but if they fail to consistently innovate overtime they
tend to loose their position to emerging firms that have innovative o↵erings (Muller et al., 2005). Such turnover
signifies the importance of sustained innovation, thus the problem is not happen-stance innovation but rather doing it
continuously on a regular basis. For sustained innovation to become a reality, a better understanding of innovation is
required, which is possible only when innovation is measured (Kuczmarskl, 2001).
The importance of innovation measurement is well emphasised in industry. According to The Boston Consulting
Group’s survey (James et al., 2008), 74% of the executives believed that their company should track innovation as
rigorously as core business operations, but only 43% of the companies actually measured innovation. Although,
some companies think that innovation cannot and should not be measured, the real issue is a lack of metrics and
measurements (James et al., 2008). This makes companies measure too little, measure the wrong things or not measure
innovation at all (James et al., 2008).
Preprint submitted to Journal of Systems and Software November 17, 2012

There is little consensus on how innovation measurement should be carried out (Jensen and Webster, 2009).
This lack of consensus is caused by the di↵erent definitions of innovation used. Each of these definitions signify a
di↵erent aspect of innovation e.g. perspectives, levels and types (Mathiassen and Pourkomeylian, 2003). This in turn
determines what is considered as elements of innovation and how these are measured.
The perspective of innovation as adopted by an organisation would delineate the ideal measures of innovation
(Jensen and Webster, 2009). For some problems a universal yardstick is enough but innovation encompasses creation
of new opportunities, businesses, markets, environments, methods of working and operating (Shapiro, 2006). In
essence no single measure can cover all these constituting aspects of innovation (Shapiro, 2006). The survey by
The Boston Consulting Group revealed that only 35% of the executives were satisfied with their current innovation
measurement practices. Part of the reason for this dissatisfaction could be that most of the companies (i.e. 58%) use
less than five metrics, which is not enough for measuring the entire range of innovation activities (James et al., 2008).
Organisations require means not only to measure their innovative output but also to assess their ability and capacity
to innovate. Measurement helps to better understand and evaluate the consequences of the initiatives geared towards
innovation. Furthermore, like any other measurements, these will allow organisations to specify realistic targets of
innovation in the future and to identify and resolve problems hindering progress towards goals, making decisions and
continuously improve the abilities to innovate (Ebert and Dumke, 2007).
In this study, the area of innovation measurement was explored. The study attempts to contribute to the innovation
measurement body of knowledge for the software industry. First, by performing a systematic literature review we
establish the current state of art of innovation measurement. Second, the literature review is then complemented
with a questionnaire and interviews to elicit the perception of innovation and the state of practice in the software
industry. Finally, an innovation measurement model was developed for, amongst many things, identifying the aspects
of innovation that are measurable and thus serving as a starting point for innovation measurement.
The remainder of this paper is structured as follows: Section 2 discusses the existing research related to this study
and Section 3 presents the research methodology undertaken in this study. The results of this study are reported in
Section 4 and discussed in Section 5. Section 6 presents the proposed model, which covers various dimension of
innovation. Validity threats of the study are discussed in Section 7, while the conclusions and future work are covered
in Section 8.
2. Related Work
To the best of our knowledge, there is no systematic review on innovation measurement in the software industry.
We have, however, identified several studies in other domains that attempt to address similar questions and issues as
presented in this paper.
(Becheikh et al., 2006) conducted a systematic review on technological innovation in the manufacturing sector
from 1993–2003. The aim was to find what the main variables are and how they are used to measure the innovation
behaviour and capacity of the organisation. The study was based on the empirical evidence reported in journals pub-
lished in ABI /INFORM of Proquest, Business Source Premier (BSP) of EBSCO, and ScienceDirect of Elsevier. The
study, however, only considered two areas, as suggested by the Oslo Manual (OECD, 2005): Product and process in-
novation. Although the study identified 36 internal and 10 external determinants, no framework to measure innovation
was proposed.
A systematic review by (Crossan and Apaydin, 2009) found the common definitions and determinants of innova-
tion based on journals published in ISI Web of Knowledge’s Social Sciences Citation Index (SSCI). The study also
proposed a comprehensive framework of organisational innovation. However, the study did not focus on software
industry and used only one database.
Several researchers have tried to develop a framework to measure innovation. (Berg et al., 2008) proposed a
model for measuring the front-end innovation activities based on three assessment areas: Process, social, and physical
environment. However, the model was developed for the manufacturing sector. Moreover, the proposed model is
solely based on research and development data. Small and Medium Enterprises (SMEs) usually cannot a↵ord to have
dedicated R&D departments (Gorschek et al., 2010a). Therefore, we claim, that this model is not applicable in their
particular context.
(Misra et al., 2005) proposed a goal-driven measurement framework for measuring innovation activities in an
organisation. The framework adopted the Goal-Question-Metric (GQM) approach to define the goals of an innovation
2

program with accompanying metrics. Although they provided a set of metrics for measuring innovation, the study did
not present a clear methodology on how they defined the goal, questions and metrics. The study also did not explain
clearly the relationship between the suggested metrics and innovation.
(Narayana, 2005) proposed an innovation maturity framework to assess the maturity of the innovation process in
a firm. The framework is modelled based on the Capability Maturity Model (CMM). However, the framework has not
been validated and it does not provide any metrics to assess the process maturity. Moreover, it only considers R&D as
the main factor to determine the maturity of a process.
3. Research Methodology
The aims of this study were to establish the state of the art of innovation measurement and to capture the state
of the practice of innovation measurement in the software industry. These aims were achieved by addressing the
following objectives:
1. To identify the perception of innovation in software industry and the definitions of innovation in literature.
2. To identify the determinants or drivers of innovation in the software industry.
3. To identify the commonly used metrics to assess innovation.
4. To identify the existing innovation measurement models and major challenges in evaluating innovation.
A systematic literature review was conducted to establish the state of the art of innovation measurement. In this
study, we conducted a systematic review as proposed by (Kitchenham and Charters, 2007) to accumulate primary
studies aiming to improve the understanding and to ascertain the validity and reliability of claims and propositions. A
defined review protocol, search strategy, explicit inclusion and exclusion criteria, and specified information that will
be retrieved from primary studies di↵erentiates a systematic review from a conventional literature review (Kitchenham
and Charters, 2007).
To capture the state of the practice of innovation measurement, we used a questionnaire and face-to-face meetings
to collect the opinions of software industry practitioners and academics. We followed the main steps in developing a
questionnaire as suggested by (Kasunic, 2005). The results of the questionnaire were then used to confirm the findings
of the systematic review.
Based on the findings from the systematic literature review and questionnaire, we conducted a conceptual analysis
to develop an innovation measurement model. This model was evaluated through interviews with industry practition-
ers and academics. Interviews were also used to capture the perception of innovation and state of practice regarding
innovation measurement in software industry, industrial interviews focused on evaluation of the usefulness and appli-
cability of the proposed model for industry. Whereas the aim of the academic interviews was to assess the complete-
ness and correctness of the proposed model. Both industrial and academic interviews in this study each lasted for one
hour. Overview of the research objectives and research methods used to achieve them are shown in Figure 1.
3.1. Systematic Literature Review
3.1.1. Search Strategy
We used seven online databases to perform the search: Inspec and Compendex (through Engineering Village),
Scopus, IEEE Xplore, ACM Digital Library, ScienceDirect and Business Source Premier (BSP). The target for this
review was journals published in the fields of engineering, economics, computer science, finance and management
between 1949 and 2010. We choose this starting point even though the concept of innovation was first introduced by
Schumpeter in 1911, the first English translation of his book, Theorie der wirtschaftlichen Entwicklung, was published
in 1949 (Schumpeter, 1949). The search was conducted in early February 2010. We did not preselect the journals and
relied on the journals included in the seven databases used.
To ensure that all the performed searches were consistent and comparable for each database, we used selected
keywords and expressions. The generic search string with combination of keywords to answer the research questions
was as follows:
(“innovation” AND (“evaluat*” OR “assess*” OR “measur*” OR “metric*” OR “determinant” OR “driver” OR
“key elements” OR “indicator*” OR “attribute”))
3

Literature review!
• Definitions!
• Determinants!
• Metrics!
• Models!
Questionnaire !
• Perception!
• Importance!
• Metrics!
• Frameworks!
• Challenges!
Conceptual
Analysis!
Used in!
Innovation
measurement model!
Resulted in!
Used in!
State of the art! State of the practice!
Interviews!
• Perception!
• Importance!
• Metrics!
• Frameworks!
• Challenges!
Used in!
Interviews!
• Feedback!
Evaluated in!
Revision based on feedback!
Figure 1: Mapping of research aims and methodology as employed in this study.
3.1.2. Study Selection Criteria
Three inclusion/exclusion criteria were defined to select the relevant articles for this study. The first criteria made
sure that we only considered those primary studies, which are published in journals and are written in English. No
duplicate studies were allowed. The second inclusion/exclusion criteria were based on the relevance of the primary
studies to innovation and innovation measurement. The relevance of the studies was decided based on the title and
abstract the criteria for this decision is listed in Table 1.a. If a decision could not be made, the introduction and
conclusion of the article were analysed. If there was still an uncertainty about the paper, it was classified as ‘Doubtful’
and submitted for discussion with the second reviewer.
The third inclusion/exclusion criteria (see Table 1.b) were intended to check whether the relevant primary studies
discussed the key concept of innovation and innovation measurement. To ensure that we had access to the selected
key papers, we collaborated with a librarian to retrieve the papers in printed format. Any article available in full-text
was included provided it met one of the remaining criterion in Table 1.b.
Before performing the actual selection procedure a pilot selection was performed where both reviewers applied
selection criteria on same 30 papers, individually. Then the results were compared to see if the two reviewers had a
shared understanding of the criteria. By discussing the conflicts a coherent understanding of the criteria and procedure
was developed. After having the same understanding, reviewers performed the actual studies selection. The list
of studies extracted from the resources was divided equally among the reviewers and each member did selection
independently.
Before executing the data extraction, we performed a pilot extraction to ensure that each reviewer understood
and had the same interpretation of the form and data to be extracted. The pilot extraction was performed in a manner
similar to the studies selection procedure. After having the same understanding and interpretation of the data extraction
strategy, the actual data extraction was performed. While reading the full-text of the studies, key concepts from each
study were extracted according to the agreed upon form finalised after piloting.
3.1.3. Characteristics of Primary Studies
We retrieved a total of 13,401 articles from all seven databases. By applying the three inclusion/exclusion criteria
sequentially, we accepted 204 articles as primary studies. In the preliminary selection, we rejected 2,683 articles for
not meeting the requirements of being written in English, being peer-reviewed article or being a duplicate. Applying
the relevance criteria, we rejected 10,273 articles based on title and abstract. We applied the advanced criteria to the
remaining articles and filtered out 75 of them, which were unavailable in full-text. We also rejected 166 articles since
4

Table 1: Selection criteria.
a. Relevance criteria
1. Include the articles that focus on organisational level and exclude the ones that explore
regional strategies and policies or other external entities for innovation.
2. Include the articles from management, software engineering, economics and computer
science and exclude the articles from chemical, mechanical, medical, education, etc.
3. Include an article that discusses or potentially discusses any of the following: defini-
tion or determinants of innovation or metrics or frameworks for innovation measure-
ment (we revisited these criteria when we performed advanced inclusion criteria as
shown in Table 1.b). However, exclude an article that presents inventions or discusses
the usage, application and evaluation of inventions.
b. Advanced inclusion criteria
1. Full-text is available.
2. The article discusses a definition of innovation.
3. The article discusses the determinants of innovation.
4. The article describes one or more metrics to measure innovation.
5. The article gives an overview of a model or framework for innovation measurement.
6. The article compares two or more existing frameworks for innovation measurement.
7. The article analyses or evaluates an existing framework for innovation measurement.
Table 2: Quality assessment results.
Inclusion/Exclusion criteria Number of publications
Yes Partially No
1 Is the aim of the study clearly explained? 194 2 8
2 Is the presented methodology/approach clearly stated? 121 24 59
3 Are the threats to validity of the study analysed? 42 16 146
4 Is an appropriate definition of innovation provided? 41 12 151
5 Is there empirical evidence provided in the study? 91 9 104
they did not discuss the key concepts that we were looking for and finally, as mentioned previously, we accepted 204
remaining articles as the selected primary studies.
As suggested by (Kitchenham and Charters, 2007), we developed a check-list to assess the quality of the selected
primary studies. This assessment was not part of the data extraction form as it was assessed separately. The primary
studies were evaluated based on the quality criteria presented in Table 2. The quality criteria were rated with ‘Yes’,
‘No’ and ‘Partially’ depending on the fulfilment of each criterion. The assessment criteria were not used to include
or exclude the papers nor did it have any influence on data synthesis. It is just used to indicate the quality of current
work.
From Table 2, it can be seen that many studies are of good quality according to criteria 1, 2 and 5. We identified
87 studies as empirical studies, but not all of these discuss validity threats. Moreover, 42 studies were industry reports
which did not clearly state the methodology used. This might explain why only a few studies discuss the validity
threats (criteria 3). Only 41 studies formulated their own definitions of innovation or used existing definitions. Twelve
studies did not clearly define innovation and 151 studies did not have a definition (criteria 4).
The goal of this identification was to analyse the trend of selected primary studies from the viewpoint of the used
research methods. The primary studies were classified based on the research method mentioned in the article. Hence,
the categories of the studies are:
• Survey: The study uses questionnaires, interviews or both to collect empirical data. We found 36 studies in this
category.
• Case study: The study declares the use of case study to answer one or more research questions. There are 5
studies that use case study method.
5

• Experiment: The study uses an experiment to examine the hypothesis. The study also clearly describes the
design of the experiment. Only one study included in this review performs an experiment.
• Conceptual analysis: The study presents a theoretical concept without empirical evaluation. 37 studies use
conceptual analysis.
• No research method specified: All the studies that do not state explicitly the research method used are grouped in
this category. It includes, many times, the studies that report sectorial or regional level studies i.e. the experience
of individual countries or groups of countries. 21 studies did not specifically mention the research method used
in their research.
Based on these categories, we found that 37% of the studies used conceptual analysis as a basis for their study
and 36% of the studies used a survey approach. Surveys are considered as the main instrument to collect quantitative
and qualitative data followed by statistical analytical methods to validate the concept. 20% of the studies presented
a theoretical concept but used existing or published data to evaluate the concept. In this study, such studies are not
considered as categorical empirical studies since the researchers just took freely available data and did not collect the
data by themselves. However, these studies are marked as ‘conceptual analysis with empirical evaluation’ and treated
as a subset of this category. Only 17% of the studies are classified as pure conceptual analysis.
3.2. Interview
We used a semi-structured interview to grasp as much information as we could get from the interviewees. With the
interviewee’s permission the interviews were recorded. We followed the interview protocol as described in (Creswell,
2009) to conduct the interviews. Four industry practitioners and three academics were interviewed in this study. Both
academics and industry practitioners were selected to ensure that we get balanced information from academia and
industry point of view. We chose four industry practitioners in managerial level of the software firm who have roles
and responsibilities related to innovation. For academics, we selected three key professors in software engineering
and business who have experience and interest in innovation and also have close relationship with the industry. Due to
geographical limitation, we only selected the interviewees who were based in Sweden. In this study, the interviewees
have been anonymised for confidentiality and they are referred to with female pronoun regardless of their actual
gender. The background information of interviewees including their experience, role, responsibilities and the size of
their company are available online at (bin Ali et al., 2011).
3.3. Questionnaire
A web-based questionnaire served as an instrument to collect the opinions of those experts whom we could not
interview face-to-face. The target respondents were software industry practitioners, i.e. software engineers, analysts,
project managers, R&D managers, CTOs, CEOs, etc. To get a higher response rate, a personalised email was sent
to industrial contacts inviting them to participate in the questionnaire. The request for participation was also posted
on researchers’ respective alumni mailing lists. Due to the limited time, we gave seven days for respondents to
complete the questionnaire. The data submitted by respondents was taken into consideration only if the respondents
fully completed the questionnaire within the deadline. In the end, we had 145 respondents out of which 94 completed
the questionnaire. This means that our completion rate was 64.83%. The responses with incorrect demographic
information, incomplete or late responses were discarded.
We categorised the respondents into similar roles. Ten di↵erent roles were identified from the responses. Approx-
imately a quarter (25.53%) of the respondents were software engineers, 20.21% were categorised as senior software
engineers and 23.40% had management or executive responsibilities in their respective organisations. Out of the 94
respondents from 13 di↵erent countries, 42.55% were from Pakistan, 22.34% were from Indonesia, 12.76% were
from the USA and 12.7% were from Europe. They came from 68 di↵erent organizations.
The respondents came from a variety of organisations of varying sizes (in terms of number of employees). For
categorisation based on organisation size, the size definitions, according to (EC, 2005), were followed. Slightly more
than a quarter of the respondents (26.59%) were from organisations with 500 to 10,000 employees, 24.46% from
organisations with 250 to 500 employees and 19.14% from organisations with 50 to 250 employees. The distribution
of respondents according to organisation size is shown in Figure 2.
6

0
5
10
15
20
25
30
35
40
45
50
1-10
10-50
50-250
250-500
500-10,000
≥ 10,000
4
14
18
23
25
10
4
13
16
6
22
8
Number of respondents
Firm size (number of employees)
Number of respondents
Unique firms
Figure 2: Distribution of respondents according to organisation size.
4. Results
We present major findings of the study in this section. Due to space limitations the detailed results and supple-
mentary information
1
are available online at (bin Ali et al., 2011).
4.1. State of the Art of Innovation Measurement
The following section describes the state of the art of innovation measurement based on systematic literature
review.
4.1.1. Definitions of Innovation as Reported in Literature
The 41 definitions found in the literature review were analysed and various aspects of innovation were identified.
These aspects are considered important as these delineate what attributes will be measured when an organisation
attempts to measure innovation. Some of these definitions are shown in Table 3.
Impact of innovation. Based on the impact on the market and the change in underlying technology, innovation is
classified into four major categories:
• Incremental innovation: These are relatively minor changes in technology based on existing platforms that
deliver relatively low incremental customer benefits (Chandy and Tellis, 1998; Cheng and Shiu, 2008).
1
The information online includes:
• The exact search strings used in respective databases along with the number of articles
• Definitions of innovation found in this literature review with the references to the source publications
• Classification of determinants and respective measures of innovation along with a reference to source publications.
• Mapping of determinants of innovation to innovation lifecycle.
• Metrics of innovation that have been studied in the context of software industry.
• Metrics with respect to innovation life-cycle activities.
• Metrics for innovation input, innovative output and its performance.
• Existing frameworks for evaluation of innovation.
• Primary studies used in this study from software context.
• The questions used in the interviews during this study.
• The questionnaire used for a web based survey in this study.
• List of the primary studies used for data extraction in this study.
7

Table 3: Definition of innovation.
(Acs and Audretsch, 1988) ‘Innovation is a process that begins with an invention, proceeds with the development of the inventions, and
results in the introduction of a new product, process or service to the market-place.’
(Damanpour, 1992) ‘Innovation is defined as the adoption of an idea or behavior whether a system, policy, program, device,
process, product or service that is new to the adopting organization.’
(de Jong and Kemp, 2003) ‘Innovation behavior can be defined as all individual actions directed at the generation, introduction and
application of beneficial novelty at any organization level.’
(Fruhling and Keng, 2007) ‘Innovation is an idea, practice or object that is perceived as new to an individual or another unit of adop-
tion.’
(Geiger and Cashen, 2002) ‘Innovation refers to the creation of new product within the firm.’
(Hage, 1999) ‘Organizational innovation has been consistently defined as the adoption of an idea of behavior that is new
to the organization. The innovation can either be a new product, a new service, a new technology, or a new
administrative practice.’
(Palmberg, 2004) ‘Innovation is defined as ”a technologically new or significantly enhanced product compared to the firm’s
previous product” which has been commercialized on the market.’
(Dibrell et al., 2008) ‘Innovations vary in complexity and can range from minor changes to existing products, processes, or
services to breakthrough products, and processes or services that introduce first-time features or exceptional
performance.’
• Market breakthrough: These are based on core technology that is similar to existing products but provides
substantially higher customer benefits per dollar (Chandy and Tellis, 1998; McMillan, 2010).
• Technological breakthroughs: These innovations adopt a substantially di↵erent technology than existing prod-
ucts but do not provide superior customer benefits per dollar (Chandy and Tellis, 1998; McMillan, 2010).
• Radical innovation: They are referred to as disruptive innovations which introduce first time features or excep-
tional performance (Dibrell et al., 2008; Assink, 2006). They use a substantially di↵erent technology (Herrmann
et al., 2006; Chandy and Tellis, 1998) at a cost that transforms existing or creates new markets (Assink, 2006)
and deliver a novel utility experience to customer (Herrmann et al., 2006).
Types of innovation. There are four types of innovation according to our primary studies:
• Product innovation: This refers to creation and introduction of new (technologically new or significantly im-
proved) products which are di↵erent from existing products (Hage, 1999; Geiger and Cashen, 2002; OECD,
2005; Herrmann et al., 2006; Singh and Singh, 2009; Amara et al., 2009; Jensen and Webster, 2009; Carmona-
Lavado et al., 2010). It includes architecture structure (Schumpeter, 1949; Romijn and Albaladejo, 2002), tech-
nology (Hage, 1999), features (Romijn and Albaladejo, 2002; Assink, 2006; Dibrell et al., 2008), performance
(Dibrell et al., 2008).
• Process innovation: This refers to implementation of a new design, analysis or development method that
changes the way how products are created (Acs and Audretsch, 1988; OECD, 2005; Singh and Singh, 2009;
Amara et al., 2009; Jensen and Webster, 2009).
• Market innovation: This refers to implementation of new or significantly modified marketing methods, strate-
gies and concepts in product design or packaging, placement, promotion or pricing (OECD, 2005; Amara et al.,
2009). It includes opening up new market opportunities, position innovations (including changes in the con-
text in which the products are introduced) and the implementation of new or significantly modified marketing
strategies (Singh and Singh, 2009; Jensen and Webster, 2009).
• Organisation innovation: This refers to the implementation of a new organisational method in the firm’s busi-
ness practices, workplace organisation or external relations (OECD, 2005). It includes changes in the archi-
tecture of production and accounts for innovations in management structure, corporate governance, financial
systems or employees remuneration system (Hage, 1999; Jensen and Webster, 2009; McMillan, 2010).
Degree of novelty of innovations. This aspect of innovations has di↵erential implications for organisation growth
(De Jong and Vermeulen, 2006). This is also useful to identify developers and adaptors of innovation who can
di↵erentiate market leaders from followers (OECD, 2005). Based on the perspectives important to the firm, there are
four types of innovation based on novelty:
8

• New to the firm: The minimum level of novelty of innovation is that it must be new to firm. It is defined as
the adoption of an idea, practice or behaviour whether a system, policy, program, device, process, product,
technology or administrative practice that is new to the adopting organisation (Damanpour, 1992; Hage, 1999;
Parashar and Sunil Kumar, 2005; Berger and Revilla Diez, 2006; Fruhling and Keng, 2007; Linton, 2007;
Carmona-Lavado et al., 2010).
• New to the market: When the firm is the first to introduce the innovation to its market (Acs and Audretsch,
1988; OECD, 2005).
• New to the world: These innovations imply a greater degree of novelty than new to the market and include
innovations first introduced by the firm to all markets and industries, domestic and international (OECD, 2005;
Berger and Revilla Diez, 2006).
• New to the industry: These innovations are new to the firm’s industry sector (Garcia and Calantone, 2002;
De Jong and Vermeulen, 2006; Beugelsdijk, 2008).
Innovation activities. Schumpeter’s definition stresses the novelty in innovation (Schumpeter, 1949). While
invention is the first occurrence of ideas, innovation refers to their first economic utilisation or commercialisation
(Schumpeter, 1949; Haner, 2002) and successful launch to market as products (Banerjee, 1998). This idea still domi-
nates the definitions and the necessity of commercialisation for innovation is emphasised. Innovation is described as
the discovery of new ideas (with value in marketplace (Orlando and Verba, 2005)) and successful exploitation through
commercialisation of these discoveries (Gaynor, 2001; Adams et al., 2006; Jensen and Webster, 2009; Oke et al.,
2009). It is the tendency to not only implement products but also to introduce them to the market (Gumusluo
ˇ
glu and
Ilsev, 2009). In situations where the innovation relates to an internal activity like the improvement in a development
process, the phrase commercialisation can be substituted with ‘implementation’ (Gaynor, 2001).
Nature of process. Innovation is an iterative process (Garcia and Calantone, 2002). In this process, idea generation
or adoption, development of products (by using competencies inside and outside the organisation), and introduction of
products to marketplace and implementation of processes in organisation are the major activities (Acs and Audretsch,
1988; Damanpor, 1996; Garcia and Calantone, 2002; Cormican and O’Sullivan, 2004; Aiman-Smith et al., 2005;
Freeman and Engel, 2007). Other definitions focus on where the ideas for an innovation are generated; (Wakasugi and
Koyata, 1997) suggest that innovations are generated as products moved through product life-cycle processes from a
research laboratory to the factory. Yet others see innovation as creation of knowledge by application, recombination
or extension of existing knowledge (Cho and Pucik, 2005; Parashar and Sunil Kumar, 2005). It can be understood as a
process in which organisations’ knowledgeable and creative people define and frame problems and then develop new
knowledge to understand and solve them (Caloghirou et al., 2004; Adamides and Karacapilidis, 2006).
4.1.2. Determinants of Innovation
We identified 244 determinants of innovation and classified them into two groups: external and internal determi-
nants. This classification was created based on the sources of factors, whether from outside or inside the organisation.
External determinants are factors outside an organisation, which a↵ect innovation, and are beyond the control of
the organisation, e.g. public policy that reduces the tax for start-up companies or R&D grants for small companies
(Frenkel et al., 2003). Internal determinants are factors inside the organisation’s influence that improve the innovation
capability of the organisation, e.g. the availability of a strategy on innovation or a creative climate (Cormican and
O’Sullivan, 2004) (confirmed also in (Gorschek et al., 2010b) which unfortunately was only published recently, thus
falling out of the scope of the systematic literature review).
We also categorised the individual internal determinants into groups based on their meaning and purpose. For
example, we grouped the determinants related to customers into customer-related determinants and the determinants
associated with marketing into marketing-related determinants. Table 4 presents the taxonomy of internal determi-
nants.
Out of 23 studies conducted in a software-related context, there were only six studies that identified determinants
of innovation. However, this does not mean that determinants found in other contexts are not applicable to the software
industry. The following discussion is based on the findings of determinants studied in a software industry context.
A study by (Akman and Yilmaz, 2008) found that customer-orientation is one of the important factors that signifi-
cantly a↵ects the innovation capability. They argue that by focusing on customers, software organisations will be able
9

Table 4: Taxonomy of internal determinants.
Market Knowledge & information Technology Empowerment
Tool-support Planning R&D Acquisition & alliances
Champions Intellectual property Alignment Size
External collaboration Structure Financial Culture
Management Commitment Risk Trust
Organisation resources Individual Customer-orientation Strategy
Policy Internal collaboration Networking Human resources
to improve their innovation capability since customers’ needs and wants are the source of innovative ideas. To this
end, (Paladino, 2007) and (Voss, 1985) suggested that managers should look for a new strategy to fulfil the market
needs and, moreover, an organisation should not focus on current needs but also future needs, and this can be done
when an organisation maintains a good relationship with its customers (Akman and Yilmaz, 2008).
Inter-functional co-ordination is another factor that has a significant e↵ect on innovative capability (Akman and
Yilmaz, 2008). Good and integrated co-ordination among all departments can promote an e↵ective knowledge transfer
inside the firm. Innovative ideas are shared among the employees and transformed into innovation outcome. However,
according to (Voss, 1985) this happens only if the customers know the problems and the technology is available inside
the organisation. Otherwise, an organisation needs to communicate with an external side to better identify a potential
market and, ultimately, reach commercial success (Voss, 1985).
To support inter-functional co-ordination, a study by (Romijn and Albaladejo, 2002) found that in small hi-tech
companies, specialised knowledge and experience in science and technology should be in place. This is more im-
portant than technical or general managerial skills. Hence, start-ups need at least two important things: access to
universities or other research institutes to train the sta↵s and R&D funding to maintain innovation while learning to
achieve technological excellence.
On the other hand, competitor-orientation does not significantly a↵ect the innovation capability (Akman and Yil-
maz, 2008). In the software industry, competitors are many times treated as partners rather than as rivals (Akman
and Yilmaz, 2008). Software companies tend to collaborate in software development with competitors. This usually
happens for SMEs in order to survive the competition from large enterprises. Therefore, being aggressive—by allo-
cating more resources to improve the market position—will not improve innovation capability per default. The same
e↵ect can also be seen when companies become defensive, i.e. by making incremental innovation on current products
(Akman and Yilmaz, 2008).
Di↵erent from other industries, technology-orientation has a negative impact on the software industry (Akman and
Yilmaz, 2008). Software development highly depends on skilled and creative employees rather than technology. This
result echoes Brooks’ statement, that ‘software construction is a creative process’ (Brooks, 1987). Moreover, Brooks
argued that no silver bullet can solve ‘the essential difficulties in the nature of software’ (Brooks, 1987). This creative
work should be supported by a creative climate, e.g. challenging work, freedom and group support (Fagan, 2004).
In hardware technology focused areas, as in the manufacturing industry, technology breakthrough has improved
the productivity and reliability of computer hardware. In the software industry, analysis and design has become the
crucial phase in software development (Voss, 1985; Brooks, 1987). A good software design can be reused in other
projects, will shorten the development period and, ultimately, the costs as well. Hence, this development aspect needs
to be managed properly.
A study by (Gumusluo
ˇ
glu and Ilsev, 2009) found that transformational leadership also has positive e↵ects on a
company’s innovation capabilities. The study found that the positive influence is visible in incremental innovation in
developmental work. When this pro-active behaviour is part of an innovation strategy, the organisations seem to be
more flexible to facilitate innovations (Akman and Yilmaz, 2008).
The importance of a product champion (in the context of the software industry) was introduced by Voss (Voss,
1985). Champions enthusiastically encourage and promote innovation during the project. They are the individuals
with a strong reputation, knowledge of the business, possessing good facilitating skills and having the authority
to bring change into the organisation (Voss, 1985; Andrew and Sirkin, 2008). They are responsible for the entire
innovation process and play their role in the development stage to help initiate the project while staying involved in
10

Table 5: Classification of metrics based on key aspects of innovation.
Category Number
of metrics
Example
Determinants 84 The existence of a project champion (Dodgson and Hinze, 2000), knowledge sharing
(Lin, 2007), government regulation e↵ect (Kumar et al., 1996)
Inputs 21 R& D expenditure (Berger and Revilla Diez, 2006), R&D intensity (ratio of R&D
expenditure to total assets) (Desyllas and Hughes, 2009), percentage of workforce
time that is currently dedicated to innovation projects (Muller et al., 2005)
Outputs 35 Patent density (Kivim
¨
aki et al., 2000), new organisational programs (Russell, 1990),
number of new processes and significant enhancement per year (Chiesa et al., 1998)
Performance 33 Percentage of sales that is generated by new products (Akman and Yilmaz, 2008), ci-
tation ratio (Banerjee, 1998), impact of brand (Sundstr
¨
om and Zika-Viktorsson, 2009)
Activities 59 Percentage of ideas funded (Baron and Tang, 2009), quality of adaptation (Buganza
and Verganti, 2006), managers survey (Jensen and Webster, 2009)
the commercialisation stage as well (Voss, 1985; Andrew and Sirkin, 2008).
(Voss, 1985), and then (Akman and Yilmaz, 2008), also found that innovation should be part of an organisation’s
strategy, plan and culture. Once innovation has become a daily habit inside the organisation, the innovation process
will also become more e↵ective (Voss, 1985).
4.1.3. Metrics for Innovation Measurement
A large number of metrics have been suggested in literature to measure innovation. In this study, 232 metrics were
identified and classified into three levels based on measurement context in Firm, Industry and Region level metrics.
We found that 88% of the studies focused on innovation in individual companies (Firm level) while 1% and 11% were
used for measurement at Industry and regional level respectively. Moreover, only 48 out of the total 232 metrics were
reported in studies related to the software industry.
37% of the metrics found in the literature have been statically validated, as opposed to dynamically validated
research results that are evaluated in live industry settings (Gorschek et al., 2006). Most of these metrics are validated
through statistical analysis applied on empirical data. The data may have been collected directly from industry, from
published data or existing databases. The majority of the metrics, 58%, were mere suggestions and have not been
subjected to any or use in practice. Only 5% of the metrics have been used in industry, but no information about the
validation of these metrics was available.
Beside the measurement level, the 232 metrics were classified based on the innovation aspect they measure, as
shown in Table 5.
4.1.4. Existing Innovation Measurement Models
We identified 13 existing innovation measurement frameworks reported in literature. We only considered the
measurement frameworks that were proposed by the studies and rejected any other frameworks developed as a tool to
prove their concepts.
Most of the existing frameworks found in this study were more focused on measuring the technological aspect of
innovation. There are eight frameworks which are intended to measure technical capability, i.e.(Kumar et al., 1996;
Chiesa et al., 1998; Furman et al., 2002; Capaldo et al., 2003; Cormican and O’Sullivan, 2004; Yam et al., 2004; Wang
et al., 2008; Nirjar, 2008). However, according to the Oslo Manual (OECD, 2005), technology is only one aspect of
innovation. Thus these frameworks cannot be used to measure overall innovation that includes non-technological
aspect as well. For this reason, in the latest version of the manual, the Organisation for Economic Co-operation and
Development (OECD) removed the term technology in the definition and introduced two new types of innovation:
marketing and organisation.
One framework focuses on product innovation (Maravelakis et al., 2006), while the remaining frameworks do not
clearly specify the measurement area. There is a lack of empirical validation since only the study by (Byrne et al.,
2007) reports the usage of six sigma in industry. (The study itself focuses on radical innovations.)
11

Out of 13 frameworks, index of innovativeness (Nirjar, 2008) is the only framework that focuses on software
firms. The framework was based on six metrics adapted from the Oslo Manual. Using the data from these metrics,
an index of innovativeness is calculated. Based on this number, the firms can be plotted into three dimensions to
see the relative position of the firms in a particular geographic market. However, this framework does not take into
consideration organisational and marketing innovations.
Surprisingly, there was only one framework that has been validated in industry: the technical audit framework by
(Chiesa et al., 1998). This might be the reason why other technical audit-based frameworks have cited this framework,
e.g. (Capaldo et al., 2003; Cormican and O’Sullivan, 2004; Yam et al., 2004).
4.2. State of the Practice in Innovation Measurement
4.2.1. Interview results regarding innovation measurement practice
The results from the five interview respondents with extensive industrial experience, related to the practice of
innovation measurement in industry are summarised here. Two Interviewees defined innovation as “development of
new processes and products”. While each of the remaining defined it as, “improvements in existing processes and
products”, “value addition” and “surpassing user expectation”. All five said that innovation was a strategic objective
of their organisation. Two of three interviewees said they have a defined process for innovation, same was the result for
innovation measurement in the organisation. All interviewees expressed dissatisfaction with the visibility and control
they get from the current metrics (number of ideas and patents) used in the organisation. Finally all five industrial
interviews said they do not use a framework for innovation measurement. The results of the interviews are further
discussed in the light of the findings of the literature review and questionnaire in Section 5.
4.2.2. Perceptions of Innovation
In general, all respondents agreed that idea generation is the most important activity of innovation and commer-
cialisation is the least. However, when the data was analysed with di↵erent roles of the respondents’ new trends appear
(see Figure 3).
020406080100
Overall
Technical
Non technology related
Managers
Senior management
Executives
Percentage of respondents
Respondents categoriezed based on roles
Commercialization
Improvement
Development
Idea
Figure 3: What constitutes innovation?
Based on the respondents’ roles, it can be seen that respondents in technical roles consider that innovation consist
of idea generation and new development. This means that respondents in technical roles are more focused on techno-
logical breakthroughs. The main reason for this is most likely the nature of their responsibilities; as technical sta↵ is
more concerned about technical aspects rather than commercialisation.
On the other hand, 66.67% of senior management and 75% of executives consider commercialisation an important
constituent of innovation in contrast with only 18.30% of the more technically-oriented respondents. This is an
12

expected trend as higher management is more concerned about business objectives and the financial success of the
company. This is also in line with the importance of commercialisation found in the systematic literature review.
Respondents in higher management roles also recognised the presence of incremental innovation. This was also
understandable since radical innovation has higher financial risk than incremental. Hence, showing, once again, that
focusing on current customers is required when wanting to sustain a long-term business relationship.
Types of innovation. The respondents with technical roles have a strong inclination on product innovation fol-
lowed by process, market and organisation (see Figure 4). The main reason for this is that practitioners with technical
perspectives are mainly involved with product and process development so they consider innovations only in this, lim-
ited, perspective. On the other hand, the respondents with management and executive roles prefer process innovation
to product innovation. Furthermore, as they have a business perspective, they considerably emphasise the ‘Market’
followed by ‘Organisational’ innovations.
020406080100
Overall
Technical
Non technology related
Managers
Senior management
Executives
Percentage of respondents
Respondents categoriezed based on roles
Organization
Market
Process
Product
Figure 4: Types of innovation.
4.2.3. Importance of Innovation Measurement
The respondents were almost unanimous in considering the importance of innovation measurement. This is inline
with the findings by the BCG survey (James et al., 2008) and validates the findings of the systematic review, where
innovation measurement has been extensively emphasised.
Innovation strategy and measurement. Only 40 respondents (42.55%) said that they were aware of an explicit
innovation strategy for the organisation while 41 respondents (43.61%) were sure of no explicit strategy for innovation
in their organisations. Similarly, only 33 respondents (35.10%) said that their organisation measured innovation
while 38 respondents (40.42 %) were sure of no innovation measurements being performed in their organisations.
Remaining, 13 respondents (13.82%) and 23 respondents (24.46%) expressed unawareness of an explicit strategy
or measurement program in the organisation, respectively. One reason for such high unawareness is the challenge
of knowledge sharing, the strategies are often not spread throughout the organisation. Therefore, in the case of
technical resources, such unawareness can be expected concerning strategies and measurement programs. Out of the
13 respondents unaware of such a strategy in their organisations, 12 were technically-oriented respondents. Similarly,
out of 23 respondents who were unsure about innovation measurement practices, 19 were technically-oriented and 4
were managers.
The 24 respondents who were aware of an explicit strategy on innovation also reported an innovation measurement
program in their organisation. Similarly, the 27 respondents who claimed there was no explicit strategy reported that
they do not engage in innovation measurement. This would be expected, as an organisation with explicit emphasis
on innovation is more likely to use measurements for innovation management and vice versa. It was interesting to
13

see there were six respondents who said their organisations do not have an innovation strategy but claimed they do
innovation measurement. One possible explanation is the fact that employees are unaware of the strategies in the
organisation, while measurement programs a↵ect everyone when it is operational. Another possible explanation for
this result is that the organisation measures aspects related to innovation but not under a defined strategy.
Organisation size, innovation strategy and measurement. The respondents from firms with 10 to 10,000 employ-
ees showed a lack of having an innovation strategy (Figure 5). However, the results are interesting at both extremes
considering employee size. The smaller firms, with less than 10 employees, are most likely not to have an explicit
strategy but 75% of respondents in this range reported having a strategy. Since, we had only four participants from
organisations with less than ten employees the result is, of course, inconclusive. On the other hand, as expected, 60%
of employees with a firm size over 10,000 employees reported having an explicit innovation strategy.
0
10
20
30
40
50
60
70
80
1-10
10-50
50-250
250-500
500-10000
шϭϬϬϬϬ
Percentage of respondents
Firm size (number of employees)
Yes
No
I do not know
Figure 5: Innovation strategy and firm size.
The trend in responses regarding innovation measurement is very similar to the responses concerning innovation
strategy (see Figure 6). From the questionnaire results (shown in Figs. 5) we can see that the practice of innovation
measurement is driven by the existence of an innovation strategy.
4.2.4. Metrics Used in Measuring Innovation
A subset of metrics from the findings of the systematic literature review, covering key aspects of innovation
measurement, was put in the questionnaire. The respondents were grouped on the basis of the company they belonged
to. We had a total of 28 unique companies that measured innovation. One response was ignored since eight other
respondents from the same company in the same and di↵erent roles reported that there was no innovation measurement
in their organisation. These metrics are examples for the six groups of metrics based on key aspect of innovation, as
described in Table 5.
The most widely used metrics were “Number of improvements in existing products” (63%) followed by “Number
of ideas converted successfully to products” (59%), “Percentage of sales spent on new projects” (56%) and “Improve-
ment in product quality as a result of innovation” (56%), whereas, “Creative environment (Subjective assessment)”
(19%) preceded by “Presence of innovation champions” (26%) and “Time taken in converting an idea to a product or
process” (33%) were the least used metrics. This shows an emphasis on measuring inputs and outputs of innovation
in terms of resources spent on innovation and results in the form of improvements in processes and products (with
regards to the last, especially improved product quality). This result was expected as factors like environment and
commitment, which are necessary to create an environment conducive to innovation, are often ignored.
Satisfaction from metrics used. Unlike the results from existing surveys, e.g. the BCG survey (James et al., 2008),
the questionnaire respondents expressed satisfaction with the metrics that they currently use to measure innovation
14

0
10
20
30
40
50
60
70
80
1-10
10-50
50-250
250-500
500-10000
ǃŗŖŖŖŖ
Percentage of respondents
Firm size (number of employees)
Yes
No
I do not know
Figure 6: Innovation measurement and firm size.
(see Figure 7). It seems that these metrics give them enough control and visibility to manage innovation. However,
since there were only six non-technical respondents who said that their organisations measure innovation we do not
have enough of a sample size to conclude anything here. One of the respondents, who expressed dissatisfaction, said
that the metrics are currently calculated in ‘Silos’ without a defined governing process. This hinders organisation-wide
sharing of metrics about innovation, which may otherwise prove very useful for business teams. Another dissatisfied
respondent mentioned that the cycle time (time taken to convert an idea to a product/process) could be a useful metric
to identify a ‘bloat in the process’. However, these metrics were not currently collected in the organisation.
0
20
40
60
80
100
120
Overall
Technical
Non
technology
related
Managers
Senior
management
Executives
Percentage of respondents
Respondents categoriezed based on roles
Yes
No
Figure 7: Satisfaction with used innovation measurements.
4.2.5. Frameworks Used for Innovation Measurement
The importance of an innovation measurement initiative has been recognised in the software industry. For exam-
ple, 32 respondents from 27 unique firms said that their organisation does measure innovation. However, of these
15

firms, 74.07% do not use any measurement frameworks. One respondent mentioned using a survey for evaluating in-
novation in the company. Another respondent mentioned the use of a framework but did not share any details because
it was confidential.
4.2.6. Challenges of Innovation Measurement
Based on the systematic literature review, we identified five common challenges of innovation measurement. We
asked the respondents whether the same challenges also existed in their context. The following section discusses the
results of the questionnaire from this perspective:
• Lack of recognition of the importance of innovation measurement: Respondents from firms smaller than 10,000
employees generally agreed to the statement that innovation measurement importance is not recognised. On
the contrary, larger organisations do not consider that there is a lack of acknowledgement of innovation mea-
surement importance. These organisations are already undertaking innovation measurements (as found in the
systematic literature review and the questionnaire; see Figure 8).
0
25
0
50
25
14
57
14
7
7
28
56
0
17
0
4
39
17
35
4
20
48
20
12
0
10
20
40
30
0
0
10
20
30
40
50
60
Strongly Agree
Agree
Undecided
Disagree
Strongly disagree
Percentage of respondents
1-10
10-50
50-250
250-500
500-10000
шϭϬϬϬϬ
Firm size (number of employees)
Figure 8: Lack of recognition of importance of innovation measurement.
• Lack of consistent definition of innovation: With the exception of firms with 1–10 employees (see Figure 9),
there seems to be a consensus that there is a lack of consistent definitions and that it hinders innovation measure-
ment. This result is consistent with the findings of the systematic literature review where 41 definitions were
found. For larger organisations the issue is a lack of consistent definitions of innovation, metrics and guidelines.
Whereas for smaller firms the issues also include a lack of recognition that innovation measurement is useful.
• Lack of metrics for innovation measurement: All respondents tend to agree that there is a lack of metrics that is
hindering organisations to pursue innovation measurement (see Figure 10).
• Lack of guidelines and frameworks: Except for respondents from 1–10 employee firms, the majority of the
respondents agree that the lack of guidelines and frameworks is a main challenge for innovation measurement
in their firm (see Figure 11).
• Cost associated with innovation measurement: Like any measurement program, innovation measurement would
also require an undertaking from the company in terms of financial, time and human resources. The respondents
were asked to express their opinion if the cost associated with a measurement program is hindering their firm
from adopting it. However, looking at the graph in Figure 12 we can draw no conclusion, as respondents from
all firm sizes tend to be unsure whether the cost associated with the innovation measurement program is a
challenge. (That in itself is worrying since it most likely a↵ects the willingness to implement measurement
programs.)
16

0
0
0
75
25
7
50
36
0
7
6
67
17
11
0
4
61
17
13
4
12
32
24
32
0
0
70
20
10
0
0
10
20
30
40
50
60
70
80
Strongly Agree
Agree
Undecided
Disagree
Strongly disagree
Percentage of respondents
1-10
10-50
50-250
250-500
500-10000
ǃŗŖŖŖŖ
Firm size (number of employees)
Figure 9: Lack of consistent definition for innovation.
0
0
25
50
25
36
43
14
7
0
28
44
11
17
0
0
61
17
17
4
24
48
16
12
0
10
60
10
20
0
0
10
20
30
40
50
60
70
Strongly Agree
Agree
Undecided
Disagree
Strongly disagree
Percentage of respondents
1-10
10-50
50-250
250-500
500-10000
≥10000
Firm size (number of employees)
Figure 10: Lack of metrics for innovation measurement.
17

0
0
0
75
25
21
64
14
0
0
22
50
6
17
6
0
65
22
9
4
24
44
16
12
4
10
70
20
0
0
0
10
20
30
40
50
60
70
80
Strongly Agree
Agree
Undecided
Disagree
Strongly disagree
Percentage of respondents
1-10
10-50
50-250
250-500
500-10000
≥10000
Firm size (number of employees)
Figure 11: Lack of guidelines and framework for innovation measurement.
0
25
25
25
25
14
21
43
7
14
11
39
17
33
0
9
35
26
26
4
4
32
32
32
0
10
30
40
10
10
0
5
10
15
20
25
30
35
40
45
Strongly Agree
Agree
Undecided
Disagree
Strongly disagree
Percentage of respondents
1-10
10-50
50-250
250-500
500-10000
≥10000
Firm size (number of employees)
Figure 12: Cost of innovation measurement program.
18

5. Discussions
In this section we compare the state of the art and the state of the practice of innovation measurement in the
software industry based on a systematic literature review, a questionnaire and interview results.
5.1. Definition of Innovation for the Software Industry
While defining innovation, one interviewee, emphasised the purposefulness of innovation. It must be targeted
towards ‘identifying and understanding existing and unfulfilled needs and fulfilling them in new ways with time, cost
or usage advantage’. She also mentioned that commercialisation is an important requirement for something to qualify
as innovation.
Another interviewee, described innovation as ‘First thing that comes to mind is invention and new products but
innovation has broader meanings. It means changing something or making something in a better way’. She further
stressed the tangible and intangible nature of innovations (in this case process and product innovations) when she said
‘It can be both something you can touch and a better way of doing things.’ She did not mention the importance of
commercialisation as a criteria to define innovation as she said, ‘Sure if you are working in a company you want to
make money.’
One academic, also emphasised that ‘Innovation is adding value, and value could be customer value, user ex-
perience, internal value, etc.’ According to her, innovation meant ‘delivering something that creates new value, the
value could be perceived by customer, user or producer.’ She highlighted the importance of commercialisation for
innovation. It can be noted here that there is some congruence between all of these definitions. The results of the sys-
tematic literature review and the interviews are also consistent with the view of the dimension of novelty in innovation.
Basically, anything that is new to the firm qualifies as innovation.
The academics and practitioners (with higher management experience) perceive innovation at a much broader level
by using abstract concepts like value creation and need fulfilment. They look at the purpose and goal to define what
may be considered as innovation. While the practitioner with R&D management experience talked about concrete
innovation types and mostly stressed new and improved processes and products, generally there was a consensus that
for something to be considered an innovation, commercialisation is very important. This is also consistent with the
view of innovation as found in the systematic literature review. Commercialisation that means introduction of products
to markets and the implementation of processes, organisational and marketing methods in actual use is an important
feature of innovation (OECD, 2005).
We noticed in the questionnaire results that only 18% of technical resources considered commercialisation an
important determinant of innovation (refers to Figure 3). According to one of the interviewees, there are many
confounding factors that may a↵ect the successful commercialisation of new products. She argued that with a strong
brand, extensive marketing and large market share an organisation might manage to sell a ‘bad innovation’. On the
contrary, without these resources, an organisation might fail to sell a ‘good innovation’. However, we think that by
di↵erentiating between commercialisation (introduction or implementation) and the performance of innovation one
can resolve this disagreement.
Identifying a comprehensive innovation definition is important as in the questionnaire we found that the lack of a
consistent innovation definition was considered a challenge (Figure 9). This is also consistent with the findings of the
survey where respondents gave di↵erent definitions focusing on di↵erent aspects. One practitioner mentioned how
the perception of innovation di↵ers across the departments of the same organisation.
Therefore, among the definitions given or used by the primary studies in this literature review, we analysed each
definition for coverage against the identified aspects of innovation. Two definitions of innovation by (Crossan and
Apaydin, 2009) and the Organisation for Economic Co-operation and Development (OECD) Oslo Manual (OECD,
2005) stood out. According to (OECD, 2005) ‘An innovation is the implementation of a new or significantly improved
product (goods or service), or process, a new marketing method, or a new organisational method in business practices,
workplace organisation or external relations.’ (Crossan and Apaydin, 2009) modified the definition given by European
Commission’s Green Paper of Innovation to: ‘Innovation is: production or adoption, assimilation, and exploitation of
a value-added novelty in economic and social spheres; renewal and enlargement of products, services, and markets;
development of new methods of production; and establishment of new management systems. It is both a process
and an outcome.’ This paper suggests use of this definition of innovation from (Crossan and Apaydin, 2009) for its
comprehensive coverage of the identified aspects of innovation as found in literature and through the interviews.
19

5.2. Importance of Innovation Measurement
Although there was disagreement on what is currently measured by the organisations and what the interviewees
thought should be measured, there was still an agreement on the importance of innovation measurement. Similarly,
the respondents of the questionnaire, regardless of their roles in the company, agreed that innovation measurement
is important. The increasing number of publications every year on the subject and the emphasis in literature also
indicates that it is considered equally important in academia. However, when we look at the presence of an explicit
strategy for innovation, actual measurement programs and innovation processes (these aspects indirectly indicate the
importance given to the innovation measurement in practice), the reality is very di↵erent. Many respondents indicated
a lack of an explicit strategy (44%) and measurement program (40%) in their organisations. The same scenario
was found in the interviews where all four industry practitioners reported either a lack of innovation strategy and
measurement program or expressed dissatisfaction with the current practice. Similarly, one firm had no innovation
process and the practitioners from the other firms expressed dissatisfaction regarding the process being followed
in their organisations. Thus, we can conclude that although there is a growing consensus on the importance of
innovation measurement in both industry and academia, the practice is still lagging behind. We will next discuss some
of the determinants of innovation in the software industry and then focus on the challenges inhibiting the innovation
measurement practice in Section 5.4.
5.3. Determinants of Innovation in the Software Industry
Most of our literature review results suggest that the existence of an innovation strategy will help organisations
in promoting innovation culture. However, our findings from the interviews and the questionnaire suggest that the
importance of having an innovation strategy in software industry is not significant. Only 42.55% of respondents are
aware of an explicit strategy on innovation in their organisation. This is in line with what one of the interviewee said
about her department that all employees have to be innovative to solve the problem, not because of the strategy, but for
the sake of the project. In short, she knew that her company had a strategy related to innovation, but does not consider
it as the main driver for innovation.
The importance of having an innovation strategy to create a creative climate is also not recognised in the software
industry. One of the interviewee argued that organisational behaviour is not necessarily determined by the top-down
approach but also bottom-up, where the employees actively change their day-to-day behaviour.
5.4. Challenges in Measurement of Innovation in the Software Industry
The following subsections discuss the challenges of innovation measurement in the software industry and the
extent to which this study alleviates them.
5.4.1. Lack of a Consistent Definition of Innovation
The lack of a consistent definition of innovation has been identified in the systematic review, during the interviews
and in the results from the questionnaire. Definitions are fundamental as they a↵ect the measurement program and
help provide a common understanding. This improves communication and understanding between di↵erent stake-
holders involved in innovation measurement. With the comprehensive definition we propose to be used in the software
industry (as it covers all the aspects identified in the literature review and the interviews), we hope this will address
this issue (see Section 5.1).
5.4.2. Lack of Metrics
This issue was identified in the questionnaire executed as part of this study. However, looking at the large number
of metrics found in the systematic literature review we think that there could be three explanations for this di↵erence
between state of art and state of practice:
• Lack of awareness of appropriate metrics: There were no studies, which have aggregated the existing metrics.
This could explain the ignorance about the existence of, quite possibly, appropriate metrics.
• Lack of validation of metrics: Only 37% of the metrics found in literature (as shown in Table 5) have been
subjected to some static validation and there have been no industry trials, i.e. dynamic validation (Gorschek
et al., 2006). This lack of empirical validation may also be a reason why industry is hesitant in adopting the
suggested metrics.
20

• Interpretation: As innovation is not fully understood, it is hard to interpret the values of the metrics. Therefore
organisations often resort to only measuring the performance of innovation (e.g. revenue generated). For exam-
ple, measuring the number of training opportunities that employees have had during the past years, provides an
indication of whether you are investing to promote innovation. However, it does not give information about the
innovation capability by itself.
By conducting a systematic literature review and aggregating the existing measures we attempted to facilitate
tackling this issue. Furthermore, this study investigated and identified a lack of validation and thus identified a
direction for future research.
5.4.3. Lack of Frameworks for Innovation Measurement
According to (Mendonc¸a et al., 1998), measurement frameworks consist of ‘a set of related metrics, data collection
mechanism and data uses inside a software organisation.’ Moreover, measurement should be based on goals and
models (Basili et al., 1994). Since there is no agreement on what innovation is, there is also no agreement on what
metrics should be collected. One interviewee mentioned that there is no consensus as to what constitutes innovation
within the same organisation. Moreover, innovation is a broader term and some aspects might be difficult to measure.
Even if metrics have been defined, the next issue is whether the metrics are reliable and give information about the
innovativeness of the organisation.
Although we found only one innovation measurement model proposed for the software context (Section 4.1.4), the
existence of such frameworks has been recognised in industry. Based on the questionnaire results, we found that some
innovation measurement frameworks have been used in 25.93% of the companies (Section 4.2.5). However, it seems
that there is no standard framework to measure innovation, it is subject to the company and often confidential. We
think that any measurement framework like e.g. GQM, Practical Software Measurement and Balanced Scorecards, can
be used. Some of the aspects missing are a comprehensive innovation definition, a model and metrics for innovation
measurement.
5.4.4. Challenges in Existing Innovation Metrics
There are a number of di↵erent metrics that were found during the systematic literature review. These metrics can
be classified into following major categories: R&D-based measures, revenue-based measures, IPR-based measures,
innovation counts-based measures, process measures, survey measures, investment-based measures. Each of these
proxies of innovation has their relative strengths and weaknesses in terms of coverage of the type of innovation they
measure (Jensen and Webster, 2009). For example, R&D is just one of the inputs to the innovation process and thus
these metrics do not cover the non-R&D inputs like e.g. market analysis and training of employees (Kleinknecht et al.,
2002).
R&D-based measures. There is a great deal of focus in traditional industries on R&D as an input and driver for
innovation. However, dedicated research units is a luxury that is not available to SMEs in particular (Gorschek et al.,
2010a). This was also noticed during the interviews e.g. the ‘scouting teams responsible for looking out for new
technologies and ideas did this as an additional task in their job role’. This is also consistent when one interviewee
related to her experience in a small firm ‘back there we did not think about measurements, whatever it took, we just
got the job done. For us innovation is in day to day activities in everything we do’.
R&D has important implications on the measurements that may be used to assess the inputs to innovation. How-
ever, this measure has not necessarily a relation to innovation output (Evangelista et al., 1998). Since most of the
software firms are SMEs, which as discussed above do not have dedicated research departments, the percentage of
sales spent on R&D, number of employees dedicated to R&D, etc. may not be useful in their context.
Revenue-based measures. Innovation measurement is often restricted to revenues generated by new o↵erings to
the market (James et al., 2008). Apparently, this is a straightforward measure and organisations often tend to use it.
Although it provides a good measure of the market success of a particular innovation, solely relying on this metric
can have negative e↵ects on innovation. If innovation is assessed by generated sales, organisations will develop a
risk-averse culture with a negative e↵ect on creativity. Over time organisations will start emphasising on incremental
innovations and overlook the importance of radical and breakthrough innovations as they have an increased risk of not
bringing monetary returns.
21

One interviewee called it the worst innovation metric, as there are confounding factors in the success of an innova-
tion brought to the market. She emphasised the importance of measuring how well an innovation fulfils the intended
purpose. Another interviewee criticised the sales-based measure, as ‘it is so after the fact’; that is, it provides no
transparency in the internal working of the organisation. Other interviewees called sales a tricky measure and that it
is difficult to track sales to individual innovations. One of the interviewees mentioned that her organisation does not
trace which patents are in which products. Another interviewee added that even if patents are tracked, the life-cycle
of the products and that of innovations makes it extremely complex to track sales. Moreover, an interviewee, ques-
tioned how to define a time period for the return on investment since an innovation may be embedded in a suite of
applications which, in combination with other components, delivers a combined value.
IPR-based measures. A large number of selected primary studies use IPR-based metrics for measuring innovation
output, especially patent counts and citations-based data. However, there are a number of limitations in using patent-
based measures for innovation measurement e.g. patents only cover a certain type of innovation, patents may not
represent innovation at all (preventing competitors from exploiting the opportunity) and the inclination to patent
innovations may vary across size of firms and industry (Kleinknecht et al., 2002; Evangelista et al., 1998).
Furthermore, the use of patent-based measures is controversial in the software industry because of the ongoing
debate about the patentability of software, process of patent protection and cost associated with it. Software giants
(Juden, 2005; Chatterjee, 2004; Aharonian, 1999), however, are in favour of making software patentable for a number
of reasons, while small companies and the open source community (Juden, 2005; McLaughlin, 2004) are against
patent protection for exactly the opposite reasons.
Counts-based measures. It is perhaps the most trivial measure and includes number of ideas generated, number
of ideas financed, number of new products introduced in the market etc. The issue with this measurement is that it
does not say much about the quality of the attribute measured. For example a couple of interviewees in this study
identified that when counting ideas, the important and challenging thing is assessing the quality of the ideas. Similarly,
another interviewee pointed out that when counting number of features, the issue is that not all features are equal or
comparable since the size of features, development e↵ort or delivered value are important aspects as well.
Process, investment-based and survey measures. When talking about innovation measurement, it is important to
measure the output and performance. It is also equally important to assess the environment for innovation and capabil-
ity of an organisation to innovate. One interviewee, said that measuring the innovation climate is more important and
useful than measuring the innovation output itself since that is where the organisation is picking the ideas; however,
her organisation currently did not measure those factors.
Innovation is not just about bringing something new to the market it is also about winning the ‘mind share’ and
being perceived as innovative said one interviewee. Hence, it is important to measure the social capital and image
of the company, which is not captured by sales of the products alone and, furthermore, measure the creativity of the
employees and then analyse if there are avenues and outlets for creativity in the company.
During the interviews and in the systematic literature review we found that the importance of these factors is well
acknowledged but the results from both the interviews and the questionnaire suggest that it is not currently practised
in industry. The reasons may include that such data is often more subjective, hard to gather, evaluate and analyse. As
one interviewee said: ‘innovation measurement is about assessment of innovation capability.’ However, by measuring
resources and other necessary factors we cannot guarantee that we will be innovative. Perhaps this is the reason why
organisations often fail to undertake this type of measurements.
6. Proposed Model
Undoubtedly, innovation is hard to measure (Anon., 2006). A model with various dimensions of innovation is
required to provide a meaningful foundation for measurements (Mairesse and Mohnen, 2002). This study is one such
attempt to develop a generic model that covers the various dimensions of innovation. Since innovation is not fully
understood, this cannot be an exact model that gives the absolute measures of innovation. However, in the absence
of the exact measures, this model will assist organisations to use the available measures to develop insights into their
innovation program.
Using the di↵erent perspectives of innovation and the key aspects for innovation measurement as identified by the
systematic literature review, we developed an innovation measurement model, as shown in Figure 13. This model
22

was further refined after preliminary evaluation by academics and practitioners. From the outset, the model identifies
three main elements of measurement: Innovation capability, innovation output and impact of innovation. Unlike
the current strict reliance on sales as the sole measure for innovation, which may produce negative e↵ects on the
innovation climate of the organisation, this model highlights the opportunity of a more comprehensive approach
towards innovation measurement. Each of these aspects identified in the model can be measured quantitatively (using
both objective and subjective metrics). Metrics for each of these aspects identified from literature have been aggregated
and categorised in this study.
In the following sections, the aspects of innovation that lead to the creation of this model, are discussed.
6.1. Innovation Process Model
Innovation is a non-linear evolving process that needs to be managed and guided (Kuczmarskl, 2001; Pires et al.,
2008). It is how a firm produces innovation output from innovation inputs under contextual determinants (Mairesse
and Mohnen, 2002; Pires et al., 2008; Atrostic, 2008). In this study, we do not propose a new innovation process
model; instead we focus on identifying the key elements for innovation measurement. Hence, our model does not
depict a flow of innovation stages.
The innovation process is divided into three main phases:
• Research phase (Jensen and Webster, 2009) including concept generation, evaluation and feasibility (Nambisan,
2002; Andrew and Sirkin, 2008). The goal of this phase is identification of new opportunities (Kuczmarskl,
2001).
• Development phase (Andrew and Sirkin, 2008; Jensen and Webster, 2009) including project planning, design,
coding and testing (Nambisan, 2002). This phase involves product development based on the innovative concept
(Kuczmarskl, 2001).
• Use in production (Jensen and Webster, 2009) or commercialisation (Nambisan, 2002). This phase includes
activities that ensure introduction of products to the market or implementation of processes within the organi-
sation.
Having these gates, or benchmarking points, help in measurements and do not in anyway suggest or dictate that
the innovation process is as simplistic as to have three sequential steps.
6.2. Key Aspects of Innovation
From the systematic literature review, we found three dimensions of innovation that can be measured: capability
(some studies exchange the term capability and capacity to address the potential ability of the organisation to innovate),
output and success. The determinants and metrics reported in literature were classified according to this dimension.
The classification was done based on the description of the determinants and metrics as provided by the selected
primary studies.
The innovation capability is the willingness and ability to create, adopt and imitate new ideas to implement and
commercialise them in order to satisfy current and future needs (Akman and Yilmaz, 2008; Tajeddini et al., 2006).
It means mobilising organisational knowledge and other competencies necessary for improving existing processes/
products, developing new processes/products and the capacity to respond to accidental technology shifts and unex-
pected opportunities created by the competitors (Akman and Yilmaz, 2008). Innovation capability includes assessment
of innovation inputs, availability and quality of positive influencing factors and innovation process assessment.
Innovation is conceptualised as a change in the input and output of innovation function (Jensen and Webster, 2009;
Aranda and Molina-Fern
´
andez, 2002). When viewed as a process like any other business process, innovation cannot
be solely characterised and captured by the input and output metrics (Banerjee, 1998). Organisations need a range
of measures to evaluate key aspects of innovation process (Andrew and Sirkin, 2008). The key aspects of innovation
processes, which have been identified as important for measurement, are the following:
• Inputs are resources like capital, labour, knowledge and time (Muller et al., 2005; Andrew and Sirkin, 2008;
Acs et al., 2002).
• Innovation process performance means tracking the people and processes acting upon these inputs (Andrew and
Sirkin, 2008).
23

Determinants
Internal factors and External factors
Inputs
x Organizational
resources
x Research &
Development
x Knowledge
x Technology
transfer
x ͙
Capability
Measurement of innovation inputs, determinants and process
Output
Measurement of
innovation output
Performance
Measurement of
innovation impact
Innovation Measurement
to
Activities
produce
result
Output
x Products
x Processes
x Market
x Organization
feedback
Performance
x Direct economic
benefits
x Indirect benefits
x Technological
benefits
Development
Research
Commercialization
effect
effect
Figure 13: A model for innovation measurement.
• Intermediate output as a proxy for economic output using patents acquired (Acs et al., 2002). Patents are good
indicators of new technology creation however they do not measure the economic value of these technologies
(Acs et al., 2002).
• Output is the measure of commercialised ideas resulting from innovation inputs (Acs et al., 2002; Tang and Le,
2007).
• Cash pay back (Andrew and Sirkin, 2008) or innovation performance refers to economic returns of innovation
(Wang and Kafouros, 2009).
• Indirect benefits are not easily quantifiable often involving survey measures, third party rankings and bench-
marking to evaluate organisation impact, brand image (Andrew and Sirkin, 2008) and competitiveness in market
(Chiesa et al., 1998).
There is a general consensus that for changes to be considered as innovation, it must be commercialised (intro-
duced to the market or implemented in the organisation). This confirms the business goals of the company and draws
the line between anything that is new and something a company considers has a value. However, development of
such products, processes and methodologies does provide the organisations a learning opportunity and adds to the
capability of the organisation. This aspect is covered through the feedback loop in the model.
6.3. Purpose of the Model
This model has benefits for both academia and industry. However it is important to identify its purpose and utility.
The model has the following characteristics:
• This model identifies the aspects of innovation that are measurable and thus serves as a starting point for
innovation measurement.
24

• The model serves as a generic map that covers the entire range of activities and aspects putting them all in
perspective.
• It serves as the basis for future research e.g. to study the e↵ect of various determinants on innovation.
• Contrary to the current practice, this model emphasises the need of comprehensive measurements covering a
range of activities (unlike the sole focus by some models on sales-based metrics).
• This model can help di↵erent stakeholders to communicate when talking about innovation. It will provide a
common perspective to everyone involved in innovation measurement.
To understand the use of this model and to put it in the perspective of the overall innovation measurement program,
consider the following example of an innovation measurement program implemented using Goal Question Metric
(GQM) (Basili et al., 1994). We take GQM as an example as it provides a well structured and directed approach for
measurements.
Below, we argue how the proposed model and the contributions in the paper of aggregated determinants and metrcs
will be useful at each level of applying GQM for innovation measurement:
• Step A (Conceptual level (Basili et al., 1994)): Based on the analysis of organisation policy statements, strategic
plans and interviews of relevant stakeholders the purpose of the goals is derived (Basili et al., 1994). The goals
are defined for objects relative to the quality models, measurement perspectives and particular environment.
These objects include processes, products and resources (Basili et al., 1994). The proposed model will be useful
for identifying the objects of goals through its conceptual categorisation of capability (inputs, processes and
determinants), output (product, process, organisation and market) and performance (direct economic, indirect
and technological benefits).
• Step B (Operational level (Basili et al., 1994)): Derive questions that try to characterise the objects of measure-
ment identified in Step A (using the proposed innovation measurement model) with respect to a selected quality
issue and to determine its quality from the selected (Basili et al., 1994).
• Step C (Quantitative level): The model, through the categorisation of metrics as found in literature, assists in
choosing appropriate metrics for answering the questions in Step B. The objective and subjective metrics avail-
able helps in quantitatively answering these questions, which in turn helps to attain the goals of measurement.
6.4. Evaluation results
All of the interviewees expressed positive feedback for the model (Figure 13). However, there were some critical
issues and concerns raised which are discussed below.
One interviewee expressed concerns about how ‘value’ is aligned with the innovation measurement programs
undertaken by a company. She also mentioned the problems with relying too much on sales-based measures. This
was alarming, as one of the purposes of this model was to emphasise the importance of non-sales and output-based
measures. Hence, we redrew the model focusing on capability, output and performance. We emphasised more the
capability aspects of innovation measurement and made it explicit that performance measurement does not solely
comprise of sales measurement. It was shown in the model that performance measurement itself has three aspects of
which direct economic benefits (based on sales data) is one dimension.
One interviewee from academia, found the term ‘front end of innovation’ misleading as it gives impression of a
linear innovation process. Thus, we substituted the term with ‘research’ as suggested by Jensen and Webster (Jensen
and Webster, 2009). Furthermore, the model was modified to give a more neutral perspective of the process of
innovation and just emphasise the three interacting, and mutually constituting, activities of research, development and
commercialisation.
One interviewee found separate feedback and learning loops from output and performance confusing. She sug-
gested having only one feedback loop. Therefore, to make it clearer, the model was modified to have feedback from
output and performance to inputs, determinants and activities.
Two industry interviewees expressed a need for customisation of this generic model to their company-specific
practices and process models for further validation and use. The purpose of this model is to give a high-level overview
of the key aspects of innovation measurement and it can be used with existing measurement frameworks to measure
innovation. This validation, however, was beyond the scope of this study.
25

One interviewee raised an issue that some organisations are not into product development and just build IPRs
and look for potential buyers. Therefore, she argued that the commercialisation step might not be applicable for
such businesses. We think that this business approach can also be accommodated in the proposed model, where the
commercialisation would mean putting the patents to the market (meaning to find a potential buyer for the IPR or to
hunt for any infringements).
7. Validity Threats
Threats to validity related to the results from this study have been identified and are discussed below.
7.1. Identification of Primary Studies
The keyword ‘innovation’ is a very generic term and the concept has been discussed almost in all domains (Sub-
ramanian, 1996), including software engineering. We found that studies of innovation in the software industry were
not restricted to software engineering journals. To minimise the probability of missing relevant studies, we took the
following steps:
• To cover literature from other important fields like management, economics and finance etc., we did not restrict
the literature review to software related databases.
• Only journal publications were included in the study, since publishing in a journal is one, indirect, sign of
quality. This decision was also motivated by the observation that the best papers presented in a conference were
subsequently published in a journal as well.
7.2. Primary Studies Selection and Data Extraction
Since this study involves many fields and hence a large number of potential primary studies, we required an
e↵ective and efficient strategy to minimise researchers’ bias during the studies selection (Kitchenham and Charters,
2007). Therefore, we defined three levels of inclusion/exclusion criteria, which were applied on the results of the
search strings.
The criteria were formulated explicitly and as clearly as possible to avoid misunderstanding between the reviewers.
Based on these criteria, each reviewer selected the relevant studies individually. If one of the reviewers could not
decide whether to accept or reject an article, the decision was made based on mutual agreement. To minimise the
threat of misunderstanding, as suggested by (Brereton et al., 2007), we piloted the selection criteria to reach an
agreement among the reviewers on the criteria. This piloting was aimed to assess whether the reviewers had the same
understanding about the criteria and to check whether the criteria were too strict or too loose. A defined data extraction
strategy was also used after piloting, in order to conduct a structured data extraction.
7.3. Questionnaire
Several threats to the validity of the questionnaire were identified. The instrumentation threat is caused by bad
design that leads to misunderstanding of the topic under discussion (Wohlin et al., 2000). This threat was minimised
by piloting the questionnaire with eight students of a Master in Software Engineering programme and two researchers
from industry with a master’s degree in software engineering. The aim of this pilot was to see whether the questions
were clear and understandable and to measure the time it took to fill out the questionnaire. We noticed that some
of the respondents were reluctant to fill in the questionnaire or left it incomplete. Hence, the layout was refined and
made simpler. Furthermore, depending on the answers of respondents they were only shown relevant questions on
subsequent pages.
Another threat to validity was the threat of not having a representative sample of the target population (Wohlin
et al., 2000). To minimise this threat we contacted respondents with di↵erent roles, from software engineers to
executives, working in micro to large organisations. To help the respondents in filling in the questionnaire and to help
us in managing the completed responses, we developed a web-based questionnaire and gave one week for respondents
to fill in. The invitations were sent through personalised individual emails and postings on alumni mailing lists.
The maturation threat caused by the participants’ change in behaviour, with the progression of time, can distort the
results (Wohlin et al., 2000). This happens if subjects acquire new knowledge during the process or become detached.
26

However, with a short questionnaire which was open only for a week we think we were able to minimise this threat.
As the respondents may develop their understanding, while filling in the questionnaire, di↵erent type of questions e.g.
multiple choices, Likert scale (1 to 5) and open-ended questions were used.
Exposing the hypothesis may a↵ect the way the survey respondents answer the questions (Juristo and Moreno,
2001). Therefore only general information about the intent of the study was given. The questions were also phrased
in a neutral manner and using indirect statements rather than using the keywords to hide the findings of the systematic
review.
7.4. Interviews
Although using interviews are useful to collect historical information, there are threats to validity. The intervie-
wees may not be articulate and may have di↵erent perceptions and hence bias the results (Creswell, 2009). Therefore,
for each organisation, we interviewed two individuals with di↵erent roles. Most of the interviewees were at man-
agerial level. Each interview lasted between 45-60 minutes in their offices. A short interview also minimised the
interviewee maturation threat and boredom e↵ect (Juristo and Moreno, 2001).
To evaluate the framework and gain valuable feedback we conducted face-to-face meetings. This enabled a better
and more e↵ective presentation of ideas and arguments. To reduce the threat to generalisation of results (Wohlin
et al., 2000), we tried to get a mix of representatives from di↵erent domains within the software industry in Sweden
(ranging from software solutions and embedded software in manufacturing, software intensive product development
in telecommunication and representatives from firms with pure software product development). Furthermore, the
academics interviewed in this study also came from both management and computing schools.
8. Conclusions
The purpose of this study was to establish the current practices, mechanisms and challenges of innovation mea-
surement in the software industry. A systematic literature review was performed to establish the state of the art of
innovation measurement. Interviews and an online questionnaire were employed to examine the state of practice of
innovation measurement.
The study found that among major challenges is a lack of a consistent perspective of innovation. This di↵erence
in views a↵ects how innovation measurement initiatives are conceived (what is considered key aspect of innovation)
and executed (which metrics are required to capture a particular aspect). A comprehensive definition of innovation
was identified that addresses various aspects and stakeholders’ concerns that may otherwise lead to above mentioned
problems.
Consistent with the established perspective, this study found that innovation measurement is considered equally
important in both academia and industry. However, there are several shortcomings in the state of practice. In the
software industry, there is a lack of defined innovation process and measurement programs. Similarly, none of the
well-known measurement frameworks are used to measure innovation. To address this deficit, this study identified
metrics and frameworks used in domains other than software development to understand how those fields manage
innovation measurement. To achieve this objective, this study aggregated the available empirical evidence reported
in literature to establish the state of the art in innovation measurement through an extensive literature review. The
outcome of this review contributed to the existing body of knowledge in the form of an innovation measurement
model, enumeration of metrics and their classification based on what aspect of innovation they are used to measure.
Lastly, the issues found in the state of the art can explain the shortcoming of industrial practice with regard to
innovation measurement. This also identifies the future direction for innovation measurement research. These issues
are:
• A lack of comprehensive innovation measurement models covering all the aspects of innovation. Furthermore,
the existing frameworks do not relate which innovation metrics can be used to measure which aspects. This may
well be a reason why measurement frameworks are not being used, as without models and concrete metrics,
launching a measurement program is not possible.
• Availability of little information about the collection, processing and interpretation of metrics proposed in liter-
ature. This limits the uses of metrics in practice.
27

• While a number of metrics have been proposed in literature, only a few of them have been dynamically validated
in industry trials. Most of the metrics are only validated through statistical analysis. This lack of reliable metrics
could be the other main reason why industry claims that they do not have enough metrics. Hence, industrial
trials are imperative to establish the reliability and usability of the metrics.
References
Acs, Z. J., Anselin, L., Varga, A., 2002. Patents and innovation counts as measures of regional production of new knowledge. Research Policy
31 (7), 1069–1085.
Acs, Z. J., Audretsch, D. B., 1988. Innovation in large and small firms: An empirical analysis. American Economic Review 78 (4), 678–.
Adamides, E. D., Karacapilidis, N., 2006. Information technology support for the knowledge and social processes of innovation management.
Technovation 26 (1), 50–59.
Adams, R., Bessant, J., Phelps, R., 2006. Innovation management measurement: A review. International Journal of Management Reviews 8 (1),
21–47.
Aharonian, G., 1999. Does the patent office respect the software community? IEEE Software 16 (4), 87–89.
Aiman-Smith, L., Goodrich, N., Roberts, D., Scinta, J., 2005. Assessing your organization’s potential for value innovation. Research Technology
Management 48 (2), 37–42.
Akman, G., Yilmaz, C., 2008. Innovative capability, innovation strategy and market orientation: An empirical analysis in Turkish software industry.
International Journal of Innovation Management 12 (1), 69–111.
Amara, N., Landry, R., Doloreux, D., 2009. Patterns of innovation in knowledge-intensive business services. Service Industries Journal 29 (4),
407–430.
Andrew, J. P., Sirkin, H. L., 2008. Aligning for innovation. Global Business and Organizational Excellence 27 (6), 21–39.
Anon., 2006. Organizing for innovation: Consistent innovation message. Strategic Direction 22, 34–36.
Aranda, D. A., Molina-Fern
´
andez, L. M., 2002. Determinants of innovation through a knowledge-based theory lens. Industrial Management and
Data Systems 102 (5), 289–296.
Assink, M., 2006. Inhibitors of disruptive innovation capability: A conceptual model. European Journal of Innovation Management 9 (2), 215–233.
Atrostic, B. K., 2008. Measuring U.S. innovative activity: Business data at the U.S. Census Bureau. Journal of Technology Transfer 33 (2),
153–171.
Banerjee, P., 1998. Indicators of ‘innovation as a process’. Scientometrics 43 (3), 331–357.
Baron, R. A., Tang, J., 2009. The role of entrepreneurs in firm-level innovation: Joint e↵ects of positive a↵ect, creativity, and environmental
dynamism. Journal of Business Venturing In Press, Corrected Proof, –.
Basili, V., Caldiera, G., Rombach, H., 1994. The goal question metric approach. Encyclopedia of Software Engineering 1, 528–532.
Becheikh, N., Landry, R., Amara, N., 2006. Lessons from innovation empirical studies in the manufacturing sector: A systematic review of the
literature from 1993-2003. Technovation 26 (5–6), 644–664.
Berg, P., Pihlajamaa, J., Poskela, J., Lempiala, T., Haner, U., Mabogunje, A., 2008. Measurement of the innovation front end: Viewpoint of process,
social environment and physical environment. In: PICMET ’08 - 2008 Portland International Conference on Management of Engineering
Technology. IEEE, Piscataway, NJ, USA, pp. 1112–1120.
Berger, M., Revilla Diez, J., 2006. Technological capabilities and innovation in Southeast Asia: Results from innovation surveys in Singapore,
Penang and Bangkok. Science, Technology and Society 11 (1), 109–148.
Beugelsdijk, S., 2008. Strategic human resource practices and product innovation. Organization Studies 29 (6), 821–847.
bin Ali, N., Edison, H., Torkar, R., 2011. Supplementary material for the publication.
URL http://www.bth.se/com/nal.nsf/pages/supplements
Brereton, P., Kitchenham, B. A., Budgen, D., Turner, M., Khalil, M., 2007. Lessons from applying the systematic literature review process within
the software engineering domain. Journal of Systems and Software 80 (4), 571–583.
Brooks, F. P., April 1987. No silver bullet: Essence and accidents of software engineering. Computer 20 (4), 10–19.
Brown, S., Eisenhardt, K., 1995. Product development: Past research, present findings, and future directions. The Academy of Management Review
20 (2), 343–378.
Buganza, T., Verganti, R., 2006. Life-cycle flexibility: How to measure and improve the innovative capability in turbulent environments. Journal of
Product Innovation Management 23 (5), 393–407.
Byrne, G., Lubowe, D., Blitz, A., 2007. Using a lean Six Sigma approach to drive innovation. Strategy and Leadership 35 (2), 5–10.
Caloghirou, Y., Kastelli, I., Tsakanikas, A., 2004. Internal capabilities and external knowledge sources: Complements or substitutes for innovative
performance? Technovation 24 (1), 29–39.
Capaldo, G., Iandoli, l., Ra↵a, M., Zollo, G., 2003. The evaluation of innovation capabilities in small software firms: A methodological approach.
Small Business Economics 21 (4), 343–354.
Carmona-Lavado, A., Cuevas-Rodr
´
ıguez, G., Cabello-Medina, C., 2010. Social and organizational capital: Building the context for innovation.
Industrial Marketing Management 39 (4), 681–690.
Chandy, R. K., Tellis, G. J., 1998. Organizing for radical product innovation: The overlooked role of willingness to cannibalize. Journal of
Marketing Research 35 (4), 474–487.
Chatterjee, A. C., 2004. Europe struggles over software patents. IEEE Spectrum 41 (9), 61–63.
Cheng, C. J., Shiu, E. C. C., 2008. Re-innovation: The construct, measurement, and validation. Technovation 28 (10), 658–666.
Chiesa, V., Coughlan, P., Voss, C. A., 1998. Development of a technical innovation audit. IEEE Engineering Management Review 26 (2), 64–91.
Cho, H. J., Pucik, V., 2005. Relationship between innovativeness, quality, growth, profitability, and market value. Strategic Management Journal
26 (6), 555–575.
28

Cormican, K., O’Sullivan, D., 2004. Auditing best practice for e↵ective product innovation management. Technovation 24 (10), 819–829.
Creswell, J. W., 2009. Research design: Qualitative, quantitative, and mixed methods approaches, 3rd Edition. Sage Publication, Inc.
Crossan, M. M., Apaydin, M., dec 2009. A multi-dimensional framework of organizational innovation: A systematic review of the literature.
Journal of Management Studies, to be published.
Damanpor, F., 1996. Bureaucracy and innovation revisited: E↵ects of contingency factors, industrial sectors, and innovation characteristics. The
Journal of High Technology Management Research 7 (2), 149–173.
Damanpour, F., 1992. Organizational size and innovation. Organization Studies 13 (3), 375–402.
de Jong, J. P. J., Kemp, R., 2003. Determinants of co-workers’ innovative behaviour: An investigation into knowledge intensive services. Interna-
tional Journal of Innovation Management 7 (2), 189.
De Jong, P. J., Vermeulen, P. A. M., 2006. Determinants of product innovation in small firms: A comparison across industries. International Small
Business Journal 24 (6), 587–609.
Desyllas, P., Hughes, A., 2009. The revealed preferences of high technology acquirers: An analysis of the innovation characteristics of their targets.
Cambridge Journal of Economics 33 (6), 1089–1111.
Dibrell, C., Davis, P. S., Craig, J., 2008. Fueling innovation through information technology in SMEs. Journal of Small Business Management
46 (2), 203–218.
Dodgson, M., Hinze, S., 2000. Indicators used to measure the innovation process: Defects and possible remedies. Research Evaluation 9 (2),
101–114.
Ebert, C., Dumke, R., 2007. Software measurement: Establish, extract, evaluate, execute. Springer-Verlag Berlin Heidelberg.
EC, 2005. The new SME definition—User guide and model declaration. Also available as http://ec.europa.eu/enterprise/policies/
sme/files/sme_definition/sme_user_guide_en.pdf.
Evangelista, R., Sandven, T., Sirilli, G., Smith, K., 1998. Measuring innovation in European industry. International Journal of the Economics of
Business 5 (3), 311–333.
Fagan, M. H., 2004. The influence of creative style and climate on software development team creativity: An exporatory study. Journal of Computer
Information Systems 44 (3), 73–80.
Freeman, J., Engel, J. S., 2007. Models of innovation: Startups and mature corporations. California Management Review 50 (1), 94–119.
Frenkel, A., Shefer, D., Roper, S., 2003. Public policy, locational choice and the innovation capability of high-tech firms: A comparison between
Israel and Ireland. Papers in Regional Science 82, 203–221.
Fruhling, A. L., Keng, S., 2007. Assessing organizational innovation capability and its e↵ect on e-commerce initiatives. Journal of Computer
Information Systems 48 (1), 133–145.
Furman, J. L., Porter, M. E., Stern, S., 2002. The determinants of national innovative capacity. Research Policy 31 (6), 899–933.
Garcia, R., Calantone, R., 2002. A critical look at technological innovation typology and innovativeness terminology: A literature review. Journal
of Product Innovation Management 19 (2), 110–132.
Gaynor, G. H., 2001. Innovator: What does it take to be one? IEEE Antennas and Propagation Magazine 43 (3), 126–130.
Geiger, S. W., Cashen, L., 2002. A multidimensional examination of slack and its impact on innovation. Journal of Managerial Issues 14 (1), 68–.
Gorschek, T., Fricker, S., Palm, K., Kunsman, S., 2010a. A lightweight innovation process for software-intensive product development. IEEE
Software 27 (1), 37–45.
Gorschek, T., Fricker, S., Palm, K., Kunsman, S., 2010b. A lightweight innovation process for software-intensive product development. IEEE
Software 27, 37–45.
Gorschek, T., Garre, P., Larsson, S., Wohlin, C., 2006. A model for technology transfer in practice. IEEE Software 23 (6), 88–95.
Gumusluo
ˇ
glu, L., Ilsev, A., 2009. Transformational leadership and organizational innovation: The roles of internal and external support for inno-
vation. Journal of Product Innovation Management 26 (3), 264–277.
Hage, J. T., 1999. Organizational innovation and organizational change. Annual Review of Sociology 25 (1), 597–.
Haner, U. E., 2002. Innovation quality—A conceptual framework. International Journal of Production Economics 80 (1), 31–37.
Herrmann, A., Tomczak, T., Befurt, R., 2006. Determinants of radical product innovations. European Journal of Innovation Management 9 (1),
20–43.
James, P. A., Knut, H., David, C. M., Harold, L. S., Andrew, T., August 2008. A BCG senior management survey—Measuring innovation 2008:
Squandered opportunities.
Jensen, P. H., Webster, E., 2009. Another look at the relationship between innovation proxies. Australian Economic Papers 48 (3), 252–269.
Juden, S., 2005. Can software be patented? Computing & Control Engineering Journal 16 (4), 16–17.
Juristo, N., Moreno, A., 2001. Basics of Software Engineering Experimentation. Kluwer.
Kasunic, M., 2005. Designing an e↵ective survey. Tech. rep., Carnegie Mellon—Software Engineering Institute.
Kitchenham, B., Charters, S., 2007. Guidelines for performing systematic literature reviews in software engineering. Tech. Rep. EBSE 2007-001,
Keele University and Durham University Joint Report.
Kivim
¨
aki, M., L
¨
ansisalmi, H., Elovainio, M., Heikkil
¨
a, A., Lindstr
¨
om, K., Harisalo, R., Sipil
¨
a, K., Puolimatka, L., 2000. Communication as a
determinant of organizational innovation. R&D Management 30 (1), 33–42.
Kleinknecht, A., Van Montfort, K., Brouwer, E., 2002. The non-trivial choice between innovation indicators. Economics of Innovation & New
Technology 11 (2), 109–.
Kuczmarskl, T. D., 2001. Five fatal flaws of innovation metrics. Marketing Management 10 (1), 34–.
Kumar, V., Persaud, A. N. S., Kumar, U., 1996. To terminate or not an ongoing R&D project: A managerial dilemma. IEEE Transactions on
Engineering Management 43 (3), 273–284.
Lin, H. F., 2007. Knowledge sharing and firm innovation capability: An empirical study. International Journal of Manpower 28 (3–4), 315–332.
Linton, J., 2007. What is innovation, anyway? Circuits Assembly 18 (4), 18–20.
Mairesse, J., Mohnen, P., 2002. Accounting for innovation and measuring innovativeness: An illustrative framework and an application. American
Economic Review 92 (2), 226–230.
Maravelakis, E., Bilalis, N., Antoniadis, A., Jones, K. A., Moustakis, V., 2006. Measuring and benchmarking the innovativeness of SMEs: A
29

three-dimensional fuzzy logic approach. Production Planning & Control 17 (3), 283–292.
Mathiassen, L., Pourkomeylian, P., 2003. Managing knowledge in a software organization. Journal of Knowledge Management 7 (2), 63–80.
McLaughlin, L., 2004. European Union struggles with new rules for software patents. IEEE Software 21 (5), 101–104.
McMillan, C., 2010. Five competitive forces of e↵ective leadership and innovation. Journal of Business Strategy 31 (1), 11–22.
Mendonc¸a, M. G., Basili, V. R., Bhandari, I. S., Dawson, J., 1998. An approach to improving existing measurement frameworks. IBM Systems
Journal 37 (4), 484–501.
Misra, S. C., Kumar, V., Kumar, U., 2005. Goal-driven measurement framework for software innovation processes. In: Proceedings of the 2005
International Conference on Software Engineering Research and Practice, SERP’05. Vol. 2. CSREA Press, Las Vegas, NV, United states, pp.
710–716.
Muller, A., V
¨
alikangas, L., Merlyn, P., 2005. Metrics for innovation: Guidelines for developing a customized suite of innovation metrics. IEEE
Engineering Management Review 33 (4), 66–72.
Nambisan, S., 2002. Software firm evolution and innovation-orientation. Journal of Engineering and Technology Management 19 (2), 141–165.
Narayana, M. G. P. L., 2005. A framework approach to measure innovation maturity. In: Proceedings - Engineering Management Conference.
Vol. 2. pp. 765–769.
Nirjar, A., 2008. Innovations and evolution of software SMEs: Exploring the trajectories for sustainable growth. Vision 12 (2), 47–57.
OECD, 2005. Oslo Manual—Guidelines for collecting and interpreting innovation data, 3rd Edition. OECD Publishing.
Oke, A., Munshi, N., Walumbwa, F. O., 2009. The influence of leadership on innovation processes and activities. Organizational Dynamics 38 (1),
64–72.
Orlando, M. J., Verba, M., 2005. Do only big cities innovate? Technological maturity and the location of innovation. Economic Review (01612387)
90 (2), 31–57.
Paladino, A., 2007. Investigating the drivers of innovation and new product sucess: A comparison of strategic orientations. Journal of Product
Innovation Management 24 (6), 534–553.
Palmberg, C., 2004. The sources of innovations—Looking beyond technological opportunities. Economics of Innovation & New Technology 13 (2),
1–.
Parashar, M., Sunil Kumar, S., 2005. Innovation capability. IIMB Management Review 17 (4), 115–123.
Pires, C. P., Sarkar, S., Carvalho, L., 2008. Innovation in services—How di↵erent from manufacturing? Service Industries Journal 28 (10), 1339–
1356.
Rejeb, H. B., Morel-Guimar
˜
aes, L., Boly, V., Assi
´
elou, N. G., 2008. Measuring innovation best practices: Improvement of an innovation index
integrating threshold and synergy e↵ects. Technovation 28 (12), 838–854.
Romijn, H., Albaladejo, M., 2002. Determinants of innovation capability in small electronics and software firms in Southeast England. Journal of
Research Policy 31 (7), 1053–1067.
Russell, R. D., 1990. Innovation in organizations: Toward an integrated model. Review of Business 12 (2), 19.
Schumpeter, J. A., 1949. The theory of economic development: An inquiry into profits, capital, credit, interest and the business cycle; translated
from the German by Redvers Opie. Harvard University Press, Cambridge, MA.
Shapiro, A. R., 2006. Measuring innovation: Beyond revenue from new products. Research Technology Management 49 (6), 42–51.
Singh, A., Singh, V., 2009. Innovation in services: Design and management. African Journal of Business Management 3 (12), 871–878.
Subramanian, A., 1996. Innovativeness: Redefining the concept. Journal of Engineering and Technology Management 13 (3–4), 223–243.
Sundstr
¨
om, P., Zika-Viktorsson, A., 2009. Organizing for innovation in a product development project: Combining innovative and result oriented
ways of working—A case study. International Journal of Project Management 27 (8), 745–753.
Tajeddini, K., Trueman, M., Larsen, G., 2006. Examining the e↵ect of market orientation on innovativeness. Journal of Marketing Management
22 (5–6), 529–551.
Tang, J., Le, C. D., 2007. Multidimensional innovation and productivity. Economics of Innovation & New Technology 16 (7), 501–516.
Voss, C. A., 1985. Determinants of success in the development of applications software. Journal of Product Innovation Management 2 (2), 122–129.
Wakasugi, R., Koyata, F., 1997. R&d, firm size and innovation outputs: Are Japanese firms efficient in product development? Journal of Product
Innovation Management 14 (5), 383–392, a Special Issue on Product Development in Japanese Companies.
Wang, C., Kafouros, M. I., 2009. What factors determine innovation performance in emerging economies? Evidence from China. International
Business Review 18 (6), 606–616.
Wang, C.-H., Lu, I.-Y., Chen, C.-B., 2008. Evaluating firm technological innovation capability under uncertainty. Technovation 28 (6), 349–363.
Wohlin, C., Runeson, P., H
¨
ost, M., Ohlsson, M. C., Regnel, B., Wesslen, A., 2000. Experimentation in Software Engineering: An Introduction.
Kluwer Acedemic Publisher, Massachusetts, USA.
Yam, R. C. M., Guan, J. C., Pun, K. F., Tang, E. P. Y., 2004. An audit of technological innovation capabilities in chinese firms: Some empirical
findings in Beijing, China. Research Policy 33 (8), 1123–1140.
30