Content uploaded by Johann Kranz
Author content
All content in this area was uploaded by Johann Kranz on Oct 27, 2015
Content may be subject to copyright.
A Framework for Rigorously Identifying Research Gaps in
Qualitative Literature Reviews
Journal:
International Conference on Information Systems 2015
Manuscript ID:
ICIS-0072-2015.R1
Track:
18. Methodological and Philosophical Foundations of IS
Keywords:
Research methods/methodology, Literature review, Literature analysis,
Research gap
Abstract:
Identifying research gaps is a fundamental goal of literature reviewing.
While it is widely acknowledged that literature reviews should identify
research gaps, there are no methodological guidelines for how to identify
research gaps in qualitative literature reviews ensuring rigor and
replicability. Our study addresses this gap and proposes a framework that
should help scholars in this endeavor without stifling creativity. To develop
the framework we thoroughly analyze the state-of-the-art procedure of
identifying research gaps in 40 recent literature reviews using a grounded
theory approach. Based on the data, we subsequently derive a framework
for identifying research gaps in qualitative literature reviews and
demonstrate its application with an example. Our results provide a modus
operandi for identifying research gaps, thus enabling scholars to conduct
literature reviews more rigorously, effectively, and efficiently in the future.
A Framework for Rigorously Identifying Research Gaps
Thirty Sixth International Conference on Information Systems, Fort Worth 2015 1
A Framework for Rigorously Identifying
Research Gaps in Qualitative Literature
Reviews
Completed Research Paper
Christoph Müller-Bloch
Copenhagen Business School
Howitzvej 60
2000 Frederiksberg
Denmark
chmu14ab@student.cbs.dk
Johann Kranz
University of Göttingen
Platz der Göttinger Sieben 5
37073 Göttingen
Germany
jkranz@uni-goettingen.de
Abstract
Identifying research gaps is a fundamental goal of literature reviewing. While it is
widely acknowledged that literature reviews should identify research gaps, there are no
methodological guidelines for how to identify research gaps in qualitative literature
reviews ensuring rigor and replicability. Our study addresses this gap and proposes a
framework that should help scholars in this endeavor without stifling creativity. To
develop the framework we thoroughly analyze the state-of-the-art procedure of
identifying research gaps in 40 recent literature reviews using a grounded theory
approach. Based on the data, we subsequently derive a framework for identifying
research gaps in qualitative literature reviews and demonstrate its application with an
example. Our results provide a modus operandi for identifying research gaps, thus
enabling scholars to conduct literature reviews more rigorously, effectively, and
efficiently in the future.
Keywords: Research methods/methodology, Literature review, Literature analysis, Research
gap
Page 1 of 19
A Framework for Rigorously Identifying Research Gaps
Thirty Sixth International Conference on Information Systems, Fort Worth 2015 2
Introduction
A literature review “is an essential feature of any academic project” (Webster and Watson 2002, p. xiii).
Most literature reviews in information systems (IS) research use the framework proposed by Webster and
Watson's (2002). However, their framework includes little information about how to rigorously identify
research gaps when conducting literature reviews. As Webster and Watson note, “a review should identify
critical knowledge gaps and thus motivate researchers to close this breach” (Webster and Watson 2002, p.
xix). However, they do not propose a modus operandi for ensuring rigor and replicability in that process.
However, for conducting literature reviews rigor (e.g., Cooper 1998; Levy and Ellis 2006) and
systematicity (Rowe 2014) are essential as they enhance the quality of the review since rigorous and
systematic reviews are more useful and replicable (Wolfswinkel et al. 2013; Rowe, 2014).
Based on the finding that “the identification of such [research] gaps has not been completed in a
systematic way” (Robinson et al. 2011, p. 1325), Robinson et al. (2011) developed a framework for
rigorously identifying research gaps. In addition, they point out that as researchers aim for conducting
relevant future research, it is important to provide them with research gaps that are rigorously identified
and characterized. However, their framework solely focuses on quantitative literature reviews (meta-
analyses) in the health care domain. Hence, there is a need for a framework that helps scholars to identify
research gaps in qualitative literature reviews whose objective is to summarize extant theory to identify
gaps in theory or research. The development of such a framework needs to be informed by justificatory
knowledge. In order to gather this knowledge, this paper analyzes how research gaps are identified in
current literature reviews using grounded theory. We assume that these literature reviews contain implicit
or explicit knowledge about the process of identifying research gaps. Subsequently, these findings will be
used to build a comprehensive framework for identifying research gaps in literature reviews. Therefore,
this paper aims at answering two research questions:
Research question 1: How are research gaps being identified in state-of-the-art literature reviews?
Research question 2: How can research gaps be rigorously identified from literature reviews?
The second research question leads to the main contribution of our article, which is a framework that
increases researchers’ ability to rigorously identify research gaps in literature reviews. We seek to improve
researchers’ effectiveness in conducting literature reviews by facilitating a literature synthesis that is
systematically targeted on identifying research gaps. To demonstrate the application and usefulness of the
framework, we will provide a brief example about how to employ the framework.
This paper focuses on literature reviews in the IS domain. Although this particular field of research has
distinctive features, such as its interdisciplinary character (Levy and Ellis 2006), the results of this paper
might as well be applicable in other domains, especially in the social sciences.
The remainder of this paper is organized as follows. In the next section we outline the theoretical
background. We then explicate our research approach. In the next section we present the findings on
which the framework for identifying research gaps is developed and applied. The paper concludes with a
discussion and implications for researchers.
Theoretical Background
Goals and Process of Literature Reviews
Hart (2009, p.13) defines a literature review as “the selection of available documents (both published and
unpublished) on the topic, which contain information, ideas, data and evidence written from a particular
standpoint to fulfill certain aims or express certain views on the nature of the topic and how it is to be
investigated, and the effective evaluation of these documents in relation to the research being proposed”.
Yet, this definition does not refer to the importance of rigor for thorough literature reviews. As Cooper
(1998, p. 3) notes, “research synthesists must be required to meet the same rigorous methodological
standards that are applied to primary researchers”. In this regard, literature reviews also refer to
systematicity, as systematicity implies reproducibility (Rowe 2014). Furthermore, a literature review
should not only examine past research, but also identify research gaps (Hart 2009; Rowe 2014; Webster
Page 2 of 19
A Framework for Rigorously Identifying Research Gaps
Thirty Sixth International Conference on Information Systems, Fort Worth 2015 3
and Watson 2002) that can be derived from the synthesis of existing research (Cooper 1998). A synthesis
should provide guidance on where future research would be most effective (Eagly and Wood 1994).
Regarding literature reviews in the field of IS, Webster and Watson’s (2002) seminal paper is widely
regarded as the standard approach (Wolfswinkel et al. 2013). The paper’s main contribution are
guidelines that assist scholars in conducting literature reviews in the domain of IS. According to vom
Brocke et al. (2009), a literature review in the field of IS consists of five phases (see Figure 1). Initially, the
scope of the review needs to be determined. Second, the topic should be conceptualized. This also includes
providing definitions of the key terms (Zorn and Campbell 2006). Third, a literature search needs to be
conducted. In the fourth step, the literature is analyzed and synthesized. Finally, a research agenda is
derived from the synthesis (vom Brocke et al. 2009).
Figure 1. Framework for literature reviewing (vom
Brocke et al. 2009, p. 8)
Thus, identifying research gaps forms part of the last step (phase V) of von Brocke et al.’s framework. It is
important to note that the process of identifying research gaps is different from the process of literature
analysis and synthesis (phase IV). As noted above, research gaps are derived from the synthesis (Cooper
1998). In contrast to the process of identifying research gaps, the process of literature analysis and
synthesis has been more thoroughly investigated and methods have been identified (e.g., Webster and
Watson 2002; Wolfswinkel et al. 2013).
Distinguishing Research Gaps, Research Problems, and Research Agenda
Robinson et al. (2011, p. 1325) define that a research gap arises „when the ability of the systematic
reviewer to draw conclusions is limited“. Nevertheless, a research gap also holds a function as a starting
point for research. While Robinson et al. (2011) emphasize that research gaps represent an output (of
literature reviews), we also perceive them as an input as they can motivate further research. The term
research problem might occasionally be used as a synonym for research gaps. However, it focuses on the
function as input for research, since a research problem is a problem statement that is resolved by means
of research (Jacobs 2011). A problem statement represents “a gap in sets of information that, when
examined carefully, results in a call for action or resolution” (Jacobs 2011, p. 127), and “research seeks to
resolve the disparate sets of information through the generation of new knowledge and the introduction of
theory” (Jacobs 2011, p. 128). As Jacobs notes, research problems must be derived, e.g., by means of
literature reviews. Therefore, research problems can also be conceived as an output.
Page 3 of 19
A Framework for Rigorously Identifying Research Gaps
Thirty Sixth International Conference on Information Systems, Fort Worth 2015 4
We integrate the definitions by Robinson et al. (2011) and Jacobs (2011) for our paper. While the
definition by Robinson et al. (2011) focuses on research gaps as a result of the process of reviewing
literature, Jacobs (2011) emphasizes the formation of research problems and how these problems might
be resolved. Hence, a research gap arises when there is a gap in sets of information that is derived from a
literature synthesis and requires further research to be resolved. It might be argued that it is possible to
derive research gaps from other sources than literature syntheses. Yet, the definition above does neither
object nor claim exclusiveness. It merely states one major possibility of how a research gap may arise.
A research agenda is a set of questions for further research (vom Brocke et al. 2009). However, the
literature does not provide information whether a set of research gaps may therefore be called research
agenda. Whereas research gaps can directly lead to questions for further research, this does not
necessarily mean that all questions for further research are derived from research gaps. Thus, it can be
assumed that research gaps are a part of research agendas, but not necessarily exclusively.
Classifying and Identifying Research Gaps
Characterizing research gaps deepens the understanding of how research gaps may be constituted and
may thus help to identify research gaps in literature reviews.
Jacobs (2011) identifies six kinds of research problems. While research problems are not necessarily
research gaps, they might be synonymous with research gaps in this case, as most researchers do not
distinguish between the two terms. Jacobs identifies six forms of research problems: Provocative
exception, contradictory evidence, knowledge void, action-knowledge conflict, methodological conflict,
and theoretical conflict.
A provocative exception arises if a new research finding contradicts widely accepted conclusions. Jacobs
notes that the provocative exception usually does not stand out prominently in the literature. To uncover
these exceptions, it is necessary to carefully analyze and scrutinize even subtle discrepancies.
Contradictory evidence is related to the provocative exception. It occurs if results from studies allow for
conclusions in their own right, but are contradictory when examined from a more abstract point of view.
The identification of contradictory evidence starts with analyzing each research stream. Subsequently, the
results from these analyses need to be synthesized in order to reveal contradictory evidence. Whereas
Jacobs doubts that there are many knowledge voids, he presents two settings where knowledge voids
might occur. First, knowledge may not exist in the actual field of research but in a related research
domain. In this case, it may be necessary for scholars to refer to theories and literature from related
research domains. Second, it might be the case that results of a study differ from what was expected. This
kind of discrepancy can motivate new research in this direction. An action-knowledge conflict arises when
the actual behavior of professionals is different from their advocated behavior. In this case, research could
seek to determine the scope of the conflict and to uncover the reasons for its existence. It is also possible
that a methodological conflict occurs due to the influence of methodology on research results. Jacobs
notes that it might be useful to vary methods, especially if certain research topics have been mainly
explored using a certain method. Finally, if one phenomenon is being explained through various
theoretical models, there might be a theoretical conflict. Scholars could examine whether one of those
theories is superior regarding its explanatory force.
To the best of our knowledge, only one approach exists for identifying research gaps in qualitative
literature reviews. This approach is called chart method, also often referred to as concept matrix (see
Table 1). The underlying principle is to chart each source “according to predetermined categories” (Imel
2011, p. 152). Webster and Watson (2002) recommend establishing categories so that it adds value to the
review. For instance, categories could feature “types of variables examined, level of analysis, gaps in the
literature, or other important theoretical issues” (Webster and Watson 2002, p. xviii).
Page 4 of 19
A Framework for Rigorously Identifying Research Gaps
Thirty Sixth International Conference on Information Systems, Fort Worth 2015 5
Sources
Concept
Concept 1
Concept 2
Concept 3
...
Source 1
X
Source 2
X
X
Source 3
X
...
Table 1. Concept matrix (based on Webster and Watson 2002, p. xvii)
However, neither Webster and Watson (2002) nor Imel (2011) provide insight into the concrete process of
identifying research gaps by means of the chart method. Vom Brocke et al. (2009) offer a rough idea on
how the process might work. During the fourth phase of their framework, prior literature is analyzed and
synthesized. For this stage, vom Brocke et al. (2009) propose employing a concept matrix. In the last
phase, the synthesis is supposed to result into a research agenda. Referring to the matrix, they note that
“certain fields […], which remain ‘blank’ during a literature study, often highlight research areas that are
significantly under-researched” (vom Brocke et al. 2009, p. 10). Thus, the framework leaves scholars with
a rather incomplete idea on how the identification of research gaps might be executed.
Methodology
Research Approach
The goal of this paper is to propose a framework to identify research gaps when conducting literature
reviews. However, as outlined above, the explicit knowledge in the literature about how research gaps are
being identified is limited. Thus, additional information is required to enable the construction of the
framework. We therefore review literature reviews to examine how research gaps are identified. The
underlying rationale is that, while the research on how to identify research gaps is limited, literature
reviews might contain implicit or explicit knowledge on how researchers identified research gaps.
Following Wolfswinkel et al. (2013), this paper employs grounded theory as a method.
The underlying principle of grounded theory is to generate theory from data (Glaser and Strauss 2008).
Glaser and Strauss (2008, p. 31) contend that “grounded theory can be presented either as a well-codified
set of propositions or in a running theoretical discussion, using conceptual categories and their
properties”. For this paper, the latter approach will be used for presenting the grounded theory since we
conduct a literature review before coding. This is necessary to explore the topic and to detect the potential
lack of research on the topic. As McGhee et al. (2007, pp. 339) put it: “Grounded theory is an appropriate
approach when there is little extant knowledge of the issue, but how can this paucity of knowledge be
ascertained unless an initial review of literature is undertaken?”
Our research approach resembles that of vom Brocke et al. (2009). They also analyze literature reviews to
enhance the methodological rigor of future reviews. However, their focus is on the literature search
process, not on the process of identifying research gaps.
Literature Search
Webster and Watson (2002) suggest starting the search for relevant literature in leading journals.
Therefore, we analyzed the AIS “basket” of 81. We did not engage in forward and backward search
(Webster and Watson 2002) since our goal is not to identify articles about a certain topic, but articles with
a certain methodology. As literature reviews characteristically investigate the literature on a particular
topic (but not in terms of methodology), forward and backward search would be of little value for our
purpose. The terms that we used in the search process were keywords that we found to be typical of
literature reviews. We searched for papers including the terms literature and review, or literature and
analysis, or research and roadmap, or research and agenda in the abstract. We limited the search to
1 www.aisnet.org/?SeniorScholarBasket
Page 5 of 19
A Framework for Rigorously Identifying Research Gaps
Thirty Sixth International Conference on Information Systems, Fort Worth 2015 6
articles published from 2000 on to emphasize the state-of-the-art character of the review. Thus,
forthcoming articles are also included. Overall, 198 articles were identified during the literature search
(see Table 2).
Journal
Database
Identified
Selected
European Journal of Information
Systems (EJIS)
Palgrave Macmillan
24
4
Information Systems Journal (ISJ)
Wiley Online
Library
29
2
Information Systems Research (ISR)
INFORMS
15
5
Journal of the Association for
Information Systems (JAIS)
AISeL
16
3
Journal of Information Technology
(JIT)
Palgrave Macmillan
47
10
Journal of Management Information
Systems (JMIS)
Business Source
Premier
11
1
Journal of Strategic Information
Systems (JSIS)
ScienceDirect
30
5
MIS Quarterly (MISQ)
Business Source
Premier
26
10
Total
198
40
Table 2. Literature search and selection
Literature Selection
As we defined the search criteria rather broadly, relevant articles from the initial sample of 198 articles
had to be identified. We deem an article as relevant, if its major goal is to systematically review the prior
literature. Overall, we identified 40 articles as ‘pure’ literature reviews which were selected for further
analysis (see Table 2 and Table 3).
The Coding Process
After identifying all relevant excerpts from the data (i.e., the literature reviews), two different coding
approaches were employed. This was expedient as Jacobs (2011) already proposed a classification on how
research gaps may be characterized. Hence, in our case the theory did not directly emerge from the data.
Instead, we aim at refining existing theory using our sample of literature reviews. For this purpose, a
coding approach called elaborative coding was employed, which is defined as “the process of analyzing
textual data in order to develop theory further” (Auerbach and Silverstein 2003, p. 104).
Apart from that, we employed open coding and axial coding using the software tool NVivo 10. While
“open coding is the analytical process of generating higher-abstraction level type categories from sets of
concepts/variables” (Wolfswinkel et al. 2013, p. 51), axial coding is “the act of relating concepts/categories
to each other” (Corbin and Strauss 2008, p. 198).
Although our review cannot claim to be exhaustive, we could well reach a point of theoretical saturation
with the selected articles. Theoretical saturation is reached, when no new categories and properties
emerge from the data (Glaser and Strauss 2008). During the process of coding, the sample appeared to be
sufficient for theoretical saturation. Thus, no further data was required.
Findings
A total of 555 research gaps were identified from the selected literature reviews (see Table 3), equaling an
average of approximately 14 gaps per paper. However, the amount of research gaps that each literature
review contains varies considerably. For instance, while the literature review by Mbarika et al. (2005)
features 49 research gaps, the literature review by Dhillon and Backhouse (2001) contains only 2 research
gaps.
Page 6 of 19
A Framework for Rigorously Identifying Research Gaps
Thirty Sixth International Conference on Information Systems, Fort Worth 2015 7
Author(s)
Source
Title
#
Research
gaps
Aksulu and Wade
(2010)
JAIS
A Comprehensive Review and Synthesis of Open Source Research
8
Arnott and Pervan
(2005)
JIT
A critical analysis of decision support systems research
4
Gefen et al. (2008)
JMIS
A Research Agenda for Trust in Online Environments
15
D’Arcy and Herath
(2011)
EJIS
A review and analysis of deterrence theory in the IS security literature: making
sense of the disparate findings
16
Leidner and
Kayworth (2006)
MISQ
A Review of Culture in Information Systems Research: Toward a Theory of
Information Technology Culture Conflict
13
Lacity et al. (2010)
JIT
A review of the IT outsourcing empirical literature and future research directions
8
Jeyaraj, et al.
(2006)
JIT
A review of the predictors, linkages, and biases in IT innovation adoption research
6
Conboy (2009)
ISR
Agility from First Principles: Reconstructing the Concept of Agility in Information
Systems Development
4
Lacity et al. (2011)
JIT
Business process outsourcing studies: a critical review and research directions
14
von Krogh et al.
(2012)
MISQ
Carrots and Rainbows: Motivation and Social Practice in Open Source Software
Development
19
Wareham et al.
(2005)
JIT
Critical themes in electronic commerce research: a meta-analysis
9
Dhillon and
Backhouse (2001)
ISJ
Current directions in IS security research: towards socio-organizational
perspectives
2
Schneider and
Sunyaev (2014)
JIT
Determinant factors of cloud-sourcing decisions: reflecting on the IT outsourcing
literature in the era of cloud computing
18
Tilson et al. (2010)
ISR
Digital Infrastructures: The Missing IS Research Agenda
42
Vodanovich et al.
(2010)
ISR
Digital Natives and Ubiquitous Information Systems
13
Bélanger and Carter
(2012)
JAIS
Digitizing Government Interactions with Constituents: An Historical Review of E-
Government Research in Information Systems
13
Xiao and Benbasat
(2007)
MISQ
E-Commerce Product Recommendation Agents: Use, Characteristics, and Impact
12
Schultze (2010)
JIT
Embodiment and presence in virtual worlds: a review
18
Xiao et al. (2013)
JIT
ICT innovation in emerging economies: a review of the existing literature and a
framework for future research
13
Avgerou (2008)
JIT
Information systems in developing countries: a critical research review
4
Whittington (2014)
JSIS
Information Systems Strategy and Strategy-as-Practice: A joint agenda
8
Merali et al. (2012)
JSIS
Information systems strategy: Past, present, future?
2
Chen et al. (2010)
MISQ
Information Systems Strategy: Reconceptualization, Measurement, and
Implications
4
Melville and
Kraemer (2004)
MISQ
Information Technology and Organizational Performance: An Integrative Model of
IT Business Value
16
Crowston and
Myers (2004)
JSIS
Information technology and the transformation of industries: three research
perspectives
4
Piccoli and Ives
(2005)
MISQ
IT-Dependent Strategic Initiatives and Sustained Competitive Advantage: A
Review and Synthesis of the Literature
9
Alavi and Leidner
(2001)
MISQ
Knowledge Management and Knowledge Management Systems: Conceptual
Foundations and Research Issues
22
Tams et al. (2014)
JSIS
Modern information technology in an old workforce: Toward a strategic research
agenda
32
Dimoka et al.
(2012)
MISQ
On the Use of Neurophysiological Tools in IS Research: Developing a Research
Agenda for NeuroIS
43
Chiasson et al.
(2008)
ISJ
Pluralist action research: a review of the information systems literature
4
Bélanger and
Crossler (2011)
MISQ
Privacy in the Digital Age: A Review of Information Privacy Research in
Information Systems
35
Ba et al. (2001)
ISR
Research Commentary: Introducing a Third Dimension in Information Systems
Design—The Case for Incentive Alignment
26
Schryen (2013)
EJIS
Revisiting IS business value research: what we already know, what we still need to
know, and how we can get there
13
Page 7 of 19
A Framework for Rigorously Identifying Research Gaps
Thirty Sixth International Conference on Information Systems, Fort Worth 2015 8
Pavlou (2011)
MISQ
State of the Information Privacy Literature: Where are we now and Where Should
we go?
3
Besson and Rowe
(2012)
JSIS
Strategizing Information Systems-Enabled Organizational Transformation: A
Transdisciplinary Review and New Directions
10
Overby et al. (2010)
ISR
The Design, Use, and Consequences of Virtual Processes
9
Weerakkody et al.
(2009)
JIT
The diffusion and use of institutional theory: a cross-disciplinary longitudinal
literature survey
7
Mbarika et al.
(2005)
JAIS
The Neglected Continent of IS Research: A Research Agenda for Sub-Saharan
Africa
49
Keutel et al. (2014)
EJIS
Towards mindful case study research in IS: a critical analysis of the past ten years
4
Ahuja (2002)
EJIS
Women in the information technology profession: a literature review, synthesis
and research agenda
12
Table 3. Analyzed articles and amount of identified research gaps
We found two main categories of research gaps: Characterization and Presentation. Table 4 presents the
categories and their respective sub-categories, as well as the frequency with which they occur (also see
Appendix 1 for illustrating examples).
Category
Sub-category
Definition
# Research gaps
Characterization
The reason for the existence of a research gap.
Contradictory evidence
Results from studies allow for conclusions in
their own right, but are contradictory when
examined from a more abstract point of view.
7
Knowledge void
Desired research findings do not exist.
467
Action-knowledge conflict
Professional behavior or practices deviate from
research findings or are not covered by research.
3
Methodological conflict
A variation of research methods is necessary to
generate new insights or to avoid distorted
findings.
36
Evaluation void
Research findings or propositions need to be
evaluated or empirically verified.
11
Theory application void
Theory should be applied to certain research
issues to generate new insights.
31
Presentation
The approach that is used to present a research
gap in a research paper.
Parallel presentation
Synthesis and research gaps are presented
alternatingly.
21
!!
Sequential presentation
The synthesis is presented first, being followed by
the presentation of research gaps.
30
Table 4. Research gaps categories and their sub-categories
Characterization of Research Gaps
As noted above, we refined Jacobs' (2011) approach to characterize research problems based on our data
sample of literature reviews. It emerged that, in addition to four of the sub-categories that Jacobs
identified, two more sub-categories were required in order to achieve theoretical saturation. These sub-
categories, evaluation void and theory application void, are defined below. Two sub-categories proposed
by Jacobs (2011), theoretical conflict and provocative exception, could not be identified. Hence, six types
of research gaps emerged from the data.
An evaluation void occurs when research findings or propositions need to be evaluated or empirically
verified. For example, Piccoli and Ives (2005) find that “no study to date has directly attempted to
evaluate the ability of the IT project barrier to produce response lag. […] Yet, without rigorous studies
confirming or challenging the notion […], no such conclusion is warranted.“
A theory application void arises when theory should be applied to certain research issues in order to
generate new insights. For instance, Arnott and Pervan (2005) find that “DSS researchers need to
Page 8 of 19
A Framework for Rigorously Identifying Research Gaps
Thirty Sixth International Conference on Information Systems, Fort Worth 2015 9
embrace contemporary research in psychology, management and related fields to provide a stronger
theoretical basis for projects.”
An example of contradictory evidence can be found in the article by Xiao and Benbasat (2007): “While
many studies have shown that recommendation agents use did result in improved decision quality and
decreased decision effort, there also exists some counter evidence.”
Bélanger and Crossler (2011) identify a knowledge void: “Even if other streams of IS research suggest that
intentions lead to behaviors, the privacy paradox should be explored further to provide an understanding
as to why such is not the case with information privacy.”
An action-knowledge conflict can be found in the literature review by Gefen et al. (2008): “As e-commerce
moves from primarily new and search products to experience products, trust as related to product
understanding and its underlying dimensions are likely to have a different role that could be the topic of
future research.”
An example of methodological conflict is part of the paper by Arnott and Pervan (2005): “Another
strategy for improving the relevance of DSS research is to increase the number of case studies, especially
interpretive case studies. DSS is lagging behind general IS in the adoption of this research paradigm.”
We find that, contrary to Jacobs’s (2011) belief, knowledge voids are not occurring infrequently. In fact,
knowledge voids account for 467 (84 percent) of all research gaps. A reason for this finding might be the
dynamics of the IS field (Swanson 1994), that tends to be recurrently under-researched due to the
sustained emergence of new research topics.
Presentation of Research Gaps
While the presentation of research gaps is not synonymous with the identification of research gaps in the
narrower sense, we assume that the identification of research gaps leads to the question on how to present
research gaps in the respective literature review. Researchers make various choices about how they
present research gaps. Thus, a classification is necessary to enable researchers to make informed
decisions about the presentation of research gaps.
We identified two possibilities of presenting research gaps from our sample: A sequential presentation
occurs when the synthesis is presented first, being followed by the presentation of research gaps. We refer
to parallel presentation when synthesis and research gaps are presented interchangeably and hence are
not clearly separated from each other. Both forms of presenting research gaps are visualized in Figure 2.
30 articles (75 percent) employ a sequential presentation, while 21 articles (53 percent) employ a parallel
presentation. Hence, there are 11 articles (28 percent) that employ both presentation approaches. This
means that they present research gaps during the synthesis as well as after the synthesis.
Figure 2. Sequential presentation and parallel presentation
Sequential presentation
Parallel presentation
Synthesis
Research
gap
Synthesis
Research
gap
Synthesis
Research
gap
...
Page 9 of 19
A Framework for Rigorously Identifying Research Gaps
Thirty Sixth International Conference on Information Systems, Fort Worth 2015 10
Towards a Framework for Identifying Research Gaps
For the framework, a distinction is being made between the identification of research gaps in the broader
and the localization of research gaps in the narrower sense. We refer to localization when sets of
information that were derived from the reviewed literature are classified as gaps that require further
research to be resolved. The term identification is being used in the broader sense. This includes
localization, but also characterization, verification, and presentation.
The construction of the framework is based on the findings from our analysis of literature reviews. The
framework consists of four components (see Figure 3). The initial stage is the localization of research
gaps, which is informed by the characterization of research gaps. After this stage has been completed,
research gaps may need to be verified. Subsequently, scholars might want to present the research gaps
that could be verified. In the following, the four components of the framework are discussed in detail.
Localization Characterization
Verification
Presentation
Localization strategy
depends on type of
research gap
After locating
a research gap, characterizing its type can help to specify
what kind of research is required to address
the research gap
Substantiating the existence
of the research gap
Adequately presenting the
research gap in a literature
review
Figure 3. Framework for identifying research gaps in literature reviews
Localization
Vom Brocke et al.'s (2009) framework suggests that the process of creating a research agenda starts after
the literature synthesis has been undertaken (see Figure 1). However, we posit that the localization of the
research gaps already commences when the literature is being synthesized. As researchers constantly
scrutinize concepts that emerge from the literature, they already start to uncover potential gaps in the
literature. The lack of methodological literature on this topic suggests that this procedure is often creative,
implicit, and informal. The chart method (see Table 1) illustrates how the processes of synthesizing the
literature and localizing research gaps are intertwined. The chart method can be used for literature
synthesis and for research gap localization (vom Brocke et al. 2009; Webster and Watson 2002). When
researchers start to chart the reviewed literature according to various concepts, they already perceive
blank fields in the chart, which may indicate research gaps. The process of localizing research gaps is
being informed strongly by the characterization of research gaps. The interaction of these two concepts is
explored in the next section.
Page 10 of 19
A Framework for Rigorously Identifying Research Gaps
Thirty Sixth International Conference on Information Systems, Fort Worth 2015 11
Characterization
In the context of this framework, characterization means to classify research gaps owing to the reasons of
their existence. Following Robinson et al. (2011), we assume this is an integral aspect of identifying
research gaps. As the literature review above revealed, we identified six different types of research gaps.
Research gaps are defined as gaps in ‘sets of information’. This gives rise to two questions. What kinds of
sets of information are related to each other, and how are they related? We believe that the various forms
of research gaps might be useful indicators to answer these questions.
For instance, it is possible to systematically analyze the methods that the reviewed literature has
employed. If literature on a certain topic always uses the same research approach, most likely there is a
research gap in the direction of methodology. For the identification of contradictory evidence, it is
reasonable to synthesize key findings of the reviewed literature concept-wise, e.g., by means of the chart
method, and to scrutinize the synthesis in terms of contradictions. For the identification of knowledge
voids, an approach for localization might be to analyze the literature concept-wise and to look for research
fields that feature specific gaps or are under-researched. The localization of action-knowledge conflicts
requires information about action, for instance, the behavior of professionals. While this kind of
information may be derived from the literature, it is also likely that in this case the reviewer’s personal
experiences play an important role. For evaluation voids, a reasonable localization strategy might be to
analyze whether research findings have been evaluated and empirically verified. In order to localize theory
application voids, researchers should analyze theories already employed to explain particular phenomena
and try to identify potential theories that might offer a complementary perspective.
These examples illustrate that the answers to the questions posed above depend on the examined type of
research gap. Different types of research gaps require different localization strategies (see Table 5). Thus,
the concept of characterization is capable of enhancing the process of localizing research gaps. When
trying to localize research gaps, researchers can constantly refer to the concept of characterization.
Type of Research Gap
Localization Strategy
Methodological conflict
Scrutinize if findings on a certain topic are inconclusive with regard to applied
research methods.
Contradictory evidence
Synthesize key findings and determine contradictions.
Knowledge void
Analyze literature with regard to theoretical concepts (e.g., using the chart method)
and look for specific gaps or under-researched areas of research.
Action-knowledge conflict
Collect information about action and relate this information to the knowledge base.
Evaluation void
Analyze if research findings have been evaluated and empirically verified.
Theory application void
Analyze the theories that have been employed to explain certain phenomena and
identify further theories that might contribute to the knowledge base as well.
Table 5. Types of research gaps and corresponding localization strategies
The two concepts of localization and characterization are also related vice versa: The characterization of
research gaps might enable scholars to specify what kind of research is required to resolve the respective
research gap. For instance, a methodological conflict indicates that there is a need to vary research
methods. For contradictory evidence, new theories that enable the incorporation of all evidence and
possibly explain the reasons for the existence of that particular contradictory evidence may be needed.
Verification
After research gaps have been localized, they may need to be verified. In this context, verification means
to ensure that the research gap does indeed exist. It may be argued that literature reviews should
synthesize all relevant literature and integrate all knowledge that exists in the respective research field.
Thus, there would be no need for verification, as research gaps would emerge from a synthesis of the
entire relevant knowledge. However, not all literature reviews are exhaustive, meaning that not the entire
relevant literature is included (Cooper 1988). If a literature review is not exhaustive, however, there is a
need to verify the research gaps derived from the synthesis since papers may have been overlooked that
reveal that the research gaps were closed before.
Page 11 of 19
A Framework for Rigorously Identifying Research Gaps
Thirty Sixth International Conference on Information Systems, Fort Worth 2015 12
For verification, it may be reasonable to conduct a forward search, based on the articles from which the
respective research gaps emerged or that are closely linked to the research gap. The rationale of this
approach is that other researchers who may have closed the research gap would also quote the articles
from which the research gap emerges to justify their studies. In case the research gap does not directly
emerge from specific research papers, it might help to search relevant databases or scan prevailing
textbooks for search terms that refer to the research gap. In addition, the researcher might need to
undertake further efforts beyond this proposed approach if there is an indication that the potential
research gap may have already been closed.
It is also possible that the verification process reveals that other reviewers have identified the respective
research gap before. This would not necessarily challenge the existence of the research gap. However, this
information would still be of good use, as it would highlight the relevance of the research gap.
Furthermore, if other scholars have identified the respective research gap, this may be an indication that
the research gap may indeed exist.
Verification could also go beyond mere literature analysis. In this regard, we believe triangulation has the
potential to increase validity of the identified research gaps. For instance, a survey or a focus group with
practitioners could be very helpful to assess and discuss the validity, applicability, and also priority of the
research gaps.
The approach of verifying research gaps has the advantage of enabling researchers to substantially reduce
the likelihood that the respective research gaps have been closed or do not exist, while not requiring an
exhaustive literature review. While it may be argued that literature reviews should always be exhaustive,
we reckon that analyzing all prior research is neither always possible nor economical and necessary.
Presentation
As noted above, there are two approaches to present research gaps in literature reviews, sequential
presentation and parallel presentation. We believe that scholars should make a well-founded decision
about the approach they employ for presenting research gaps. Thus, there is a need to discuss the
characteristics of both approaches.
Parallel presentation features the research gaps during the discussion of the synthesis. As argued above,
the processes of research synthesis and research gap localization cannot be separated. Thus, the parallel
presentation may be a more ‘natural’ approach, as it allows researchers to illustrate how the research gaps
unfold as a result from the synthesis. Parallel presentation facilitates the comprehensive disclosure of the
sets of information that the research gap stems from, allowing readers to recognize the origins of the
respective research gap and to gain a better understanding for what reasons the gap exists.
Sequential presentation describes the research gaps after the synthesis, such that the research gaps are
presented separately from the synthesis. Hence, readers can quickly locate research gaps in the review.
Furthermore, this form of research gap presentation resembles the framework for literature reviewing by
vom Brocke et al. (2009), as it considers the act of setting a research agenda as following the research
synthesis. Thus, it may be argued that the sequential presentation is a more structured approach to
present research gap compared to parallel presentation.
In our opinion, both approaches complement each other. Thus, literature reviews would benefit from
employing parallel presentation to demonstrate the origins of the research gaps and additionally present
them separately to facilitate finding the research gaps in the paper. A helpful example of this approach can
be found in Schneider and Sunyaev (2014).
Demonstrating the Framework
In this section, the framework for identifying research gaps is demonstrated to show how the framework
can be used by further research. For the demonstration, the identification of the research gap that
motivated this paper is being discussed. The research gap has emerged from a review of literature about
the methodology of literature reviews.
The research gap has been localized using the framework of vom Brocke et al. (2009). The framework
features five stages for literature reviewing (see Table 6). The goal was to identify methodological
Page 12 of 19
A Framework for Rigorously Identifying Research Gaps
Thirty Sixth International Conference on Information Systems, Fort Worth 2015 13
guidance for each of the five stages. However, as Table 6 illustrates, we were unable to locate a specific
method for conducting stage five of the framework. This concept-driven analysis revealed that there was
no method for identifying research gaps when conducting qualitative literature reviews. Thus, a
knowledge void has been localized (Localization).
Phase
Task
Sources offering methodological guidance
1
Definition of review scope
Cooper (1988)
2
Conceptualization of topic
Rowley and Slack (2004); Zorn and Campbell (2006)
3
Literature search
vom Brocke et al. (2009); Webster and Watson (2002)
4
Literature analysis and
synthesis
Webster and Watson (2002); Wolfswinkel et al. (2013)
5
Research gap identification
No method
Table 6. Demonstration of research gap localization
In our case, the localization of the research gap was not directly informed by its characterization.
However, another benefit from the characterization process could be seized. As the research gap was a
knowledge void, it was clear that its closure required generating additional knowledge. This finding
subsequently led to the idea that literature reviews may possibly contain the required knowledge, at least
implicitly (Characterization).
Subsequently, the potential research gap required verification. In this process, we found a framework for
identifying research gaps (Robinson et al. 2011). Although Robinson et al.’s (2011) framework focuses on
quantitative literature reviews in the health care domain, we reckon that researchers who might have
addressed the same research gap we had localized would refer to this paper. Thus, we conducted a
forward search for this article. Furthermore, we searched Google Scholar for papers that might be
relevant and additionally scanned prevailing textbooks on literature review methodology. In this process
of verification, we did not find a framework for identifying research gaps from qualitative literature
reviews. Hence, we consider our research gap as verified (Verification).
In the last step of our framework, there may be a need to present the research gap. However, in our
particular case, the research gap was not part of a ‘pure’ literature review as argued above. Instead, the
research gap motivated this article. Thus, the final step of our framework (Presentation) is not applicable
in this paper.
Discussion
Our paper sought to answer two research questions: For answering the first question regarding how prior
literature reviews identified research gaps, we thoroughly reviewed the IS literature. Employing a
grounded theory approach, two categories (characterization and presentation) and their respective sub-
categories were identified from a sample of 40 literature reviews published in leading IS journals. Based
on these insights, we subsequently developed a framework for identifying research gaps from literature
reviews to address the second research question.
The framework comprises four steps: Localization, characterization, verification, and presentation. The
term localization is employed to describe the identification of research gaps in a narrower sense, i.e.,
scrutinizing the sets of information that were classified by the reviewed literature as gaps meaning that
they require further research to be resolved. The stage of localization is informed by the characterization
of research gaps, which means to classify research gaps according to the reasons why they exist. In a third
step the research gaps should be verified to substantiate their existence. The fourth step of the framework
entails presenting the research gaps. After introducing the framework, we exemplarily demonstrated how
the framework could be applied.
Implications
An essential aspect of reviewing the literature is to identify research gaps (Webster and Watson 2002).
While reviews ought to be conducted rigorously (e.g., Fink 2010), there has been a lack of methodological
support for the process of identifying research gaps. Our framework offers researchers important
Page 13 of 19
A Framework for Rigorously Identifying Research Gaps
Thirty Sixth International Conference on Information Systems, Fort Worth 2015 14
guidelines to identify research gaps and enhance the rigor of the identification process. This increases the
quality of the review as “better legitimization of every choice made during the review process enhances the
value of the review” (Wolfswinkel et al. 2013) and as a consequence the chance of getting published.
Furthermore, the framework enables researchers to conduct their reviews more efficiently, as awareness
of the types of research gaps can help to guide the synthesis. Syntheses can be tailored to search for and
potentially reveal research gaps using the localization strategies.
An objection to scientific guidelines for the identification of research gaps that may possibly arise is that
they stifle creativity. However, Cooper (1998, p. 184) notes that “rigorous criteria will not produce
syntheses that are mechanical and uncreative”. Instead, our framework aims to offer support for the
creativity of researchers. Furthermore, as the demonstration above indicates, the framework is not a
‘straitjacket’. It merely represents a guideline that may be altered depending on the requirements of the
reviewer’s purpose. Therefore, researchers should feel free to adjust the framework according to the
particular characteristics of their respective reviews.
Limitations and Future Research
This paper has several limitations, thus providing numerous opportunities for further research. First, the
approach of analyzing literature reviews in terms of how they identified research gaps did not always lead
to the desired results. For instance, it was not possible to isolate the act of localization from our data. This
aspect clearly requires further scrutiny, as it is pivotal for the identification of research gaps. An approach
that may provide illuminating results is to interview scholars about how they localized research gaps. The
findings from such interviews could then be employed to refine the framework. Second, the process of
verifying research gaps is not grounded in the data as well. While we believe that this paper provides
substantial arguments for the inclusion of this process, the aspect certainly requires further examination.
Third, while the applicability of the framework is exemplarily demonstrated in our paper, it requires
further evaluation. Fourth, as argued above, the processes of literature synthesis and research gap
localization are closely intertwined. Future research could investigate this relation more intensely,
especially regarding various approaches for literature syntheses such as grounded theory.
As the analysis of the sample of literature reviews revealed, a predominant number of research gaps are
knowledge voids, hence indicating that a more granular classification for knowledge voids may be
possible. The feasibility of such a classification could be discussed in this context as well. Furthermore, it
could be analyzed why such a large number of research gaps are knowledge voids.
Conclusion
This paper’s main contribution is a framework that should enable researchers to identify research gaps
more rigorously from qualitative literature reviews in the IS domain. While it is widely acknowledged that
literature reviews should identify research gaps, we found that there were no guidelines for this process.
As literature reviews need to be conducted rigorously, such a framework fills a gap in the literature.
In order to build the framework, we identified the relevant justificatory knowledge. As prior
methodological literature was rather silent on how research gaps were identified, we conducted a
literature review analyzing 40 papers to search for explicit or implicit knowledge on this topic. We
employed a grounded theory approach, leading to two categories, characterization and presentation, and
six and two respective sub-categories from our sample of literature reviews.
We hope that our framework provides researchers guidance for the essential task of identifying research
gaps in a more rigorous, effective, and efficient manner. We further hope that our paper serves as a
starting point for a scholarly debate on the important, yet understudied, topic of rigorously identifying
research gaps in literature reviews.
Page 14 of 19
A Framework for Rigorously Identifying Research Gaps
Thirty Sixth International Conference on Information Systems, Fort Worth 2015 15
References
Ahuja, M. 2002. "Women in the information technology profession: a literature review, synthesis and
research agenda," European Journal of Information Systems (11:1), pp. 20–34.
Aksulu, A., and Wade, M. 2010. "A Comprehensive Review and Synthesis of Open Source Research,"
Journal of the Association for Information Systems (11:11), pp. 576–656.
Alavi, M., and Leidner, D. E. 2001. "Knowledge Management and Knowledge Management Systems:
Conceptual Foundations and Research Issues," MIS Quarterly (25:1), pp. 107–136.
Arnott, D., and Pervan, G. 2005. "A critical analysis of decision support systems research," Journal of
Information Technology (20:2), pp. 67–87.
Auerbach, C. F., and Silverstein, L. B. 2003. Qualitative Data: An Introduction to Coding and Analysis,
New York: New York University Press.
Avgerou, C. 2008. "Information systems in developing countries: a critical research review," Journal of
Information Technology (23:3), pp. 133–146.
Ba, S., Stallaert, J., and Whinston, A. B. 2001. "Research Commentary: Introducing a Third Dimension in
Information Systems Design—The Case for Incentive Alignment," Information Systems Research
(12:3), pp. 225–239.
Bélanger, F., and Carter, L. 2012. "Digitizing Government Interactions with Constituents: An Historical
Review of E-Government Research in Information Systems," Journal of the Association for
Information Systems (13:5), pp. 363–394.
Bélanger, F., and Crossler, R. E. 2011. "Privacy in the Digital Age: A Review of Information Privacy
Research in Information Systems," MIS Quarterly (35:4), pp. 1017–1041.
Besson, P., and Rowe, F. 2012. "Strategizing Information Systems-Enabled Organizational
Transformation: A Transdisciplinary Review and New Directions," The Journal of Strategic
Information Systems (21:2), pp. 103–124.
Chen, D. Q., Mocker, M., Preston, D. S., and Teubner, A. 2010. "Information Systems Strategy:
Reconceptualization, Measurement, and Implications," MIS Quarterly (34:2), pp. 233–259.
Chiasson, M., Germonprez, M., and Mathiassen, L. 2008. "Pluralist action research: a review of the
information systems literature," Information Systems Journal (19:1), pp. 31–54.
Conboy, K. 2009. "Agility from First Principles: Reconstructing the Concept of Agility in Information
Systems Development," Information Systems Research (20:3), pp. 329–354.
Cooper, H. 1998. Synthesizing Research - A Guide for Literature Review , Thousand Oaks et al.: SAGE
Publications.
Cooper, H. M. 1988. "Organizing knowledge syntheses - A taxonomy of literature reviews," Knowledge in
Society (1:1), pp. 104–126.
Page 15 of 19
A Framework for Rigorously Identifying Research Gaps
Thirty Sixth International Conference on Information Systems, Fort Worth 2015 16
Corbin, J., and Strauss, A. 2008. Basics of Qualitative Research: Techniques and Procedures for
Developing Grounded Theory, Los Angeles et al.: SAGE Publications.
Crowston, K., and Myers, M. D. 2004. "Information technology and the transformation of industries:
three research perspectives," The Journal of Strategic Information Systems (13:1), pp. 5–28.
D’Arcy, J., and Herath, T. 2011. "A review and analysis of deterrence theory in the IS security literature:
making sense of the disparate findings," European Journal of Information Systems (20), pp. 643–
658.
Dhillon, G., and Backhouse, J. 2001. "Current directions in IS security research: towards socio-
organizational perspectives," Information Systems Journal (11:2), pp. 127–153.
Dimoka, A., Banker, R. D., Benbasat, I., Davis, F. D., Dennis, A. R., Gefen, D., Gupta, A., Ischebeck, A.,
Kenning, P. H., Pavlou, P. A., Müller-Putz, G., Riedl, R., vom Brocke, J., Weber, B. 2012. "On the Use
of Neurophysiological Tools in IS Research: Developing a Research Agenda for NeuroIS," MIS
Quarterly (36:3), pp. 679–702.
Eagly, A. H., and Wood, W. 1994. "Using Research Syntheses to Plan Future Research," in The Handbook
of Research Syntheses, H. Cooper and L. V. Hedges (eds.), New York: Russell Sage Foundation, pp.
485–500.
Fink, A. 2010. Conducting Research Literature Reviews - From the Internet to Paper, Thousand Oaks et
al.: SAGE Publications.
Gefen, D., Benbasat, I., and Pavlou, P. A. 2008. "A Research Agenda for Trust in Online Environments,"
Journal of Management Information Systems (24:4), pp. 275–286.
Glaser, B. G., and Strauss, A. L. 2008. The Discovery of Grounded Theory: Strategies for Qualitative
Research, New Brunswick, London: AldineTransaction.
Hart, C. 2009. Doing a Literature Review - Releasing the Social Science Research Imagination, London
et al.: SAGE Publications.
Imel, S. 2011. "Writing a Literature Review," in The Handbook of Scholarly Writing and Publishing, T. S.
Rocco and T. Hatcher (eds.), San Francisco: Jossey-Bass, pp. 145–160.
Jacobs, R. L. 2011. "Developing a Research Problem and Purpose Statement", in The Handbook of
Scholarly Writing and Publishing, T. S. Rocco and T. Hatcher (eds.), San Francisco: Jossey-Bass,
pp. 125–141.
Jeyaraj, A., Rottman, J. W., and Lacity, M. C. 2006. "A review of the predictors, linkages, and biases in IT
innovation adoption research," Journal of Information Technology (21:1), pp. 1–23.
Keutel, M., Michalik, B., and Richter, J. 2014. "Towards mindful case study research in IS: a critical
analysis of the past ten years," European Journal of Information Systems (23:3), pp. 256–272.
Lacity, M. C., Khan, S., Yan, A., and Willcocks, L. P. 2010. "A review of the IT outsourcing empirical
literature and future research directions," Journal of Information Technology (25:4), pp. 395–433.
Lacity, M. C., Solomon, S., Yan, A., and Willcocks, L. P. 2011. "Business process outsourcing studies: a
critical review and research directions," Journal of Information Technology (26:4), pp. 221–258.
Page 16 of 19
A Framework for Rigorously Identifying Research Gaps
Thirty Sixth International Conference on Information Systems, Fort Worth 2015 17
Leidner, D. E., and Kayworth, Ti. 2006. "A Review of Culture in Information Systems Research: Toward a
Theory of Information Technology Culture Conflict," MIS Quarterly (30:2), pp. 357–399.
Levy, Y., and Ellis, T. J. 2006. "A Systems Approach to Conduct an Effective Literature Review in Support
of Information Systems Research," Information Science Journal (9), pp. 182–212.
Mbarika, V. W. A., Okoli, C., Byrd, T. A., and Datta, P. 2005. "The Neglected Continent of IS Research: A
Research Agenda for Sub-Saharan Africa," Journal of the Association for Information Systems
(6:5), pp. 130–170.
McGhee, G., Marland, G. R., and Atkinson, J. 2007. "Grounded theory research: literature reviewing and
reflexivity," Journal of Advanced Nursing (60:3), pp. 334–342.
Melville, N., and Kraemer, K. 2004. "Information Technology and Organizational Performance: An
Integrative Model of IT Business Value," MIS Quarterly (28:2), pp. 283–322.
Merali, Y., Papadopoulos, T., and Nadkarni, T. 2012. "Information systems strategy: Past, present,
future?," The Journal of Strategic Information Systems (21:2), pp. 125–153.
Overby, E., Slaughter, S. a., and Konsynski, B. 2010. "Research Commentary - The Design, Use, and
Consequences of Virtual Processes," Information Systems Research (21:4), pp. 700–710.
Pavlou, P. A. 2011. "State of the Information Privacy Literature: Where are we now and Where Should we
go?," MIS Quarterly (35:4), pp. 977–988.
Piccoli, G., and Ives, B. 2005. "IT-Dependent Strategic Initiatives and Sustained Competitive Advantage:
A Review and Synthesis of the Literature," MIS Quarterly (29:4), pp. 747–776.
Robinson, K. A., Saldanha, I. J., and McKoy, N. A. 2011. "Development of a Framework to Identify
Research Gaps from Systematic Reviews," Journal of Clinical Epidemiology (64:12), pp. 1325–1330.
Rowe, F. 2014. "What literature review is not: diversity, boundaries and recommendations," European
Journal of Information Systems (23:3), pp. 241-255.
Schneider, S., and Sunyaev, A. 2014. "Determinant factors of cloud-sourcing decisions: reflecting on the
IT outsourcing literature in the era of cloud computing," Journal of Information Technology, pp. 1–
31.
Schryen, G. 2013. "Revisiting IS business value research: what we already know, what we still need to
know, and how we can get there," European Journal of Information Systems (22:2), pp. 139–169.
Schultze, U. 2010. "Embodiment and presence in virtual worlds: a review," Journal of Information
Technology (25:4), pp. 434–449.
Swanson, B. E. 1994. "Information Systems Innovation Among Organizations," Management Science
(40:9), pp. 1069–1092.
Tamm, T., Seddon, P. B., Shanks, G., and Reynolds, P. 2011. "How Does Enterprise Architecture Add
Value to Organisations?," Communications of the Association for Information Systems (28:1), pp.
141–168.
Tams, S., Grover, V., and Thatcher, J. 2014. "Modern information technology in an old workforce: Toward
a strategic research agenda," The Journal of Strategic Information Systems (23:4), pp. 284–304.
Page 17 of 19
A Framework for Rigorously Identifying Research Gaps
Thirty Sixth International Conference on Information Systems, Fort Worth 2015 18
Tilson, D., Lyytinen, K., and Sørensen, C. 2010. "Research Commentary - Digital Infrastructures: The
Missing IS Research Agenda," Information Systems Research (21:4), pp. 748–759.
Vodanovich, S., Sundaram, D., and Myers, M. 2010. "Research Commentary - Digital Natives and
Ubiquitous Information Systems," Information Systems Research (21:4), pp. 711–723.
Vom Brocke, J., Simons, A., Niehaves, B., Riemer, K., Plattfaut, R., and Cleven, A. 2009. "Reconstructing
the Giant: On the Importance of Rigour in Documenting the Literature Search Process," in
Proceedings of the ECIS 2009, S. Newell, E. A. Whitley, N. Pouloudi, J. Wareham, & L. Mathiassen
(eds.), Verona, Italy, pp. 2206–2217.
Von Krogh, G., Haefliger, S., Spaeth, S., and Wallin, M. W. 2012. "Carrots and Rainbows: Motivation and
Social Practice in Open Source Software Development," MIS Quarterly (36:2), pp. 649–676.
Wareham, J., Zheng, J. G., and Straub, D. 2005. "Critical themes in electronic commerce research: a
meta-analysis," Journal of Information Technology (20:1), pp. 1–19.
Webster, J., and Watson, R. T. 2002. "Analyzing the Past to Prepare for the Future: Writing a Literature
Review," MIS Quarterly (26:2), pp. xiii–xxiii.
Weerakkody, V., Dwivedi, Y. K., and Irani, Z. 2009. "The diffusion and use of institutional theory: a cross-
disciplinary longitudinal literature survey," Journal of Information Technology (24:4), pp. 354–
368.
Whittington, R. 2014. "Information Systems Strategy and Strategy-as-Practice: A joint agenda," The
Journal of Strategic Information Systems (23:1), pp. 87–91.
Wolfswinkel, J. F., Furtmueller, E., and Wilderom, C. P. M. 2013. "Using grounded theory as a method for
rigorously reviewing literature," European Journal of Information Systems (22:1), pp. 45–55.
Xiao, B., and Benbasat, I. 2007. "E-Commerce Product Recommendation Agents: Use, Characteristics,
and Impact," MIS Quarterly (31:1), pp. 137–209.
Xiao, X., Califf, C. B., Sarker, S., and Sarker, S. 2013. "ICT innovation in emerging economies: a review of
the existing literature and a framework for future research," Journal of Information Technology
(28:4), pp. 264–278.
Yang, H., and Tate, M. 2012. "A Descriptive Literature Review and Classification of Cloud Computing
Research," Communications of the Association for Information Systems (31:2), pp. 35–60.
Zorn, T., and Campbell, N. 2006. "Improving the Writing of Literature Reviews Through a Literature
Integration Exercise," Business Communication Quarterly (69:2), pp. 172–183.
Page 18 of 19
A Framework for Rigorously Identifying Research Gaps
Thirty Sixth International Conference on Information Systems, Fort Worth 2015 19
Appendix 1
Category
Sub-category
Definition
Example
Characterization
The reason for the existence
of a research gap.
Contradictory
evidence
Results from studies allow for
conclusions in their own
right, but are contradictory
when examined from a more
abstract point of view.
“While many studies have shown that
recommendation agents use did result in
improved decision quality and decreased
decision effort, there also exists some counter
evidence.” (Xiao and Benbasat 2007)
Knowledge void
Desired research findings do
not exist.
“Even if other streams of IS research suggest
that intentions lead to behaviors, the privacy
paradox should be explored further to provide
an understanding as to why such is not the case
with information privacy.” (Bélanger and
Crossler 2011)
Action-
knowledge
conflict
Professional behavior or
practices deviate from
research findings or are not
covered by research.
“As e-commerce moves from primarily new and
search products to experience products, trust as
related to product understanding and its
underlying dimensions are likely to have a
different role that could be the topic of future
research.” (Gefen et al. 2008)
Methodological
conflict
A variation of research
methods is necessary to
generate new insights or to
avoid distorted findings.
“Another strategy for improving the relevance of
DSS research is to increase the number of case
studies, especially interpretive case studies. DSS
is lagging behind general IS in the adoption of
this research paradigm.” (Arnott and Pervan
2005)
Evaluation void
Research findings or
propositions need to be
evaluated or empirically
verified.
“[…] no study to date has directly attempted to
evaluate the ability of the IT project barrier to
produce response lag. […] Yet, without rigorous
studies confirming or challenging the notion
[…], no such conclusion is warranted.“ (Piccoli
and Ives 2005)
Theory
application void
Theory should be applied to
certain research issues to
generate new insights.
“DSS researchers need to embrace contemporary
research in psychology, management and related
fields to provide a stronger theoretical basis for
projects.” (Arnott and Pervan 2005)
Page 19 of 19
