Methodological Rigor in Mixed Methods: An Application in Management Studies

Abstract

As mixed methods continues to grow as a discipline, work to define what constitutes quality mixed methods research has become an emergent conversation. While progress has been made in this area, there has been some debate as to what quality entails and how to achieve it. This article contributes to mixed methods by highlighting the importance of rigor as an interdisciplinary baseline for quality evaluation and proposes a rigorous mixed methods framework. This framework is then applied to the management studies literature to give insight into a literature base where mixed methods research is still relatively nascent. Findings give examples of current practices in management studies as well as an example of how the rigorous mixed methods framework can be operationalized.

Full text

Empirical Research
Journal of Mixed Methods Research
1–23
ÓThe Author(s) 2020
Article reuse guidelines:
sagepub.com/journals-permissions
DOI: 10.1177/1558689819900585
journals.sagepub.com/home/mmr
Methodological Rigor in Mixed
Methods: An Application in
Management Studies
Robert L. Harrison
1
, Timothy M. Reilly
2
and John W. Creswell
3
Abstract
As mixed methods continues to grow as a discipline, work to define what constitutes quality
mixed methods research has become an emergent conversation. While progress has been made
in this area, there has been some debate as to what quality entails and how to achieve it. This
article contributes to mixed methods by highlighting the importance of rigor as an interdisciplin-
ary baseline for quality evaluation and proposes a rigorous mixed methods framework. This
framework is then applied to the management studies literature to give insight into a literature
base where mixed methods research is still relatively nascent. Findings give examples of current
practices in management studies as well as an example of how the rigorous mixed methods
framework can be operationalized.
Keywords
management studies, mixed methods quality, mixed methods rigor
While combining qualitative and quantitative methods constitutes a long-standing research
practice (Greene, 2006; Pluye et al., 2009), the conceptualization of such practices as mixed
methods research is a relatively recent occurrence (Tashakkori & Teddlie, 2003). However, as
mixed methods research continues to grow as a discipline, including more than 31 books
devoted primarily to mixed methods research (Onwuegbuzie, 2012), it is important to review
usage patterns to advance an understanding of how scholars are utilizing the methodological
approach. The methodological aim of this article is to identify a framework for rigorous mixed
methods (Rigorous Mixed Methods) research and then apply that framework to the area of man-
agement studies to illustrate how mixed methods are being utilized in a discipline where the for-
mal adoption of mixed methods is relatively recent. The first phase of this study focuses on the
conceptualization, operationalization, and reporting of rigor rather than a discussion of quality.
While rigor and quality are often correlated in academia, there are idiosyncrasies across various
1
Western Michigan University, Kalamazoo, MI, USA
2
University of Minnesota–Duluth, Duluth, MN, USA; Sacred Heart University, Fairfield, CT, USA
3
University of Michigan, Ann Arbor, MI, USA
Corresponding Author:
Robert L. Harrison, Western Michigan University, 3210 Schnieder Hall, Kalamazoo, MI 49008, USA.
Email: robert.harrison@wmich.edu

disciplines and regions as to what constitutes high-quality and low-quality work. For example,
researchers in Europe might see qualitative research as more valuable than researchers in the
United States. In the publishing process, it is often possible for an imperfect article with a sig-
nificant contribution to be seen as a high-quality article, while highly rigorous work with a rela-
tively small contribution is seen as holding lesser value. This reflects concern about quality
where it is subjective to individuals and disciplines, and discussions of quality often become a
compensatory process where shortcomings in one area can be overcome by excellence in
another (and vice versa). By focusing on the rigor in the mixed methods research process and
the rigorous reporting of said research, this article advance the discipline of mixed methods
research in an objective and clear manner. To demonstrate the Rigorous Mixed Methods frame-
work, we review and evaluate the rigor of recently published research in management science
and offer descriptions of best practices related to mixed methods rigor. Furthermore, a content
analysis of a 6-year sample of published management science articles is presented to identify a
representative sample of articles that employ a mixed methods approach. Those articles are then
analyzed using the Rigorous Mixed Methods framework to highlight the characteristics of rigor
employed by management scholars in conducting mixed methods research. In doing so, we
address issues related to quality and rigor assessments in mixed methods management research
practices, while discussing how mixed methods is currently being used by management scholars
as well as investigating current trends and best practices.
We begin this article by discussing definitions of mixed methods research and introducing
our framework for Rigorous Mixed Methods. We then present a content analysis of mixed meth-
ods articles across four highly ranked management studies journals over a 6-year span between
2009 and 2014. To illustrate further the application of rigor in management science, we then
provide two exemplars of mixed methods research in management science. Our final discussion
highlights the elements of rigor seen most commonly in our subset of articles and suggests rec-
ommendations for researchers/trainees conducting and reporting mixed methods in management
studies, as well as for reviewers/editors who are assessing mixed methods manuscripts, papers,
and grant applications.
Definitions of Mixed Methods and Mixed Methods in Management
To begin, mixed methods research should not be confused with ‘‘mixed model’’ research,
wherein statistical analysis is conducted of fixed and random effects in a database. Mixed meth-
ods research should also not be confused with ‘‘multiple methods’’ research in which multiple
forms of qualitative data (e.g., interview and observations) or multiple forms of quantitative
data (e.g., surveys and experiments) are collected and analyzed. Several definitions exist as to
what constitutes ‘‘mixed methods.’’ In this article, we utilize the following definition based on
an analysis of definitions used by leaders in the field of mixed methods research:
Mixed methods research is the type of research in which a researcher or team of researchers com-
bines elements of qualitative and quantitative research approaches (e.g., use of qualitative and quan-
titative viewpoints, data collection, analysis, inference techniques) for the broad purpose of breadth
and depth of understanding and corroboration. (R. B. Johnson et al., 2007, p. 123)
The core assumption of this approach is that when a researcher, or team of researchers, com-
bines statistical trends (quantitative data) with lived experiences (qualitative data), the collective
strength of both data types provides a better understanding than one data type alone (Bryman,
2006; Greene et al., 1989). Although both data types are coalesced into one study, it does not
mean that the scope of each type is diminished in any way. That is, the rigorousness of each
2Journal of Mixed Methods Research

monomethodological approach should be maintained as part of the mixed methods approach.
As there are guidelines for assessing the rigor in any monomethod design, we now introduce
guidelines with which to evaluate the rigor of mixed methods research. This discussion allows
for both mixed methods researchers and reviewers to understand and assess methodological
rigor in articles that employ a mixed methods approach.
A history of the use of mixed methods in management studies can be traced back to the
Hawthorne studies conducted in the early 20th century. The Hawthorne studies investigated
principles of employee behavior and emphasized experiment and extensive interviewing and
observational data. These studies of social relations among employees highlighted the need for
unobtrusive methods of inquiry in management research (Tashakkori & Teddlie, 2003). A more
detailed account of the history and pioneers of mixed methods research in business are offered
by Molina-Azorin and Cameron (2015).
The complex nature of human behaviors often requires management scholars to understand
behaviors in both breadth and depth, a task for which mixed methods research is especially well
suited. In addition, management studies often include varying contexts with complex open sys-
tems and the management of people at different levels within the organization. Molina-Azorin
and Cameron (2015) suggest that mixed methods research can add value in management
research because it offers the opportunity to yield insight regarding both process (qualitative)
and outcomes (quantitative)—or individual-level (qualitative) and firm-level (quantitative)
domains. Furthermore, Hurmerinta-Peltomaki and Nummela (2006) found that in the field of
business, mixed methods add value by increasing validity in the findings, informing the collec-
tion of the second data source, and assisting with knowledge creation. Gibson (2017) suggested
the concept of ‘‘methodological fit’’ (p. 6) between method and research question as being a
potential driver of the increase in impact and rigor of a mixed methods study, suggesting that
authors should analyze their research question, the current state of the literature, as well as their
intended theoretical contributions when deciding to do mixed methods work. Gibson goes on to
propose that mixed methods research tend to offer four main values to researchers: elaboration,
generalization, data integration, and interpretations. She proposes that if these values are pur-
sued by researchers, they give an opportunity to develop further insight into an area than a sin-
gle monomethod presentation. This perspective is supported by Molina-Azorin (2011) who
found that mixed method articles had a higher impact on the management discipline. While the
value of mixed methods research is generally reaching consensus, the management discipline
has the opportunity to formalize its approach to mixed methods. Cameron, Sankaran, and
Scales (2015), in a review of the product management literature, found that while the preva-
lence of mixed methods articles had increased, the reporting of the methods had not kept pace
with other disciplines. In concert with advancing the Rigorous Mixed Methods framework, this
article seeks to close that gap.
Defining Rigorous Mixed Methods
Understanding of what constitutes methodological rigor is a key step in developing the
Rigorous Mixed Methods framework, and a brief discussion of quality in mixed methods is
included here to facilitate this definition. In their recent article, Hong and Pluye (2018) define
methodological quality as being ‘‘concerned with how a study is conducted. It is usually related
to the construct of trustworthiness: Is a study good enough for the results to be trustworthy?’’
(p. 5). They go on to further state that this ‘‘trustworthiness’’ is generally assessed based on the
methodologies used and the potential biases that might exist in a given study. Similarly, Leech
et al. (2010) proposed a validation framework that included a combination of both qualitative
and quantitative validity to assess mixed methods validity, which they posit will lead to higher
Harrison et al. 3

quality mixed methods articles. Central to the validation framework is the concept of legitima-
tion, drawn from Onwuegbuzie and Johnson (2006) and discussed using words such as ‘‘trust-
worthy,’’ credible,’’ and ‘‘dependable.’’ Another recent study included interviews with
researchers across several disciplines who had written methodological pieces on mixed methods
research (Fa
`bregues et al., 2018). While these authors identified several criteria that appeared
to hold across academic disciplines, they note that any discussion of quality ‘‘must include the
views of researchers regarding the very nature of criteria generation, including the approach to
appraising quality or even whether a consensus on criteria is appropriate to the field of MMR’’
(Fa
`bregues et al., 2018, p. 425). This presents a key problem for researchers, editors, and
reviewers. If all quality must be viewed through the lens of the discipline in which the study is
conducted, and if key aspects of quality are related to a subjective concept like ‘‘trustworthi-
ness,’’ how can an integrated definition of quality be created that will hold across disciplines?
Fa
`bregues et al. (2018) make headway on this question, as their results show that there are at
least two perspectives currently held toward quality, and that these perspectives are related to
the discipline of the researcher interviewed. Rather than expand this discussion of quality, we
narrow our focus to solely that of rigor in an effort to provide a foundation for further quality
discussions.
Current best practices for reporting rigor (although often used synonymously with quality)
are perhaps best exemplified by O’Cathain et al. (2008), who proposed a reporting paradigm
called the good reporting of a mixed methods study (GRAMMS). The GRAMMS framework
asks authors to report a justification, the design type, the components of each mono method, the
data integration of the study, limitations of the study, and insights gained from the mixing of
data. O’Cathain (2010) has also written about developing a framework for quality in mixed
methods research, which included a discussion of rigor, but focused primarily on judging the
overall quality of a research piece. O’Cathain discusses rigor as a subcomponent of quality, and
rigor is mentioned in several of her quality domains, yet a formal set of rigorous steps is not pre-
sented for authors. Further discussion of the concept of rigor in mixed methods can be found in
the area of health services (Wisdom et al., 2012), where authors propose a key point in the rigor
debate—that quality and rigor are separate—and further suggest that journal editors consider
publishing guidelines for rigor in mixed methods research. The following Rigorous Mixed
Methods framework answers that call. As suggested by Wisdom et al. (2012), rigor is primarily
concerned with both the actions of the researcher (i.e., the steps taken in the scientific process)
and the reporting mechanisms used to describe those steps to the reader, providing a better
understanding of exactly what the researchers did during their study. Rigor in this case is not
designed to replace quality but, rather, to facilitate the review of quality in a manuscript. By def-
inition, rigor does not necessarily make an article high quality, but properly utilized rigor does
allow for a reader, editor, or reviewer to judge the quality of the article. The first major contribu-
tion of this work is to clearly delineate between quality and rigor and provide guidelines for con-
ducting rigorous research. Again, as noted previously, the Rigorous Mixed Methods framework
is not intended as a substitute for quality but, rather, a framework that allows quality to be
assessed. The second major component of the Rigorous Mixed Methods framework is that it
conceptualizes rigor as a continuum in mixed methods. This continuum, currently operationa-
lized into categories of high, medium, and low for most criteria, adds a more nuanced evaluation
of a project to existing reporting checklists. The next section of the article describes the develop-
ment and components of the Rigorous Mixed Methods framework.
Our Rigorous Mixed Methods framework extends prior work on mixed methods reporting by
highlighting a process for conducting and reporting research according to primary and advanced
criteria for rigor. Rigorous Mixed Methods incorporates aspects of prior mixed methods report-
ing guidelines, but it goes further by delineating author tasks when conducting mixed methods
4Journal of Mixed Methods Research

research as well. In addition, the Rigorous Mixed Methods framework presented here provides
prescriptive advice on how to conduct mixed methods research and includes a conceptualization
of what constitutes high, medium, and low rigor for each of the Rigorous Mixed Methods cate-
gories. Our framework is informed by criteria utilized by one of the authors as evaluative cri-
teria for articles submitted to the Journal of Mixed Methods Research during his tenure as editor
of the journal.
Our discussion of Rigorous Mixed Methods is organized into two categories—primary and
advanced elements. The four primary elements represent what we describe as the core charac-
teristics of mixed methods research. The first element describes rigorous data collection of each
data strand (i.e., qualitative and quantitative). The second element describes rigorous data anal-
ysis of each data strand (i.e., qualitative and quantitative). The third element describes the
‘‘integration’’ or mixing of both data strands. This mixing is the cornerstone of mixed methods
research, as the premise of mixed methods is that the integration of data leads to more than the
sum of its parts, and potential methods of data integration will be discussed later on in this arti-
cle. The fourth primary element describes the use of a specific mixed methods design type.
While all primary elements of a mixed methods research design are important, a brief dis-
cussion of design types is useful here to set the stage for our content analysis. Mixed methods
design types are as central to conducting mixed methods projects as they are in projects labeled
as ‘‘experiments,’’‘‘surveys,’’ or ‘‘ethnographies,’’ because they help scholars understand and
convey how the studies are presented. The three basic mixed methods design types highlighted
in this study are exploratory sequential designs, explanatory sequential designs, and convergent
designs (Creswell & Clark, 2017). In exploratory sequential designs, researchers first collect
qualitative data, analyze the qualitative data, and then build on the qualitative data for the quan-
titative follow-up. In explanatory sequential designs, researchers first collect and analyze quan-
titative data, then build on those findings in a qualitative follow-up, which seeks to provide a
better understanding of the quantitative results. In convergent designs, researchers collect and
analyze quantitative and qualitative data with the intent to merge the results of the quantitative
and qualitative data analysis.
Beyond these four primary elements are advanced procedures or techniques that have devel-
oped in mixed methods research (Creswell & Clark, 2017). These procedures add to overall
mixed methodological rigor and include (1) providing a discussion of the aims and purpose for
the use of mixed methods and (2) including elements of the writing that aim to promote the
usage of mixed methods research through language and terminology. Presenting the aims and
purposes involves providing a clear rationale for conducting a mixed methods study, including
a mixed methods research question, and discussing the value of mixed methods research.
Presenting the elements of writing involves referencing mixed methods literature, using of joint
displays to show integration, and including mixed methods in the title. Examples of the varying
rankings of each element are outlined in Table 1.
In sum, these elements highlight the methodological rigor associated with various sections
within a manuscript. Next, we investigate how mixed methods rigor in management studies is
reported. Specifically, we conduct a content analysis of the management studies literature to
explore the ways in which scholars utilize the methodology.
Methodology
In this article, mixed methods articles were reviewed across four highly ranked management
studies journals over a 6-year span: the Academy of Management Journal, Administrative
Science Quarterly, Strategic Management Journal, and the Journal of Management. These
journals were chosen to provide data from eminent journals, and it is assumed that they
Harrison et al. 5

Table 1. Rigorous Mixed Methods (Rigorous Mixed Methods) Coding Scheme.
Rigorous Mixed
Methods
elements
High levels of rigor Medium levels of rigor Low levels of rigor
Aims and
purpose
Includes a rationale for using mixed methods.
Includes a mixed methods research
question.
Includes a discussion of the value of mixed
methods.
Includes a rationale for using mixed methods.
May include a mixed methods research
question.
May include a discussion of the value of
mixed methods.
Includes no discussion of a rationale for
rationale for using mixed methods, a
mixed methods research question, or a
discussion of the value of mixed
methods.
Data collection Includes the reporting of specific data
collection procedures for both qualitative
and quantitative data strands (e.g., sampling
procedures, types of data to be collected,
and instruments used in data collection).
Includes the collection of both qualitative and
quantitative data strands, but limits the
discussion of collection procedures for both
data types.
Includes the collection of both qualitative
and quantitative data strands, but does
not discuss data collection procedures
Data analysis Includes the reporting of analysis procedures
for both qualitative and quantitative data
strands that range from basic to more
sophisticated approaches; from descriptive
to inferential quantitative analysis, to coding
and thematic development qualitative
analysis.
Includes qualitative and quantitative analyses,
but at least one is not well reported. It is
unclear how mixed methods are used to
support the overall analysis.
Includes the reporting of analysis
procedures for only one, primary data
strand.
Data integration Includes the linking of both data strands.
Depending on the design type, both data
strands are either merged or one data
strand is used to explain, or build from, the
other. Joint displays and/or data
comparisons are utilized.
Includes the linking of both data strands, but
researchers do not describe a clear plan or
reason for doing so. Some discussion of
how integration affects the overall study.
Includes little to no integration of both
data strands. Little to no discussion of
how integration affects the study.
Mixed methods
design type
Includes a mixed methods design type (e.g.,
sequential explanatory).
Uses a diagram to show the design type.
Includes no discussion of a mixed methods
design type.
Includes a discussion of qualitative and
quantitative components individually.
Includes no discussion of a mixed
methods design type. Either the
qualitative or quantitative component is
missing or significantly lacking.
Elements of
writing
Includes references to mixed methods
literature. Identifies the study as mixed
methods in the title, abstract, and/or paper.
Includes a discussion of mixed methods, but
fails to cite any mixed methods literature.
Does not identify the study as mixed
methods.
Includes no discussion of mixed methods
or references to mixed methods
literature.
6

represent trends in best practice research. We feel that 6 years represent a sufficient period of
time to assess trends of recently published research. Next, we conduct a content analysis inves-
tigating the current state of mixed methods rigor in management studies. This analysis is simi-
lar to a mixed studies review as defined by Pluye et al. (2009)and Hong and Pluye (2018)
1
;
however, rather than focusing on a topic and presenting both quantitative and quantitative stud-
ies, we focus in on the methodological makeup of each article in the data set. Content analysis
is an observational technique that allows for a systematic evaluation of recorded communica-
tions (Kolbe & Burnett, 1991). In the current investigation, the recorded communication of
interest are the four selected journal publications. For the purpose of coding, articles were
required to contain qualitative research centered on primary data collection in nonnumerical
form (words, images, symbols, etc.) and quantitative research centered on data collection in
numerical form. Also, as part of our exclusion criteria, articles that quantified qualitative data
(also called ‘‘quantitizing’’), or transformed quantitative data in qualitative data (also called
‘‘qualitizing’’), are the topic of debate among mixed methods scholars in terms of mixed
method distinction; to prevent obscuring the central focus of this article, these articles were
omitted.
Our coding procedures followed a two-part process. First, articles were identified as includ-
ing the collection and reporting of both qualitative and quantitative data in some form. We were
incredibly inclusive in this portion of the data collection in an effort to identify as many articles
as possible. In Phase 2, articles were coded according to the criteria outlined in Table 1 and
identified as having an overall level of rigor. Most of the articles in the data set fell into the
low level of rigor column entirely. Technically, many of these low-level studies would not be
considered true ‘‘mixed methods studies’’ due to their lack of data integration. However, they
were included as a way to highlight the varying ways in which scholars are collecting and ana-
lyzing both qualitative and quantitative data in management studies. To identify an article’s
overall level of rigor, we identified each article’s level of rigor across the Rigorous Mixed
Methods elements. Most articles fell nearly entirely within one level, but several articles were
cross-classified with at least one Rigorous Mixed Methods element in the nonmajority (i.e.,
low, medium, or high category). For these articles, we created a cross-level classification
scheme (e.g., low–medium and medium–high). Most articles tended to have a certain level of
rigor, and no article reached across all three levels of rigor. This is likely due to the require-
ments for an article to be classified as having a high level of rigor. All disagreements in coding
were discussed and resolved by the authors.
Findings
There were 195 articles, out of 1,164 empirical articles, that incorporated the inclusionary ele-
ment of having collected both qualitative and quantitative data. Each article was analyzed using
the coding scheme outlined in Table 1, resulting in the data set of articles as shown in Table 2.
The breakdown across journals showed 72 articles in the Academy of Management Journal,
15 in Administrative Science Quarterly, 73 in the Strategic Management Journal, and 35 in the
Journal of Management. Each article was classified as having a low level of Rigorous Mixed
Methods to a high level of Rigorous Mixed Methods using the coding criteria presented in
Table 1. Of the 195 articles coded combining both qualitative and quantitative data, 128 were
classified as having low rigor. The most common reason for having a low level of rigor was a
complete lack of emphasis on the mixed portion of the methodology, despite reporting some
component of both qualitative and quantitative data collection as part of the research study.
These articles were often quantitative articles that referenced depth interviews as part of scale
development or pretesting but did not report any aspect of the qualitative work other than to
Harrison et al. 7

potentially reference it when discussing how their survey items were created. This type of study
nominally fits the ‘‘low’’ level for each criteria in Table 1. By R. B. Johnson et al.’s (2007) defi-
nition, this low level of rigor would not meet the standards for being classified as a mixed meth-
ods study. However, we included these low-level articles to highlight both the frequency of
articles that combine both data types and the opportunity to enhance methodological rigor. One
step up from the low rigor articles were the low–medium rigor articles. Of the 195 mixed meth-
ods articles classified, 20 were qualified as having a low–medium rigor level. To be classified as
a low–medium article, an article had to perform at a medium level for at least one of the criteria
in Table 1. For example, a scale development article might report having conducted depth inter-
views then nominally report and analyze the qualitative data collected. These types of articles
were relatively rare; however, this may be due to researchers making choices about what to
include and exclude in a study when subject to length requirements. It is important to note that
these articles with lower levels of rigor should not be viewed as poor scholarship, as they present
a methodology that answers specific research intentions. They merely are not viewed as rigorous
in mixed methods terms—something that was most likely outside of the goals of the authors. By
nature, mixed methods articles have more involved methods and data reporting requirements
than a monomethod study, and this may have resulted in one type of data being removed or de-
emphasized during the review or editing process. The next tier of mixed methods rigor involved
medium rigor articles, of which 28 were found out of 195 articles. For an article to be classified
as medium, it had to meet the majority of the criteria in Table 1 at the medium level.
Of the articles remaining, 19 were classified as either medium–high or high mixed methods
rigor. For an article to be classified as medium–high, it had to meet all criteria in Table 1 at the
medium level and have at least one criteria at the high level. Only 13 articles were classified as
medium–high. To achieve a high level of rigor, an article had to meet most of the criteria in
Table 1 at the high level. Of the 1,164 articles reviewed, only six were labeled at this level of
rigor. These 19 medium–high and high-level articles have been listed in Table 3 for further
review. Table 3 highlights how mixed methods is currently being conducted in management stud-
ies by listing the research topic area, journal of publication, mixed methods research components,
and qualitative and quantitative components. This table is designed to provide insight into a
developing mixed methods research discipline as well as to provide insight into management
scholars seeking to conduct mixed method studies. From a purely mixed methods standpoint, it
can be valuable to assess how other disciplines adopt and utilize mixed methods protocols. To
facilitate this, we conduct an in-depth discussion of the trends within the management discipline.
Researchers may also use this table as a starting point to see what types of work have been done
in their areas of study. Last, Table 3 gives further insight into the state of mixed methods research
Table 2. Rigorous Mixed Methods Inclusionary Rankings.
Journal title Low Low–medium Medium Medium–high High
Total
mixed methods
Total
articles (%)
Academy of
Management Journal
39 8 13 8 4 72 357 (20.1%)
Administrative
Science Quarterly
5 1 7 1 1 15 92 (16.3%)
Strategic Management
Journal
58 6 4 4 1 73 365 (20%)
Journal of
Management
26 5 4 0 0 35 350 (10%)
Total 128 20 28 13 6 195 1,164 (16.75%)
8Journal of Mixed Methods Research

Table 3. Rigorous Mixed Methods in Management Journals.
Author(s) Journal Topic area Mixed methods design
and operationalization
Qualitative sample/analysis Quantitative sample/analysis
Graffin et al.
(2011)
SMJ Impression
Management
Design: Exploratory Sequential
MM Research Questions? N
MM Referencing? N
Rationale for MM? Y
Value of MM? N
Emphasis? Quan
Unknown number of CEO
interviews
Not stated
Secondary data from 601 firms
Descriptive statistics and bivariate
correlations
Davies et al.
(2010)
SMJ Reputation and
Firm
Performance
Design: Multiple
Phases (Qual–Quan–Qual
MM Research Questions? N
MM Referencing? N
Rationale for MM? N
Value of MM? N
Emphasis? Quan
Unknown number of customer
interviews
Not stated
4,307 surveys (2,575 from
customers; 1,732 employees)
Descriptive statistics and
regression
Jonsson &
Regne
´r (2009)
SMJ Social
Complexity in
Core
Competences
Design: Exploratory Sequential
MM Research Questions? N
MM Referencing? N
Rationale for MM? Y
Value of MM? N
Emphasis? Quan
Interviews with 24 executive and
product developers interviews
Not stated
Event-history data from all
mutual fund firms in Sweden
Descriptive statistics and
regression
Mors (2010) SMJ Innovation Design: Exploratory Sequential
MM Research Questions? N
MM Referencing? N
Rationale for MM? Y
Value of MM? N
Emphasis? Quan
In-depth interviews with 32 senior
partners
Qualitative coding
Surveys of 102 senior partners
(69%)
Descriptive statistics, bivariate
correlations, and regression
Ethiraj et al.
(2012)
SMJ Innovation Design: Explanatory Sequential
MM Research Questions? N
MM Referencing? N
Rationale for MM? Y
Value of MM? Y
Emphasis? Quan
Interviews with 6 heads of
marketing, sales, and product
development
Not stated
Analysis of 120 service requests
Descriptive statistics, correlations,
and bivariate probit analysis
(continued)
9

Table 3. (continued)
Author(s) Journal Topic area Mixed methods design
and operationalization
Qualitative sample/analysis Quantitative sample/analysis
Bunderson &
Thompson
(2009)
ASQ Meaningful
Work
Design: Exploratory Sequential
MM Research Questions? N
MM Referencing? N
Rationale for MM? Y
Value of MM? N
Emphasis? Qual
Interviews with 23 zookeepers
Grounded theory
Survey of 982 zookeepers
Descriptive statistics, CFA, and
regression
Bernstein
(2012)
ASQ Organizational
Learning
Design: Exploratory Sequential
MM Research Questions? N
MM Referencing? N
Rationale for MM? Y
Value of MM? N
Emphasis? Qual
Collection of 800 hours of
observational field notes; three
exit interviews
Grounded theory approach
Experimental design
Descriptive statistics; GLS fixed
effect modeling
Battilana &
Casciaro
(2012)
AMJ Organizational
Change
Design: Convergent
MM Research Questions? N
MM Referencing? N
Rationale for MM? Y
Value of MM? N
Emphasis? Equal
12-20 interviews with clinical
managers
Longitudinal case study
Survey of 68 clinical managers
Descriptive statistics and
regression
Aime et al.
(2014)
AMJ Power
Transitions in
Cross-
Functional
Teams
Design: Exploratory Sequential
MM Research Questions? N
MM Referencing? N
Rationale for MM? Y
Value of MM? N
Emphasis? Equal
Interviews with 21 firm team
members
Qualitative field study
Experimental study of 45 teams
(131 participants)
Descriptive statistics, multilevel
modeling, and regression
Grant et al.
(2013)
AMJ Job Titles and
Emotions
Design: Exploratory Sequential
MM Research Questions? N
MM Referencing? N
Rationale for MM? Y
Value of MM? Y
Emphasis? Equal
Semistructured interview with 22
staff members and nonparticipant
observation
Inductive analytic approach
Survey of 169 senior managers
(75%)
Descriptive statistics, correlations,
and regression
(continued)
10

Table 3. (continued)
Author(s) Journal Topic area Mixed methods design
and operationalization
Qualitative sample/analysis Quantitative sample/analysis
Yang et al.
(2010)
AMJ Learning and
Knowledge
Spillovers in
Firms
Design: Exploratory Sequential
MM Research Questions? N
MM Referencing? N
Rationale for MM? Y
Value of MM? Y
Emphasis? Quan
10 in-depth interviews with
scientists and R&D managers
Not stated
Secondary financial/patent data
Descriptive statistics, correlations,
and regression
Almandoz
(2012)
AMJ New Banking
Ventures
Design: Convergent
MM Research Questions? N
MM Referencing? N
Rationale for MM? N
Value of MM? Y
Emphasis? Quan
50 interviews with bank directors
Not stated
Secondary data from bank
applications (253 documents)
Descriptive statistics and
correlations
Hiemeriks et al.
(2012)
AMJ Postacquisition
Learning
Design: Exploratory Sequential
MM Research Questions? N
MM Referencing? Y
Rationale for MM? Y
Value of MM? Y
Emphasis? Equal
Interviews with 30 executives
Not stated
Survey of 85 corporate
executives
Descriptive statistics, correlations,
and regression
Sonenshein
et al. (2014)
AMJ Environmental
Issues
Design: Exploratory Sequential
MM Research Questions? N
MM Referencing? Y
Rationale for MM? Y
Value of MM? Y
Emphasis? Equal
29 interviews and two field
observations
Coding, Categorization, and Theory
induction
Survey of 89 participants
CFA and ANOVA
Vergne (2012) AMJ Public
Disapproval of
Organizations
Design: Exploratory Sequential
MM Research Questions? Y
MM Referencing? Y
Rationale for MM? Y
Value of MM? Y
Emphasis? Equal
44 interviews with industry
personnel
Not stated
Panel data from arms industry
(2,014 excerpts; 2,046
observations)
Descriptive statistics, correlations,
and regression
(continued)
11

Table 3. (continued)
Author(s) Journal Topic area Mixed methods design
and operationalization
Qualitative sample/analysis Quantitative sample/analysis
Raffaelli &
Glynn (2014)
AMJ Institutional
Complexity
and Practice
Adoption
Design: Convergent
MM Research Questions? N
MM Referencing? Y
Rationale for MM? Y
Value of MM? Y
Emphasis? Quan
Unknown number of customer
interviews
Not stated
Surveys of 161 firm participants.
Descriptive statistics, correlations,
and regression
Kistruck et al.
(2013)
AMJ Base-of-the-
Pyramid
Markets
Design: Convergent
MM Research Questions? N
MM Referencing? Y
Rationale for MM? Y
Value of MM? N
Emphasis? Quan
Interviews with 31 salespeople
Not stated
Survey of 63 salespeople
Descriptive statistics, correlations,
and ANCOVA
Van Wijk et al.
(2013)
AMJ Organizational
Field Change
Design: Convergent
MM Research Questions? N
MM Referencing? Y
Value of MM? Y
Rationale for MM? Y
Emphasis? Quan
Interviews with 22 firm participants
Not stated
237 unique people/233 events
Network Analysis
Detert &
Edmondson
(2011)
AMJ Implicit Voice
Theories
Design: Exploratory Sequential
MM Research Questions? Y
MM Referencing? N
Rationale for MM? Y
Value of MM? Y
Emphasis? Quan
Interviews with 190 employees
Multistep inductive process
Survey of 450 students
Descriptive statistics and
correlations
Note. SMJ =Strategic Management Journal; ASQ =Administrative Science Quarterly; AMJ =Academy of Management Journal; MM = mixed methods; Y = yes; N = no; Quan = quantitative;
Qual = qualitative; CEO =chief executive officer; R&D = research and development; CFA = confirmatory factor analysis; GLS = generalized least square; ANOVA = analysis of
variance; ANCOVA = analysis of covariance.
12

in management. This presentation is similar to that of J. S. Johnson (2015) and allows for scho-
lars to easily compare existing articles with the criteria of Rigorous Mixed Methods suggested in
this article.
The articles in Table 3 can first be delineated by research area and publication outlet. While
all articles are in management journals, two articles focus on learning within the organization,
while two others focus on innovations. The articles labeled as high or medium–high are primar-
ily drawn from Academy of Management Journal (12) and Strategic Management Journal
(five), although Administrative Science Quarterly had two articles in the data set.
The most common mixed methods design type was the Exploratory Sequential design, wherein
a qualitative data collection was followed with a quantitative data collection (13/19). Other design
types included Convergent designs, where both qualitative and quantitative data were collected
simultaneously (5/19), and one multiphase design, which included an initial qualitative data col-
lection, a subsequent quantitative data collection, and a final qualitative follow-up.
The qualitative components of these mixed methods articles were primarily composed of
depth interviews with research subjects; however, two articles used at least some form of obser-
vational data collection and analysis. Quantitative analyses were generally well reported and
covered a wide range of techniques; most common were descriptive analyses with some form
of correlation or regression done to investigate relationships within the data.
To further the investigation of Rigorous Mixed Methods in management research, the fol-
lowing section identifies and discusses examples of the mixed methods research elements that
have been drawn from our data set. This examination of individual elements is then followed
by a more detailed discussion of two articles that serve as best practice exemplars of Rigorous
Mixed Methods management articles.
Illustrating Rigorous Mixed Methods
Several articles were identified as containing multiple elements of Rigorous Mixed Methods.
Although Table 1 offers a cursory description of Rigorous Mixed Methods elements, this sec-
tion offers a more detailed discussion of how the various elements of mixed methods rigor are
methodologically reported in the management literature. The reporting below also includes brief
discussions of mixed methods techniques, providing management researchers with a clear
understanding of the discussion. While both quantitative and qualitative elements of mixed
methods research are important, an in-depth discussion of them individually is beyond the scope
of this article. We begin by first discussing the element of mixed methods research designs, fol-
lowed by a brief discussion of data integration techniques.
Though none of the articles specifically identified a mixed methods research design, the
exploratory sequential approach was the dominant design type employed by organization scho-
lars. This is not surprising, as others have found this to be the dominant design type in business
research (Harrison, 2013; Harrison & Reilly, 2011).
Figure 1 shows the three basic mixed methods design strategies and the rationale an author
might use for implementing one of these design choices. For Convergent designs, a mixed
methods research question might be, ‘‘To what extent do the quantitative and qualitative results
agree?’’ For exploratory sequential designs, authors might ask, ‘‘To what extent can the qualita-
tive findings be generalized in a quantitative setting?’’ and for the explanatory sequential
design, the authors might ask, ‘‘To what extent can the qualitative findings help explain the
quantitative results?’’ These are quite simplistic as they are absent in any research context but
can easily be put into use. For example, if a research question requires the development of a
new survey instrument, a researcher can incorporate an exploratory sequential design (Creswell
and Clark, 2017). The exploratory sequential design is especially suited to scale development
Harrison et al. 13

as it allows authors to develop their items from the qualitative data collection phase. These
items can then be evaluated quantitatively. Each design type should be chosen with regard to
the research question of interest, as each rationale suggests formal research questions that may
be examined. Convergent designs, for example, work quite well for comparison research ques-
tions where the findings from the different research types can be investigated for similarities
and discrepancies. This list of research questions is not intended as an exhaustive list at this
time but merely as a guide for authors to understand how these designs may be useful when
crafting mixed methods research questions (Creswell & Clark, 2017).
While having a mixed methods research design can help identify and guide a study, the way
the data are mixed is also important. Data integration refers to how the qualitative and quantita-
tive results are brought together in a mixed methods study (Creswell & Clark, 2017). It is the
point of interface between the two data strands (Morse & Niehaus, 2009). This point of inter-
face can occur in the planning stages of research, during analysis, or at the conclusion of
research for interpretation. Strategies for integrating the data are connected to the three basic
designs and include (1) merging (convergent design), (2) explaining (explanatory sequential
design), or (3) building (exploratory sequential design). Merging strategies in convergent
designs have also been described as varying forms of data triangulation (see Turner et al.,
2017, for a more detailed discussion). The latter two strategies are sometimes collectively con-
sidered ‘‘connecting’’ strategies for integration. In our data set, data integration most often
Figure 1. Rigorous Mixed Methods Elements by Design Type.
14 Journal of Mixed Methods Research

followed standards associated with the design type utilized. That is, they were sequentially
linked when discussing how the qualitative data were used to develop quantitatively testable
theories or how the qualitative data helped explain aspects of the quantitative results. However,
data integration can be achieved in various ways. For example, Almandoz (2012) discussed
how mixing occurred at multiple stages of the research process:
This mixed methods design gains from the strengths of its component parts. Archival evidence is
unobtrusive and objective, while interviews add greater depth of interpretation that infuse the previ-
ous analysis with rich context. Qualitative support was critical in guiding the hypothesis generated,
the controls used, and the interpretation of the [quantitative] results. (p. 1390)
This mixing of data, especially in the data analysis section of the article, allows for stronger
overall analysis and interpretation by the researcher. A qualitative study may do a strong job of
describing an individual’s lived experience, while having poor generalizability. A quantitative
study may do a good job of describing an ‘‘average’’ person while describing no individual spe-
cifically. The mixed methods paradigm allows for researchers to accomplish both of these tasks.
This mixing of research perspectives allows for the best of both monomethod perspectives to be
included in a single study.
Having discussed the primary elements of Rigorous Mixed Methods, we will now discuss
the advanced elements that may contribute to an article’s mixed methods rigor. The first
advanced element involves the discussion of the aims and purpose for conducting mixed meth-
ods research. This is important because it offers evidence for the appropriateness of utilizing a
mixed methods approach and includes discussing the rationale and value of mixed methods
research, and the inclusion of a mixed methods research question. Several articles offer clear
rationales and discussions of value for conducting mixed methods research. For example, the
purpose statement created by Aime et al. (2014) states, ‘‘By combining a qualitative interview-
based study and a quantitative, laboratory-based study, we were able to improve the validity of
our study by countering the limitations and trade-offs inherent in each method’’ (p. 346). It is
recommended that authors identify both a general rationale for using mixed methods and a
more specific rationale related to the different mixed methods design types (Creswell & Clark,
2017). This will often reflect a need for approaching a research problem from multiple perspec-
tives. Two studies in our data set included mixed methods research questions that address the
integration, calling attention to both data strands (see Plano Clark & Badiee, 2010, for a more
detailed discussion of mixed methods research questions). For example, Detert and Edmondson
(2011) discussed goals in connecting their two studies.
In Study 2, we examine the generalizability of the implicit theories identified in Study 1. Our goal
is this study was not to develop an exhaustive taxonomy of all self-protective implicit voice theories
(hereafter, simply ‘‘implicit voice theories’’), but rather to confirm that those identified in Study 1
are not idiosyncratic to a single organization and are common enough to merit subsequent investi-
gation. (p. 462)
An additional aspect of the elements of writing includes the discussion of the value of mixed
methods. Mixed methods scholars are not often explicit about stating the value of the mixed
methods approach (Creswell & Clark, 2017). However, in our data set, McDermott et al. (2009)
specifically discussed the value of the mixed methods approach in their study of a firm’s ability
to upgrade their products. The authors state,
Although we do not claim to present definitive, linear causality, our research design combines the
strengths of comparative qualitative and statistical analyses to capture configurative causation—the
plausibility of certain policies’ reshaping the organizational and institutional factors that signifi-
cantly impact firm-level product upgrading. (p. 1271).
Harrison et al. 15

This presentation of value represents an understanding of mixed methods and adds rigor as a
discussion of the aims and purpose for utilizing the methodology.
The writing elements involve the inclusion of a mixed methods title, including joint displays
and citing mixed methods research. The title provides the focus for the entire project, and it is
recommended that mixed methods title include the words ‘‘mixed methods’’ to denote the meth-
odology being used (Creswell & Clark, 2017). Some may debate the importance of including a
mixed methods title; however, we feel it is important as a signaling device for reviewers, read-
ers, and editors of the type of research to follow, especially in disciplines where the technique is
not well-known. In addition, the title should use neutral language, thus words that convey a qua-
litative leaning should be avoided (e.g., explore, meaning, or discovery) as should words that
convey a quantitative leaning (e.g., relationship, correlation, or explanation). In our data set, it
was common for authors to cite methods literature related to qualitative or quantitative methods.
However, it was rare for them to cite mixed methods literature.
Another writing element is the inclusion of a joint display. A mixed methods joint display
arrays the quantitative and the qualitative results together for a comparison in a table or a graph
(Creswell & Clark, 2017). Joint displays are valuable in mixed methods research because they
can assist both the researcher and the reader in thinking about the integration of qualitative and
quantitative data. Joint displays may facilitate the cognitive process involved with mixing and
adding perspectives. For sequential designs, joint displays link qualitative and quantitative find-
ings by showing how the results from one phase proceeded to data collection in the subsequent
phase. In an explanatory design, the researcher determines what quantitative results need fur-
ther explanation and then conducts a qualitative follow-up phase (Creswell & Clark, 2017).
Similarly, in an exploratory sequential design, themes, codes, and quotes may be useful to
design items, variables, and scales (Creswell & Clark, 2017). In convergent designs, the pur-
pose of the display is to present the convergent and divergent findings side by side. While none
of the articles in our samples included joint displays, several articles included multiple primary
and advanced elements and had high levels of mixed methods rigor. We will now discuss two
such articles that employed a high level of mixed methods rigor. For each study, we will begin
with a summary of the study’s context before discussing the methodology.
Exemplar Study I: It’s Not Easy Being Green: The Role of Self-Evaluations in Explaining Support of
Environmental Issues. In this first study, Sonenshein et al. (2014) examine the role of self-
evaluations in influencing support for environmental issues. Research suggests that supporting
a social issue, whether it be climate change, gender/racial equality, corporate responsibility, or
other issues, can come with a cost to the individual’s career, personal endeavors, and family
commitments (Ashford et al., 1998). However, management articles do not adequately address
the multifaceted context in which social issue supporters operate, both inside and outside of
organizations. This study offers a richer view of both how contexts shape social issue support
and how individuals’ self-evaluations play a meaningful role in understanding the experiences
and, ultimately, the issue-supportive behaviors of individuals working on social issues.
The authors employed an exploratory sequential mixed methods research design. They first
used qualitative data to develop theory about how environmental issue supporters evaluate
themselves. They then quantitatively validated key constructs from that theory. They began the
research with a focus on understanding how issue supporters interpreted and framed social
issues. As is often the case in qualitative research, the authors mention, ‘‘Our research shifted
as we collected and analyzed data . . . and we adjusted our focus to self-evaluations, posing the
following emergent research question: How do issues supporters’ everyday experience influ-
ence their self-evaluations?’’ (Sonenshein et al., 2014, p. 9). In answering this question, the
16 Journal of Mixed Methods Research

authors developed a grounded theory of a process that featured two core constructs that emerged
from the data—self-assets and self-doubt. They describe their next steps as follows:
After unpacking issue support challenges, self-assets, and self-doubts using grounded theory, we
conducted a quantitative study using observational methods to inductively examine a second
research question. . . . This allowed for the presentation of two different but complementary studies
using mixed methods (e.g., (Creswell & Clark, 2011)) to examine both how social issue supporters’
everyday experiences influence their self-evaluations (Study 1) and why these self-evaluations mat-
ter through their ability to predict issue-related actions (Study 2). (p. 9)
The passage above highlights the author’s recognition of an advanced element of rigor (i.e., cit-
ing mixed method literature). The authors also demonstrate an understanding of rigorous quali-
tative analysis in the methodological discussion of the first study. In the first study, 29
interviews and two field observations were conducted and the qualitative data analysis followed
a three-step process that included (1) initial data coding, (2) developing theoretical categories,
and (3) theory induction (Strauss & Corbin, 1990). Furthermore, the authors offered a graphical
depiction of their data structure that included first-order categories, second-order themes, and
aggregate dimensions. In the second study, the authors created and validated items with a pret-
est that conceptualized and measured the emergent, or latent, constructs from Study 1 (i.e., self-
doubts and self-assets). They then collected more quantitative data via administered surveys
and conducting confirmatory factor analysis on all independent measures. They also conducted
a cluster analysis to examine how participants grouped into combinations of mixed selves based
on degree of self-assets and self-doubts, effectively combining rigorous quantitative analysis
with the rigorous qualitative analysis.
The authors linked the two studies by stating that ‘‘study 1 showed that informants interpret
being an issue supporter as posing severe challenges. These challenges have important implica-
tions for how social issue supporters stay motivated to continue to support an issue’’
(Sonenshein et al., 2014, p. 29). The authors discussed how Study 2 investigates these chal-
lenges. Linking the two data strands is part of the data integration process. The authors also dis-
cuss the value of the mixed methods design to begin their general discussion section:
Using two complementary studies, we developed a theory of situated self-work and then examined
how two core constructs in this theory—self-assets and self-doubts—related to real issue-supportive
behaviors. Collectively, these two studies present a view that is an alternative to that of organiza-
tional scholars who emphasize an outward-looking self-concerned with image and reputational risk
developed from contextual sensemaking inside organizations, as well as an alternative to psycholo-
gical perspectives that emphasize theories that downplay context (p. 30).
This article also specifically discussed the strengths of mixed methods for their study (i.e., sam-
pling from different field bases and presenting evidence of validated measures of emergent induc-
tive constructs). From a best practices perspective, this article highlights the primary elements of
Rigorous Mixed Methods as well as aims and purpose elements and elements of writing.
Exemplar Study II: Does Complexity Deter Customer-Focus? In this study, Ethiraj et al. (2012) iden-
tify a theoretical tension between economic models that suggest that firms use a simple cost–
benefit calculation to evaluate customer requests for new product features and the extensive
management literature that shows the decision to implement innovation as being more nuanced.
Specifically, this article examines how firms prioritize customer requests for incremental inno-
vation and how product complexity creates organizational constraints that alter firm’s incentive
to be customer focused.
Harrison et al. 17

The authors employed an explanatory sequential mixed methods research design (i.e., quan-
titative procedures followed by qualitative ones) to test theory-based hypotheses related to cus-
tomer demand for product innovation changes and the likeliness that incremental innovations
will occur at the firm level when the product is complex. First, they tested their hypotheses
using a panel data set of incremental innovation decisions at a single firm. Each time the firm
considered investing in an innovation requested by a customer, they collected data about the
decision-making process related to the decision to innovate (or not) in response to the customer
request. In all, the authors gathered and analyzed 120 customer requests, requiring incremental
innovation. They performed a statistical analysis, estimating a binary choice model (using pro-
bit equations) of the likelihood of customer request fulfillment, including three sets of predic-
tors—demand, complexity, and other controls—to account for alternative explanations. They
conclude the quantitative study with the following summary:
The large sample empirical analyses suggest that in the case of software products, complexity is
indeed an important driver of the decision to invest resources in incremental innovation. We find
that whereas customer demand is an important predictor of the decision to standardize a customer
request, it has little predictive power in the decision to fulfill the request. The important unanswered
question at the conclusion of the empirical analysis is what accounts for the observed results. What
kinds of managerial and/or organizational decision processes explain these empirical patterns? The
qualitative study in the following sections examines this question. (Ethiraj et al., 2012, p. 153)
It is in the quotation above that the authors provide a rationale for conducting mixed methods
research, utilizing an explanatory sequential design. The authors also provide a mixed methods
research question that asks what kinds of processes (understood through manager’s lived expe-
rience) explain the quantitative results.
The qualitative study involved in-depth interviews with heads of product development, prod-
uct management department, and marketing and sales staff. The focus of the qualitative study
was to understand the innovative decision-making process and how it might account for the
observed quantitative results. In sum, the qualitative study identified three underlying elements
of the decision-making process (or themes) that might prevent the demand driven innovation
(DDI) model from explaining investment in incremental innovation—the importance of organi-
zational structures, competitive pressures, and incentives for resource allocation processes.
Data integration occurred in the discussion section that summarizes the theoretical contributions
of the study. The quantitative results identified patterns related to whether or not to allocate
resources for product innovations and how firms implemented the investment. The qualitative
study revealed that fulfillment decisions were dependent on the firm’s decision-making pro-
cesses related to resource allocation. While the previous exemplar did a good job of connecting
the data between the qualitative and quantitative studies, this article demonstrates the inherent
value of mixing both data types. Overall, the authors were able to make stronger conclusions
by using the strengths of each monomethod design type. This study represents an explanatory
sequential design, one that highlights the primary elements of Rigorous Mixed Methods as well
as provides a rationale for the use of mixed methods.
These two exemplar articles demonstrate that Rigorous Mixed Methods research is being
done in management studies, but the overall data set suggests that the discipline as a whole
could benefit from the Rigorous Mixed Methods criteria previously presented. Each piece pre-
sented here had strengths that merited its inclusion in this study and allow for prospective
authors to model their work after existing, successful, publications. The following discussion
highlights the current state of mixed methods in management research and discusses ways in
which the field as a whole may move forward.
18 Journal of Mixed Methods Research

Discussion
Contribution to the Field of Mixed Methods Research Methodology
This article contributes to the field of mixed methods research by creating a framework for con-
ducting and reporting mixed methods studies in a rigorous manner. This varies from previous
work on conducting ‘‘high-quality’’ or ‘‘good’’ mixed methods as any debate about quality
should be viewed through the lens of how a researcher was trained in their academic discipline.
Researchers can disagree about what constitutes a high-quality article, yet this framework aims
to provide guidance for recognizing what qualifies as highly rigorous mixed methods. The dis-
cussion of Rigorous Mixed Methods procedures is a relatively new phenomenon and as such
has been mostly limited to the evaluation, education, and health science literatures. This article
advances this ongoing discussion by providing more evidence from a business discipline,
allowing for the discussion of discipline-specific influences that can affect the development of
a research method over time. One example of this is the relatively strong focus on quantitative
methods in the business literature (i.e., exploratory sequential designs with a quantitative
emphasis were most common in the data). This may be due to a general quantitative emphasis
in the business disciplines, as well as publishing trends within the journals reviewed. Further
work is needed to identify the ‘‘paradigmatic’’ lens with which mixed methods is viewed
within a given discipline, which may at some time allow for an overall discussion of mixed
methods quality, but that cannot happen until a common agreement on rigorous methods is
achieved.
Our findings also highlight patterns in Rigorous Mixed Methods across different journal out-
lets. For example, while the Academy of Management Journal and the Strategic Management
Journal both have approximately the same number of articles that combine qualitative and
quantitative data, the Academy of Management Journal had a higher percentage of rigorous arti-
cles. The Academy of Management Journal is considered one of the highest ranking manage-
ment journals; thus, perhaps the most rigorous articles in the discipline are more likely to be
published in this journal. While the Journal of Management included more articles, the journal
did not contain a study that rated medium–high or higher in Rigorous Mixed Methods. This
journal is relatively new and may be more rigid in its orientation to alternative methods.
Administrative Science Quarterly only had one study with such a rating. So how are readers to
interpret these findings about journals that offer limited examples of redundant Rigorous Mixed
Methods? Are they less oriented toward or less knowledgeable about the tenets of the approach?
Or, are there more systemic, political, or historical reasons for the results? Bazeley’s (2009)
review of management literature concluded that there is continuing prominence of quantita-
tively based, statistical approaches in management research. Furthermore, Hurmerinta-
Peltomaki and Nummela (2006) suggest that mixed methods research faces publication chal-
lenges due to paradigmatic views within the management discipline. Cameron (2011) suggests
that some management journals exclude mixed methods research, explicitly and implicitly
through methodological preferences. Other issues include page restrictions that limit how
exhaustive an article can be in terms of mixed methods rigor and still fit within the number of
pages allocated by journal editors. While we argue strongly for the usefulness and feasibility of
mixed methods research within a journal’s page constraints, we do acknowledge this as a weak-
ness of the method. With that said, one way to include methodological detail, in light of page
restrictions, could be to use appendices or online open access forums to present additional
material.
Overall, our findings reveal that Rigorous Mixed Methods articles are published in the man-
agement literature. In these articles, the authors collected and analyzed qualitative and
Harrison et al. 19

quantitative data, showed some integration of the two data strands, and often indicated that they
were thinking about the rationale for why they conducted mixed methods research.
Furthermore, the core components of Rigorous Mixed Methods were well represented in the
high-ranking articles. However, while our findings offer several best practice examples, a
majority of the articles found in this analysis would not rate as highly rigorous according to the
mixed methods literature and the Rigorous Mixed Methods framework. Overall, 65.6% of arti-
cles reviewed for this article qualified as having low mixed methods rigor. Again, these articles
are not ‘‘true’’ mixed methods studies by many definitions. This suggests significant room for
improvement in the utilization of mixed methods in the management discipline. Of the 195 arti-
cles, only 9.7% were deemed to have medium to high methodological rigor. This shows that
while highly Rigorous Mixed Methods work is being done, it has yet to become the norm.
Furthermore, of the articles in this analysis, only one referenced the extant mixed methods
literature, and only two were labeled as mixed methods articles. None of the articles identified
a specific mixed methods design type or included a joint display of the findings. Referencing
mixed methods literature and labeling articles as mixed methods give a study more sophistica-
tion and signal an understanding of the distinct methodological approach. Tapping into the
existing body of mixed methods literature could lead to more rigorous articles, as authors
become aware of the challenges and benefits of established approaches. Citing relevant mixed
methods literature also helps editors identify manuscript reviewers. Utilizing a specific research
design opens up the possibility of knowing where the challenge points exist in the different
mixed methods approaches. Identifying the design type also affects the procedures that unfold,
as mixed methods scholars often match the writing structure, the mixed methods research ques-
tion, the title, and the specific design type, because design types become central to tying
together various structural elements (Creswell & Clark, 2017). The presentation of joint dis-
plays and visual methodological diagrams is a selling point for stakeholders, research teams,
and journal reviewers because it facilitates an understanding of the project, particularly impor-
tant in the practitioner field like management. Overall, these issues show the value and need for
the Rigorous Mixed Methods framework proposed in this article.
So how can authors enhance the level of mixed methods rigor in the field of management
studies? Clearly, this can be done by improving empirical projects by adding in Rigorous
Mixed Methods elements that were missing in some of the articles. As the mixed methods liter-
ature develops in management studies, it is likely that the discipline will begin to see more sys-
tematic initiatives to enhance mixed methods research rigor and, in turn, quality as appreciated
by management scholars. This may include emboldening journal editors to open up to mixed
methods, as it appears that some journals are much more receptive than others. Academic con-
ferences, panels, and workshops on mixed methods may also foster the debate necessary to
introduce or change thoughts about what are rigorous research methods. Other initiatives or
opportunities that can potentially ignite institutional change include the development of books
tailoring mixed methods to the management context. What makes specific mixed methods
design types particularly useful for management scholars? Several disciplines have developed
texts that address contextualized issues related to both methodological rigor and design, adapted
specifically for scholars in their discipline. Management studies have the opportunity to follow
suit. Regardless of the approach, it is not a revolutionary change that will improve the rigorous-
ness of management studies—but an evolutionary one. As mixed methods researchers continue
to discuss issues related to improving the use of the method, we have the opportunity to change
mind-sets related to the value and rigorousness of the approach, while also providing guidance
for editors, reviewers, and researchers.
20 Journal of Mixed Methods Research

Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or pub-
lication of this article.
Funding
The author(s) received no financial support for the research, authorship, and/or publication of this article.
ORCID iD
Robert L. Harrison https://orcid.org/0000-0002-9759-3126
Note
1. As noted in their 2009 article, Pluye et al. define an MSR as ‘‘a form of mixed methods research where the
data is primarily the text of publications reporting qualitative findings and quantitative results of primary
empirical studies.’’ (p. 532) Pluye et al. offer a scoring system for MSRs, and while there is overlap in the
reporting system suggested, an MSR is much closer to a meta-analysis guideline than a rigor framework.
References
Aime, F., Humphrey, S., DeRue, D. S., & Paul, J. B. (2014). The riddle of heterarchy: Power transitions
in cross-functional teams. Academy of Management Journal,57(2), 327-352. https://doi.org/10.5465/
amj.2011.0756
Almandoz, J. (2012). Arriving at the starting line: The impact of community and financial logics on new banking
ventures. Academy of Management Journal,55(6), 1381-1406. https://doi.org/10.5465/amj.2011.0361
Ashford, S. J., Rothbard, N. P., Piderit, S. K., & Dutton, J. E. (1998). Out on a limb: The role of context
and impression management in selling gender-equity issues. Administrative Science Quarterly,43(1),
23-57. https://doi.org/10.2307/2393590
Battilana, J., & Casciaro, T. (2012). Change agents, networks, and institutions: A contingency theory of
organizational change. Academy of Management Journal,55(2), 381-398. https://doi.org/10.5465/
amj.009.0891
Bazeley, P. (2009). Integrating data analyses in mixed methods research [Editorial]. Journal of Mixed
Methods Research,3(3), 203-207. https://doi.org/10.1177/1558689809334443
Bernstein, E. S. (2012). The transparency paradox a role for privacy in organizational learning and
operational control. Administrative Science Quarterly,57(2), 181-216. https://doi.org/10.1177/
0001839212453028
Bryman, A. (2006). Integrating quantitative and qualitative research: How is it done?Qualitative Research,
6(1), 97-113. https://doi.org/10.1177/1468794106058877
Bunderson, J. S., & Thompson, J. A. (2009). The call of the wild: Zookeepers, callings, and the double-
edged sword of deeply meaningful work. Administrative Science Quarterly,54(1), 32-57. https:
//doi.org/10.2189/asqu.2009.54.1.32
Cameron, R. (2011). Mixed methods in business and management: A call to the ‘‘first generation.’’Journal
of Management & Organization,17(2), 245-267. https://doi.org/10.1017/S1833367200001644
Cameron, R., Sankaran, S., & Scales, J. (2015). Mixed methods use in project management research.
Project Management Journal,46(2), 90-104. https://doi.org/10.1002/pmj.21484
Creswell, J. W., & Clark, V. L. P. (2011). Designing and Conducting Mixed Methods Research (2nd ed.).
Sage publications.
Creswell, J. W., & Clark, V. L. P. 2017. Designing and Conducting Mixed Methods Research. Sage
publications.
Davies, G., Chun, R., & Kamins, M. A. (2010). Reputation gaps and the performance of service
organizations. Strategic Management Journal,31(5), 530-546. https://doi.org/10.1002/smj.825
Detert, J. R., & Edmondson, A. C. (2011). Implicit voice theories: Taken-for-granted rules of self-
censorship at work. Academy of Management Journal,54(3), 461-488. https://doi.org/10.5465/
amj.011.61967925
Harrison et al. 21

Ethiraj, S. K., Ramasubbu, N., & Krishnan, M. S. (2012). Does complexity deter customer-focus?Strategic
Management Journal,33(2), 137-161. https://doi.org/10.1002/smj.947
Fa
`bregues, S., Pare
´, M. H., & Meneses, J. (2018). Operationalizing and conceptualizing quality in mixed
methods research: A multiple case study of the disciplines of education, nursing, psychology, and
sociology. Journal of Mixed Methods Research,13(4), 424-445. https://doi.org/10.1177/
1558689817751774
Gibson, C. B. (2017). Elaboration, generalization, triangulation, and interpretation: On enhancing the
value of mixed method research. Organizational Research Methods,20(2), 193-223. https://doi.org/
10.1177/1558689817751774
Graffin, S. D., Carpenter, M. A., & Boivie, S. (2011). What’s all that (strategic) noise? Anticipatory
impression management in CEO succession. Strategic Management Journal,32(7), 748-770. https:
//doi.org/10.1002/smj.906
Grant, A., Berg, J., & Cable, D. (2013). Job titles as identity badges: How self-reflective titles can reduce
emotional exhaustion. Academy of Management Journal,57(4), 1201-1225. https://doi.org/10.5465/
amj.2012.0338
Greene, J. (2006). Toward a methodology of mixed methods social inquiry. Research in the Schools,13(1),
93-98. https://doi.org/10.1177/1558689807309969
Greene, J. C., Caracelli, V. J., & Graham, W. F. (1989). Toward a conceptual framework for mixed-
method evaluation designs. Educational Evaluation and Policy Analysis,11(3), 255-274. https://doi
.org/10.3102/01623737011003255
Harrison, R. L. (2013). Using mixed methods designs in the Journal of Business Research, 1990–2010.
Journal of Business Research,66(11), 2153–2162.
Harrison, R. L., & Reilly, T. M. (2011). Mixed methods designs in marketing research. Qualitative Market
Research: An International Journal,14(1), 7–26.
Hiemeriks, K. H., Schijven, M., & Gates, S. (2012). Manifestations of higher-order routines: The
underlying mechanisms of deliberate learning in the context of postacquisition integration. Academy of
Management Journal,55(3), 703-726. https://doi.org/10.5465/amj.2009.0572
Hong, Q. N., & Pluye, P. (2018). A conceptual framework for critical appraisal in systematic mixed
studies reviews. Journal of Mixed Methods Research,13(4), 446-460. https://doi.org/10.1177/
1558689818770058
Hurmerinta-Peltomaki, L., & Nummela, N. (2006). Mixed methods in international business research: A
value-added perspective. Management International Review,46, 439-459. https:
//doi.org/10.1007/s11575-006-0100-z
Johnson, J. S. (2015). Broadening the application of mixed methods in sales research. Journal of Personal
Selling & Sales Management,35(4), 334-345. https://doi.org/10.1080/08853134.2015.1016953
Johnson, R. B., Onwuegbuzie, A. J., & Turner, L. A. (2007). Toward a definition of mixed methods
research. Journal of Mixed Methods Research,1(2), 112-133. https://doi.org/10.1177/
1558689806298224
Jonsson, S., & Regne
´r, P. (2009). Normative barriers to imitation: Social complexity of core competences
in a mutual fund industry. Strategic Management Journal,30(5), 517-536. https:
//doi.org/10.1002/smj.739
Kistruck, G. M., Sutter, C. J., Lount, R. B., & Smith, B. R. (2013). Mitigating principal-agent problems in
base-of-the-pyramid markets: An identity spillover perspective. Academy of Management Journal,
56(3), 659-682. https://doi.org/10.5465/amj.2011.0336
Kolbe, R. H., & Burnett, M. S. (1991). Content-analysis research: An examination of applications with
directives for improving research reliability and objectivity. Journal of Consumer Research,18(2),
243-250. https://doi.org/10.1086/209256
Leech, N. L., Dellinger, A. B., Brannagan, K. B., & Tanaka, H. (2010). Evaluating mixed research
studies: A mixed methods approach. Journal of Mixed Methods Research,4(1), 17-31. https://doi.org/
10.1177/1558689809345262
McDermott, G. A., Corredoira, R. A., & Kruse, G. (2009). Public-private institutions as catalysts of
upgrading in emerging market societies. Academy of Management Journal,52(6), 1270-1296. https:
//doi.org/10.5465/amj.009.47084929
22 Journal of Mixed Methods Research

Molina-Azorin, J. F. (2011). The use and added value of mixed methods in management research. Journal
of Mixed Methods Research,5(1), 7-24. https://doi.org/10.1177/1558689810384490
Molina-Azorin, J. F., & Cameron, R. (2015). History and emergent practices of multimethods and mixed
methods in business research. In S. Hesse-Biber & B. Johnson (Eds.), Oxford handbook of multimethod
and mixed methods research inquiry (pp. 466-485). Oxford University Press.
Mors, M. L. (2010). Innovation in a global consulting firm: When the problem is too much diversity.
Strategic Management Journal,31(8), 841-872. https://doi.org/10.1002/smj.837
Morse, J. M., & Niehaus, L. (2009). Mixed method design: Principles and procedures. Left Coast Press.
O’Cathain, A. (2010). Assessing the quality of mixed methods research: Toward a comprehensive
framework. Handbook of Mixed Methods in Social and Behavioral Research, 531-555.
O’Cathain, A., Murphy, E., & Nicholl, J. (2008). The quality of mixed methods studies in health services
research. Journal of Health Services Research & Policy,13(2), 92-98. https://doi.org/10.1258/
jhsrp.2007.007074
Onwuegbuzie, A. J. (2012). Introduction: Putting the MIXED back into quantitative and qualitative
research in educational research and beyond—Moving toward the radical middle. International Journal
of Multiple Research Approaches,6(3), 192-219. https://doi.org/10.5172/mra.2012.6.3.192
Onwuegbuzie, A. J., & Johnson, R. B. (2006). The validity issue in mixed research. Research in the
Schools,13(1), 48–63.
Plano Clark, V. L., & Badiee, M. (2010). Research questions in mixed methods research. In A. Tashakkori &
C. Teddlie (Eds.), SAGE handbook of mixed methods in social & behavioral research (pp. 275-304). Sage.
Pluye, P., Gagnon, M. P., Griffiths, F., & Johnson-Lafleur, J. (2009). A scoring system for appraising
mixed methods research, and concomitantly appraising qualitative, quantitative and mixed methods
primary studies in mixed studies reviews. International Journal of Nursing Studies,46(4), 529-546.
https://doi.org/10.1016/j.ijnurstu.2009.01.009
Raffaelli, R., & Glynn, M. A. (2014). Turnkey or tailored? Relational pluralism, institutional complexity,
and the organizational adoption of more or less customized practices. Academy of Management
Journal,57(2), 541-562. https://doi.org/10.5465/amj.2011.1000
Sonenshein, S., DeCelles, K., & Dutton, J. (2014). It’s not easy being green: The role of self-evaluations
in explaining support of environmental issues. Academy of Management Journal,57(1), 7-37.
https://doi.org/10.5465/amj.2010.0445
Strauss, A., & Corbin, J. M. (1990). Basics of qualitative research: Grounded theory procedures and
techniques (2nd ed.). Sage.
Tashakkori, A., & Teddlie, C. B. (2003). Handbook of mixed methods in social & behavioral research.
Sage.
Turner, S. F., Cardinal, L. B., & Burton, R. M. (2017). Research design for mixed methods: A
triangulation-based framework and roadmap. Organizational Research Methods,20(2), 243-267. https:
//doi.org/10.1177/1094428115610808
Van Wijk, J., Stam, W., Elfring, T., Zietsma, C., & Den Hond, F. (2013). Activists and incumbents
structuring change: The interplay of agency, culture, and networks in field evolution. Academy of
Management Journal,56(2), 358-386. https://doi.org/10.5465/amj.2008.0355
Vergne, J. P. (2012). Stigmatized categories and public disapproval of organizations: A mixed methods
study of the global arms industry (1996-2007). Academy of Management Journal,55(5), 1027-1052.
https://doi.org/10.5465/amj.2010.0599
Wisdom, J. P., Cavaleri, M. A., Onwuegbuzie, A. J., & Green, C. A. (2012). Methodological reporting in
qualitative, quantitative, and mixed methods health services research articles. Health Services
Research,47(2), 721-745. https://doi.org/10.1111/j.1475-6773.2011.01344.x
Yang, H., Phelps, C., & Steensma, H. K. (2010). Learning from what others have learned from you: The
effects of knowledge spillovers on originating firms. Academy of Management Journal,53(2),
371-389. https://doi.org/10.5465/amj.2010.49389018
Harrison et al. 23