Content uploaded by Florian G. Kern
Author content
All content in this area was uploaded by Florian G. Kern on Aug 16, 2017
Content may be subject to copyright.
!
1
TITLE PAGE:
The Trials and Tribulations of Applied Triangulation:
Weighing Different Data Sources
Florian G. Kern
University of Essex
fkern@essex.ac.uk
Wivenhoe Park
Colchester CO4 3SQ, UK
+44 1206 87-2745
Abstract
Triangulation is employed to increase the validity of inference in qualitative and quantitative re-
search. Leuffen, Shikano, and Walter have presented guidance as to which strategies to use when
triangulating data sources. In this paper, I ask: how can their findings be translated for practical
research purposes? I offer an illustrative application concentrating on the political power of tradi-
tional political authorities in Uganda and Tanzania. I analyze the status quo of political power and
the preferred political power of traditional leaders. To triangulate, I use three sources: (1) consti-
tutional-legal texts; (2) the Afrobarometer survey; and (3) in-depth interviews. I shed light on
possible problems and analytical strategies for triangulation in practice, with a specific focus on
convergence and divergence of sources.
Keywords triangulation, aggregation, divergence, traditional political authorities, Sub-Saharan
Africa
7.975 words
Acknowledgments: This study has been partially funded by the German Academic Exchange Ser-
vice and the German Foundation for Peace Research (Grant No. PA 003/11-Nr. 005/12-2010). I
thank Federica Genovese, Katharina Holzinger, Daniela Kromrey, Rob Johns, Dirk Leuffen, and
the anonymous referees for their invaluable critique and comments. I am indebted to Marius
Bayer and our project assistants for their support in the collection of the interview data. The Cen-
ter on Democracy, Development, and the Rule of Law (CDDRL) and the Department of Political
Science at Stanford University provided an inspiring environment for this research.
!
1
The Trials and Tribulations of Applied Triangulation:
Weighing Different Data Sources
Abstract
Researchers employ triangulation to increase the validity of inference in qualitative and quantita-
tive research. Leuffen, Shikano, and Walter have presented guidance as to which strategies to use
when triangulating data sources. In this paper, I ask: how can their findings be translated for prac-
tical research purposes? I offer an illustrative application concentrating on the political power of
traditional political authorities in Uganda and Tanzania. I analyze the status quo of political pow-
er and the preferred political power of traditional leaders. To triangulate, I use three sources: (1)
constitutional-legal texts; (2) the Afrobarometer survey; and (3) in-depth interviews. I shed light
on possible problems and analytical strategies for triangulation in practice, with a specific focus
on convergence and divergence of sources.
!
2
Triangulation is a strategy in qualitative and quantitative research to increase the validity
of inference by combining “varieties of data, investigators, theories, as well as methodol-
ogies” in the study of the same phenomena (Denzin, 1978, 295).1 Further, triangulation
may reveal both convergence and divergence of different measures, and thereby strength-
ens the understanding of a variable of interest (Hesse-Biber, 2010). The idea of triangula-
tion is not new to the social sciences (see e.g. Blaikie, 1991; Denzin, 1978; Jick, 1979;
King, Keohane, & Verba, 1995; Tarrow, 1995). However, especially with the more recent
increased focus on mixed-methods approaches, triangulation has re-gained attention in
academic fora (see Archibald, 2015; Howe, 2012; Denzin 2012; Johnson, Onwuegbuzie,
& Turner, 2007). To date, though, practical advice on how to triangulate is scarce.
Recently, Leuffen, Shikano, and Walter (2013) have offered guidance by compar-
ing the inferential leverage of five different strategies of triangulation. The authors show
that
collecting more information and using all of this information, weighted by the
quality of the source typically leads to better measurement results. However, this
only holds under the assumption that the sources are not systematically biased. (p.
49)
Yet, their results are drawn from computer simulations. Those of us evaluating
more unstructured data sources may wonder how in practice to weight and triangulate
data from very different sources such as surveys, in-depth interviews, and texts. In this
paper, I take the suggestions from Leuffen et al. (2013) and ask: how can their advice be
implemented? And what are related potentials and challenges?
!
3
Therefore, I offer an illustrative application of Leuffen et al. (2013), using my
own research into the political power of traditional political authorities (TPA) in Uganda
and Tanzania. I focus on two types of concepts related to the analysis of preferences – the
status quo of political power and the preferred level of political power of TPA. To trian-
gulate, I use three data sources: (1) constitutional and legal texts, (2) the Afrobarometer
survey (2009),2 and (3) in-depth interviews with actors and experts I conducted in
2012/13. I show that, to produce valid inference, when triangulating scholars need to as-
sess both convergence and divergence of different sources.
In the following, I first review the academic discussion on triangulation, and out-
line the argument of Leuffen et al. (2013). Second, I describe how I test the identified
strategies of triangulation. Then, I describe the three data sources and measure the two
variables for each source on a common scale. Subsequently, I discuss the practical pro-
cess of triangulation, assessing the applicability of the guidance from Leuffen et al.
(2013) and others. Finally, I summarize the findings in the conclusion.
Triangulation: Concept, Purpose, and Practice
The concept and purpose of triangulation
Some authors define triangulation narrowly as the combination of qualitative and quanti-
tative methodology (see e.g. Tarrow 1995; Erzberger & Prein 1997). Yet, more generally,
triangulation entails the analysis of the same event, concept, or variable by combining
several different angles or perspectives (King et al., 1995; Marks, 2007). Denzin (1978,
p. 295ff.) distinguishes four categories of triangulation, of which two are of particular
importance for the analysis below: (1) data triangulation, as I combine text, survey, and
!
4
interview data; and (2) methodological triangulation, as I combine text analysis, descrip-
tive survey measures, and software-based qualitative inference.3 Denzin further subdi-
vides data triangulation as combining analytical angles across time, space, or person,
while methodological triangulation is subdivided into within-method and between-
method triangulation. In addition, Morse (1991, 2010) further distinguishes two types of
methodological triangulation – simultaneous and sequential. In the former, two methods
are used at the same time; in the latter the inference gained from one method informs the
use of a second method (see also Johnson & Onwuegbuzie, 2004).
While some are critical about its potentials (Blaikie 1991; Jick, 1978), triangula-
tion embraces the idea that a combination of perspectives will (a) enhance the under-
standing of the variable of interest, and/or (b) increase the validity of the overall findings,
even if each angle comes with a different bias (Stoker, 2011). As Marks (2007) points
out, if measurements from several perspectives are considered jointly, and if the meas-
urement techniques are differently biased, this lessens systematic measurement error
(2007; see similar in Stoker, 2011; Turner & Turner, 2009). Thus, when more sources of
information are used to measure the same phenomenon, convergence of the different
measurements on the same value can be said to approximate the true value of a variable
of interest (Fielding, 2012; King et al., 1995).
The practice of triangulation
Triangulation itself does not guarantee convergence of measurements. Different angles
may show considerable variance in evaluating the same phenomenon. As Moran-Ellis et
al. (2006) point out, some authors attribute this divergence to invalid measurement. Oth-
!
5
ers emphasize the gain in conceptual understanding of the variable of interest when trian-
gulated sources diverge – even if variance in measurement may mean reformulation of
concepts and/or repetition of measurement (see similar in Hesse-Biber, 2010). Diver-
gence can also be amplified through triangulation in case of varying “epistemological
stances” of triangulating investigators (cf. Archibald, 2015, 16). Yet, the disagreement
and debating about a measurement of a variable of interest may at least help reduce the
range of possible values wherein to find a “true” value. To capture both convergence and
divergence, it is all the more important to practice triangulation according to clear and
reproducible rules (Denzin, 2010; Johnstone 2007).
Five aggregation strategies for triangulation
Leuffen et al. (2013) provide practical guidance by discussing the strategies of triangula-
tion. They assess which approaches perform best under which conditions. The authors
distinguish five strategies for aggregating information when triangulating three hypothet-
ical sources: (1) random selection, in which the value provided by one of the sources is
taken arbitrarily to measure the variable of interest; (2) arithmetic mean, in which the
values provided by all sources are averaged to measure the variable of interest; (3) major-
ity strategy, in which the value that the majority of sources agree on is taken to measure
the variable of interest; (4) weighted average, in which the values provided by all sources
are averaged and weighted by the quality of each source’s information to measure the
variable of interest; and (5) winner takes it all, in which only the most reliable source is
taken to measure the variable of interest.
Employing computer-simulations, the authors show that using maximum infor-
!
6
mation through the ‘weighted average’ strategy generally leads to the most accurate find-
ings. If the sources are all systematically biased in the same direction, the ‘winner takes it
all’ strategy is the most accurate. In sum, the choice for triangulation strategies relies on
“the number of sources, their respective trustworthiness, and their level of independence”
(Leuffen et al., 2013, p. 49). The authors offer one of the few explicit evaluations of dif-
ferent triangulation techniques and identify best-performing strategies. Scholars may po-
tentially use the authors’ findings as heuristic guidance of their own applied triangulation.
Yet, Leuffen et al. (2013) do not explain how to calculate an average (weighted or not)
over different sources stemming from, for instance, survey data, in-depth interviews, and
text data. Thus, below I provide an extension of the authors’ argument, showing how to
weigh data sources for triangulation in applied mixed methods research.
Applied Triangulation in Preference Analysis
Research design
In order to apply the five types of aggregation mechanism for data triangulation, I use the
measurement of concepts related to political preferences (in analogy to Leuffen et al.,
2013). More specifically, I concentrate on the topic of political power of traditional polit-
ical authorities (TPA) in contemporary Uganda and Tanzania. TPA are, for example,
chiefs, elders, and headmen: that is, community leaders whose authority is understood
and validated through narratives or procedures deemed “traditional” by constituents. The
term “traditional” is not equivalent to ancient here, but refers to a mode of legitimization
of authorities. In many countries in Sub-Saharan Africa, TPA still play an important role
in domestic politics – locally and, in some cases, nationally (Baldwin, 2013; Holzinger,
!
7
Kern, & Kromrey, 2016; Logan, 2013). Because the politics involving TPA are often in-
formal, data on the topic are sparse and often of poor quality. Thus, data triangulation is
especially useful in studying TPA, allowing the validation of information gathered using
field research and the limited additional data available.
In many of the communities in which TPA are still important, there is a tension
between the actual political power of traditional leaders on the ground and the level of
power that various actors – whether constituents, state actors or the TPA themselves –
would like these traditional leaders to wield. Therefore, I triangulate different data
sources to assess both assessments of the status quo (the level of political power currently
held by TPA) and preferences (about the level of political power that TPA should hold)
in Uganda and Tanzania. Given that the purpose of this paper is to make a methodologi-
cal point about the practice of triangulation, I solely focus on the scaling and measure-
ment of the status quo and preference. Holzinger, Kern, and Kromrey (2015) provide an
in-depth analysis of TPA in Uganda and Tanzania .
The variables of interest: status quo and preferred power
Several authors argue that to maximize the benefit from triangulation sources should be
triangulated against the same theoretical or empirical construct (cf. Denzin, 1978; Stoker,
2011). While it is arguably difficult to produce exactly equivalent measures from differ-
ent data sources (cf. Fielding, 2012), the measure should be constructed so as to serve as
a triangulation target from many different angles. Hence, I construct a simple scale to
measure the status quo and the preferred level of political power of TPA in both coun-
tries. The scale bears strong similarities to the one used in the Afrobarometer survey
!
8
(Round 4, see below). Panel A and panel B in Figure 1 show the scales for the two varia-
bles of interest.
Figure 1 about here.
The ordinal steps of the scale are defined as follows: The political power of TPA
in a country is “none” if TPA have no political leverage whatsoever; “small” power refers
to cultural recognition of TPA only, without any political influence – cultural here refers
to rituals, ceremonies, or the maintenance of heritage; “some” power signifies that there
is not only cultural influence but also a political role played by TPA, e.g. responsibilities
for land or dispute management on the local level, as well as influence in shaping elec-
toral outcomes; finally, “large” power indicates a major political role in the country, e.g.
with TPA occupying national level institutions or taking over classic state activities such
as the provision of public goods.
As Figure 2 shows for three hypothetical Sources A, B, and C, all three different
data sources will be triangulated against this scale to measure the variables of interest. If
sources cannot be scaled to a unique point with certainty, I add a “–” or “+”-sign for the
scaling, indicating a tendency to a lower or higher scaling (illustrated by the directional
‘flags’). One illustrative example in Figure 2 shows three sources converging on the val-
ue “some” (line arrows). However, Figure 2 also shows a scenario without convergence:
Source C is scaled as “large–”, and Source A is scaled as “none” (dotted arrows). Espe-
cially for the divergent scenarios, weighting the different sources becomes important.
Figure 2 about here.
!
9
Data Sources
Data source 1: Constitutional and legal texts
I examine present constitutions and specific legal acts in order to identify rules that give
TPA political power. To be sure, these texts can only be used to assess the status quo, be-
cause preferences are not expressed in these documents. Table A.1 in the appendix lists
the constitutional and legal texts by country. In Uganda, Chapter 6, 16, and article 178 of
the current 1995 constitution define the role of TPA. Chapter 6 (Art. 80) disqualifies rec-
ognized traditional leaders from becoming members of parliament. Chapter 16 recognizes
TPA as community leaders. Still, the document establishes the primacy of state authority
over TPA (Art. 246 (4)). The “Institution of Traditional and Cultural Leaders Act” of
2011 acknowledges the traditional selection procedures of TPA by the constituency. New
appointments of TPA must be reported to the ministry. Traditional leaders shall ensure
the cultural preservation and development of their community (Art. 9). They are allowed
to resolve disputes in their own community, but do “not have or exercise any administra-
tive, legislative or executive powers of Government or local government” (Art. 12). They
cannot participate in party politics, unless they give up their traditional mandate and the
benefits coming with that position (e.g. official cars or tuition scholarships for natural
children). Given these constitutional provisions and the subsequent 2011 Act, I scale TPA
as having “some” power in Ugandan politics.
For Tanzania, I scale the status quo of political power of TPA as “none”. The
“Local Government Ordinance (Amendment) Act” (1962), the “African Chiefs Ordi-
nance (Repeal) Act” (1963), and the “African Chiefs Act” (1969) abolished all powers of
!
10
TPA. The 1977 constitution mentions TPA only indirectly, emphasizing that lineage or
tradition are not to give any advantage to an individual (§29.-1). The Tanzanian “Village
Land Act” (1999) refers to TPA but only in that a past agreement on land use between
two villages reached by traditional leaders can be adopted as a joint land use agreement
(§11.-(1)). This effectively reduces the power of TPA and transfers authority to the ad-
ministrative village land councils. Similar to the constitution, the 2002 “Courts (Land
Disputes Settlement) Act” does not mention TPA and refers back to the 1999 act.
Data source 2: Afrobarometer survey
I draw on data from the fourth round of the Afrobarometer collected in 2008/2009 in 19
countries, as it is the only round with a series of questions evaluating the political powers
of TPA and including data for Uganda and Tanzania. Regarding the status quo of politi-
cal power of TPA, Table 1 summarizes the responses to the question “How much influ-
ence do traditional leaders currently have in governing your local community?”. The col-
umns show the aggregate country-level shares for each response category for Uganda and
Tanzania.
Table 1 about here.
Regarding Tanzania, over 60 percent of respondents assess the political power of
TPA as either small or none. Of all countries surveyed, Tanzania displays the largest pro-
portion of respondents indicating no influence of TPA. Still, a fifth of respondents rate
the influence as “some” or higher. Thus, I scale the status quo of political power of TPA
as “small-”.
!
11
In Uganda, a larger share of the respondents than in Tanzania state that TPA have
“some” or “a great deal” of political influence. A slightly larger share of 46 percent sees
no or only a small influence. Yet additional Afrobarometer data show that 61 percent of
respondents express that they trust traditional leaders somewhat or even “a lot”. I there-
fore scale the status quo political power of TPA in Uganda as “some-”.
Regarding the preferred level of political power of TPA, I juxtapose shares of re-
sponses to the question “Do you think that the amount of influence traditional leaders
have in governing your local community should increase, stay the same, or decrease?”
(Q2) to the response shares of the previous question (Q1) (see cross-tabulations in Table
A.2 in the appendix). For Uganda, a majority of 59 percent wants the political influence
of TPA to increase at least somewhat or a lot. Furthermore, a quarter of Ugandan re-
spondents assess the status quo influence of TPA as “none” and prefer this to “stay the
same”. However, a large majority of those indicating the status quo as small or higher
prefer to see the political influence of TPA increase. Having previously scaled the status
quo of political power of TPA as “some-”, I scale the preferred level of political power of
TPA in Uganda as “large-”.
For Tanzania, for Q2 35 percent prefer the governing influence of traditional
leaders to increase, while only 37 percent prefer a decrease or the status quo. A sum of 55
percent of those Tanzanians who designate the status quo as “none” prefer that state of
affairs o or would like political influence somehow to decrease further. 42 percent of
those who assess the status quo influence as “small” would like it to stay that way or to
decrease. As the majority of those who answered “some” or “great” for the status quo
prefer political power to stay the same or increase, and having previously determined the
!
12
status quo as “small-”, I scale the preferred level of political power of TPA in Tanzania
as “small+”.
Data source 3: In-depth interviews
In 2012/2013, I conducted a series of qualitative interviews on issues involving TPA in
Uganda and Tanzania. The respondents were actors such as state authorities and TPA,
e.g. a former prime minister of Uganda or traditional chiefs. Furthermore, I interviewed
experts, civil society actors, and members of the population with knowledge of the politi-
cal significance of TPA for a larger collaborative project on the political power of TPA.
Overall, I conducted 66 interviews – 35 in Uganda and 31 in Tanzania. Table 2 summa-
rizes the total number of interviews by country and respondent categories – state actors
(actS), traditional actors (actT), and experts/civil society (exp).
Table 2 about here.
In each country the interviews focused on two ethnic groups with a view to meas-
uring whether the form of the traditional polity’s organization affects the significance of
TPA. For Uganda, we selected the Baganda, whose polity is organized as a kingdom, and
the Iteso, who traditionally organize in more horizontal clan systems; for Tanzania, we
selected the Sukuma, who organize in decentralized chiefdoms, and the Maasai, who or-
ganize based on an age-set system (see Table A.3 in the appendix for respondents by
country and group association).
Interview approach and data analysis
To measure the status quo of political power of TPA we asked each respondent: “How
!
13
significant do you think are traditional leaders in today’s politics in [COUNTRY]?”, and
the follow-up item “Please name the most significant areas.” Further questions provide
complementary information – e.g on the relations between TPA and the state. As for the
preferred level of political power of TPA, we asked for knowledge about the constitu-
tional-legal context of TPA: “Have traditional leaders been granted legal rights by the
state?/In which areas?”. Then, a follow-up item asked “If yes/no, should these rights be
increased, stay the same, or be decreased?/In which areas?/Why?”. In explaining their
reasoning, respondents expressed what level of power they would like TPA to have in
their country.
I categorized responses according to the four-step scale using the R-package RQDA (R-
based Qualitative Data Analysis, Huang, 2012). Thick description in the interviews
makes scaling into distinct categories more difficult, but the “-/+”-scaling provides some
flexibility. Table A.4 and A.5.1-4 (see Appendix) give typical examples for the scaling of
statements by respondent group for each country.
Source scaling
For Uganda, I scale the state actors’ status quo position as “some” and the preferred polit-
ical power of TPA as “small”. State respondents frequently explain that the cultural role
of TPA as defined by the Ugandan constitution does not match the actual role that TPA
play in shaping local politics. State actors claim that TPA heavily influence politics and
demand obedience from constituents. Moreover, state actors would prefer TPA to retain a
small influence as custodians of culture only, without them holding substantial adminis-
trative powers – although they admit that TPA might be useful in mobilization for gov-
!
14
ernment policies.
I scale the traditional actors’ perceived status quo position as “small+”, and the
preferred political power as “some+”. Both Buganda Kingdom and Iteso Cultural Union
(ICU) authorities express that the constitution determines their powers to be only cultural.
Then again, Iteso clan leaders and Buganda authorities claim to provide a number of
small-scale services (e.g. dispute resolution). Yet their financial leverage is small, ham-
pering the implementation of larger policies. As for the preferred position, the Buganda
authorities in particular emphasize their wish for a federal arrangement in which the Bu-
ganda administration would take over governing powers. Iteso leaders would like to see
more funding and administrative powers flowing from the state to TPA, and claim that
the Buganda Kingdom’s position is driving other traditional leaders’ preference for an
increased administrative role.
I scale the experts’ position regarding the status quo of TPA political power as
“some-”. Most experts point to the cultural role of TPA and emphasize the dominant po-
sition of the Buganda Kingdom, as well as the local role of TPA in service provision (e.g.
land management, dispute resolution). However, experts underline that, while influential,
the elites of the Baganda and Iteso do not play a role in functional service provision. Ac-
cording to the interviewed experts, political authority for TPA should be more formalized
in areas where they have real expertise. None of the experts expressed a preference for
TPA to hold either no power at all or major administrative power. I therefore scale the
preferred level of political power for experts as “some+”.
For Tanzania, I scale the state actors’ status quo position as “small-” and the pre-
!
15
ferred political power of TPA as “none”. State actors pointed to how insignificant TPA
have become since the abolition in the 1960s. However, regarding the rural level, re-
spondents admitted that, for policies to be effective, TPA where they exist can be used as
intermediaries. The majority of state respondents also feared that a stronger influence of
TPA may create a culture of tribalism. Respondents preferred the legal status quo where-
by TPA hold no political power.
The traditional actors’ perceived status quo position was scaled as “small”, and
the preferred political power as “some-”. For the Sukuma, traditional leaders defined their
own role as “cultural”, i.e. maintaining rituals and cultural activities. TPA described their
minimal powers, e.g. in advising state administrators about land distribution. The inter-
viewees admitted that constituents turn to the state for service provision. Yet, Sukuma
TPA expressed their wish to be recognized for the services they could provide given their
local knowledge.
The position of experts is scaled as “none+” for both the status quo and the pre-
ferred level of TPA power. In almost all of the interviews, respondents expressed that
TPA are simply insignificant in today’s Tanzanian politics. While a minimum role as cul-
tural custodians may be desirable, experts mostly rejected the idea of increasing the pow-
ers of TPA.
Finally, the Maasai were described as exceptional in Tanzania. Because of their
pastoralist lifestyle, the formal abolition of TPA powers does not seem to have had any
strong effects on the political role of Maasai traditional leaders. Maasai leaders seemed
content with their cooperation with local state institutions. In the light of their small pop-
!
16
ulation share (about 1 percent), I decide to treat the Maasai as an outlier when measuring
the overall political power of TPA in Tanzania – albeit an important one e.g. with regard
to conflicts between pastoralists and farmers in Northern Tanzania.
From scaled sources to triangulated measures
Table 3 about here.
Table 3 and Figure A.1 (see Appendix) summarize the scalings described above. Figure
A.1 shows the measurement for all three data sources by country on the scales termed S. I
now proceed to discuss how to triangulate the gathered information with the aggregation
strategies suggested by Leuffen et al. (2013). In Figure A.1, triangulated values are pro-
jected on the M-scale (for “measurement”, as in Leuffen et al., 2013, p. 42).
On aggregation strategies and convergence in triangulation
Strategy 1: Random Selection (RS)
If a source from S were chosen at random to project a measure on M, for any scale, it is
highly likely that a true value would not be approximated. Granted, for the A-panels on
the status quo, the variance of the scaling of the sources is relatively small, and hence
random selection could produce a good estimation on M. However, the spread of values
when it comes to the preferred level of political influence is greater. Thus, RS potentially
produces biased estimates. Clearly, the picture emerging from a RS aggregation mecha-
nism would not adequately capture the dynamics of the political power of TPA in Uganda
and Tanzania.
!
17
Strategy 2 and 3: Arithmetic Mean (AM) and Weighted Average (WA)
Because the S-scale is ordinal, an arithmetic mean cannot be calculated. However, for
conceptual purposes we can imagine taking the sum of the scalings from the different
sources and dividing them by the number of sources used for each scale. The AM strate-
gy presumes either that the researcher is convinced that all sources’ measurements are of
equal quality and weight, or that a confident measure of source quality cannot be generat-
ed. If the quality of the sources varies, the WA strategy can be used by attaching different
quality weights to each source.
Assessing source quality
Yet, for these kinds of data, the “quality” of the measurement of each source can be as-
sessed in various ways. Some reasonable differentiations in terms of data quality can be
put forward. For instance, on the aggregate level, the Afrobarometer data has the ad-
vantage of being based on a large-N sample. This should minimize random error. Is the
quality of the Afrobarometer data the highest because it is based on the largest number of
respondents? When disaggregating the data (Table A.6, see Appendix) we see that, for
example, for Tanzania the Afrobarometer samples only five Maasai. However, while
making up only one percent of the population, the in-depth interviews have shown that
the Maasai are of central importance as an outlier when studying the political influence of
TPA in Tanzania.
Are then qualitative interviews most valid because they provide detailed infor-
mation? One the plus side, the expert assessment is presumably less biased than the state
and traditional actors’ responses, given that the latter two groups have a greater stake in
!
18
what the true value is. For example, traditional actors may understate the status quo be-
cause their preference is for TPA to have more influence. The reverse is true of state ac-
tors. Further, the experts should be more reliable than the constitutional-legal text be-
cause they take account of informal dynamics. Still, one problem here is the representa-
tiveness of the interview sample, which focuses on rather small numbers of experts and
state and traditional actors.
Should we therefore weigh the constitutional-legal texts as most reliable because
they are formalized, and most likely the result of a discursive process between political
authorities? This is problematic because both the Afrobarometer and the interview data
show that there is more to the power of TPA on the ground than the constitutional-legal
texts suggest. These considerations suggest that assessments of source quality may also
depend on whether the empirical interest is more in qualitative depth or representative
width. When studying the overall contemporary political power of TPA in Uganda and
Tanzania on a national level, the Afrobarometer would be the better source given its rep-
resentativeness. Yet, if one is interested in the causal processes on the local level, one
might want to rely more on the qualitative interviews.
Source measurement
Assuming that we do not have information on source quality, or that all sources are of
equal weight, measurement with the AM strategy by triangulating all sources on S would
yield AM-measures in the interval shown on the M-scales in Figure A.1. Comparing AM-
values across countries, this points to two conclusions. First, in Uganda, the status quo
and the preferred level of political power of TPA are higher than in Tanzania. Second,
!
19
there is a discrepancy in Uganda between the status quo (“some-”) and a higher level of
preferred political power of TPA (“some+”), whereas in Tanzania the status quo and pre-
ferred level are alike (“small-”). These AM-ranges provide a glimpse into the compara-
tive dynamics of the two countries but they also obscure intrinsic political dynamics.
Comparing source quality
If the quality of the sources varies, the WA-strategy can be used by attaching different
quality weights to each source. Here, it is important to note that weighting requires a de-
cision not just about the relative quality of each source but also about the magnitude of
that difference. For reasons of simplicity, Leuffen et al. (2013) in their simulations as-
sume that one source is twice as revealing as each of the other two. For their three
sources and a dichotomous variable of interest, this produces results that show the prima-
cy of WA over other strategies.
In the case of the political power of TPA, magnitude assignment seems somewhat
arbitrary – are the experts twice or three times as accurate as the Afrobarometer data, or
vice versa? What if there are more than three sources to be triangulated? A reasonable
solution for this problem might be to compare the generated AM ranges on the M-scale
with the sources that are assumed to be of higher weight than others, and make a qualita-
tive statement about such weights. For example, if one assumes that expert interviews
and Afrobarometer data should be of equal importance but given greater weighting than
other sources, the preferred level of political power of TPA in Uganda would be slightly
higher than the AM-value (WA > “some-”). The status quo in Tanzania would be as-
sessed as less than the AM-value (WA < “small-”). For the status quo in Uganda, the
!
20
AM-value and the WA would be virtually the same, because the AM range already corre-
sponds to the experts’ and the Afrobarometer values (WA ≈ “some-”). Finally, the pre-
ferred level for Tanzania is trickier to interpret, because Afrobarometer and expert data
“pull” in different directions. Yet, because the expert value is further away from the AM-
range, it pulls the WA value to a slightly lower scaling.
If one can confidently assess the quality of the data and weight the sources ac-
cordingly, the WA provides a more pronounced and possibly more valid aggregated
measure on M. For the two cases, a weighted interpretation of the sources via the WA
should allow for more valid qualitative statements about the dynamics of interest. For ex-
ample, in comparison to the AM, with the WA we should be more confident in stating
that in Uganda there is a disparity between the status quo and preferences – which might
lead to political tensions. Moreover, we could be more assertive in our interpretation that
TPA are a less salient political issue in Tanzania than in Uganda, because the status quo
matches the preferred level of political power of TPA.
Strategy 4: Majority Strategy (MS)
If an MS is used to triangulate then we again assume that all sources are of equal weight.
I use three sources with four to five different values depending on the scale. While a
modal value might exist, it is theoretically possible that there is no majority. There could
even be a multimodal sample of sources.
Source measurement
For instance, regarding Panel A.1 for Uganda, we find that two sources were scaled as
!
21
“some” and two were scaled as “some-”. Taking the average value of the modes would
only make sense where they are very proximate in value – otherwise, where modes are
fairly distant, this would obstruct valid inference. Alternatively, for this particular scale,
we could simply transform values to the original four-step scaling, which would treat the
majority opinion as “some”. Figure A.1 shows the MS-values generated in this way.
It becomes apparent that considering the MS changes the interpretation of the po-
litical power of TPA. A comparison of the MS-values for Uganda indicates that there is
no discrepancy between status quo and preferred level of TPA power. In Tanzania, in
turn, there is discrepancy: the preferred level of power is lower than the status quo.
A triangulation problem here might be that “a requirement [for MS] is that the
sources are independent of each other” (Leuffen et al., 2013, 43) so that sources’ agree-
ment is not biased. However, for the sources at hand the interviews may not be independ-
ent of the constitutional and legal texts. The respondents refer back to these texts, and the
documents were to some extent drafted by the interviewed experts and state authorities.
Thus, independence among these sources can only be assumed for the Afrobarometer da-
ta.
Overall, MS appears to be a flawed strategy. It ignores the quality of sources, as
well as the information that a triangulation of all sources provides. This is especially
problematic for “smoking gun” evidence (Mahoney and Goertz, 2006), if one important
source of information can potentially alter the inference. Using MS could force the re-
searcher to ignore such important information.
!
22
Strategy 5: Winner Takes It All (WTIA)
WTIA involves weighting the sources to identify the one of highest quality. This strategy
somewhat resembles finding the aforementioned “smoking gun” – in terms of the content
and the quality of information.
Leuffen et al. (2013) describe WTIA as focusing on the most reliable and con-
sistent source. The Afrobarometer provides a good starting point to identify reliable val-
ues. To generate a measure via triangulation, the Afrobarometer values (ab) would be
projected on the M-scale. Yet, the Afrobarometer also ignores important qualitative in-
formation, as indicated by the small sample of Maasai in Tanzania. Then again, some of
the qualitative interviews with actors and experts offer rare and reliable insights. For ex-
ample, one interviewee held a central position in the Ugandan federal government for
over a decade, and in that role participated in the negotiation which led to the “The Insti-
tution of Traditional or Cultural Leaders Bill” of 2010. He had previously been a minister
for the Buganda Kingdom. For Tanzania, several of the experts had detailed knowledge
of the Maasai and the Sukuma cultures. Hence, one particular value given by one of the
respondents would be projected on the M-scale.
It becomes apparent that WTIA is wasteful. The strategy ignores the information
of sources ranked as less consistent. A sole concentration on the “winner” is less useful if
the analytical goal is qualitative description. While keeping this weakness in mind, ac-
cording to Leuffen et al. (2013), WTIA performs better than other strategies in case there
is strong systematic bias in the sample of sources. As argued before, there is evidence
that the sources used for studying TPA power are not independent. In addition, the snow-
!
23
ball sampling used for the qualitative interviews is prone to create interdependent inter-
views.
While sole consideration of the “winner” would be detrimental for qualitative in-
ference, being able to discriminate consistent sources will also shed light on inconsistent
information. One way to identify reliable “winners” could be to introduce “quality of in-
formation” items when collecting information from sources. For instance, interviewees
could be asked factual knowledge questions to assess their expertise in the aggregation
process. Other factors could be the interviewee’s hierarchical, or the years spent working
on the topic of interest. In the aggregation and triangulation of information, particular
importance should be given to the accounts of more reliable sources.
Source Timing
Finally, the timing of each source may not be an issue if triangulated sources are all col-
lected at the same time. However, all sources used here offer an assessment of the power
of TPA in different time periods. The constitutions and legal acts were adopted at differ-
ent times beginning in the 1960s, the Afrobarometer data was collected in 2008/2009,
and the qualitative interviews gathered in 2012/2013. For a weighted aggregation of in-
formation, should one trust the source most that has existed for the longest time, therefore
having the strongest effect on the power of TPA, i.e. constitutional-legal texts? Should
the greatest weight be given to more recent sources, i.e. the survey and the qualitative in-
terviews? This issue is even further complicated by some of the legal texts being adopted
after the Afrobarometer survey was conducted. Ultimately, if the main research interest
were in the contemporary status quo power of TPA, then a greater reliance on the recent
!
24
sources would be more adequate. If the interest is in a process tracing of the power of
TPA, then sources should be weighted depending on the stage of the observed process. In
any case, weighting sources based on timing should go hand in hand with considerations
of source quality.
On variation and divergence in triangulation
As much as it is desirable to aggregate details from the various sources, the actual varia-
tion of source information can be as informative for accurate inference (cf. Blaikie, 1991;
Stoker, 2011). Divergence of source information is interesting in two ways. First, it may
trigger a methodological re-assessment of the accuracy of source measurement and scal-
ing. This is due to variation of source quality. Second, if divergence persists then this is
rather due to variation of content and needs to be incorporated when conducting in-depth,
thick qualitative analysis. These dynamics are not discussed by Leuffen et al. (2013) but
seem equally important when discussing triangulation.
Strategies RS and MS ignore divergence of source information. WTIA considers
variation of source quality, but then selects only one reliable source, ignoring variation of
content. Both the arithmetic mean (AM) and the weighted average (WA) process sources
that vary in terms of content, with WA additionally taking into account the sources’ qual-
ity. Following this reasoning, an additional reason to prefer WA to other aggregation
strategies (as suggested by Leuffen et al., 2013) is that WA incorporates both variation of
source quality and content. However, both AM and WA reduce the divergence to one
value. Thus, considering either of them without also including the range of values in the
analysis may reduce validity of inference. Here again, the incorporation of source timing
!
25
is crucial, as divergence of content may of course be due to change over time in the
measurement of a variable of interest. Ignoring these differences in the timing of meas-
urements may impair the accuracy of triangulation.
Triangulation strategies and source divergence
Figure A.1 shows the variance in values of the status quo and the preferred level of polit-
ical power as a dotted line on the M-scale. If anything, accounting for the variance of
source scalings allows for an exclusion of irrelevant values. This should increase the va-
lidity of inference of the aggregated values. For the analysis of the political power of
TPA, we can generate comparative inference within and across countries: In Uganda, the
divergence of the preferred level of political power of TPA is greater than the divergence
in the status quo of political power. This may point to conflictual potential of the issue in
this country, with opposing interests held by various parties. In Tanzania, the opposite is
the case: the divergence of the status quo is larger than the divergence of sources on the
preferred influence scale, which suggests that TPA power is a less controversial issue. If
we take the Afrobarometer data as indicative of the population’s majority opinion, we can
further extrapolate that, while the Ugandan state authorities and the population have a
similar assessment of the status quo of political powers of TPA, their preferred level of
such power is highly divergent, with the population close to the traditional leaders’ pref-
erence. In Tanzania, the Maasai can be included in the analysis, with the weighted aver-
age measuring a relatively low status quo power of TPA in general. Interestingly, while
the experts’ assessment for Tanzania is mostly on the lower ends of the scaling range, the
experts for Uganda are scaled on higher levels for both scales. This suggests that experts
see potential for the role of TPA in Uganda, but less so for TPA in Tanzania.
!
26
These examples of the use of divergent source information emphasize the poten-
tial complementarity of aggregation and variation for triangulation. In any case, the use
of aggregated or disaggregated sources should not be an “either/or”. Together they gener-
ate valid qualitative analysis pointing to central tendencies as well as contradictory dy-
namics for the subject of interest.
In sum, I offer an illustrative application of Leuffen et al. (2013) using research
into the political power of traditional political authorities (TPA) in Uganda and Tanzania.
Leuffen et al. (2013) find that aggregation strategies using information that is weighted
by the quality of the source provide more accurate results compared to other triangulation
strategies – yet, if sources are systematically biased, researchers should focus on those
sources they deem most reliable and consistent. Following the qualitative application in
this paper, a similar argument can be made (Table 4 summarizes the findings for all tri-
angulation strategies). The accuracy of a measurement of interests can gain from triangu-
lating various sources. A strategy that mirrors a weighted average to aggregate data
sources is most fruitful for generating qualitative evidence. Other strategies seem less
amenable for qualitative research because they leave out information that may be central
for valid inference.
Table 4 about here
I have also discussed a second aspect of triangulation not explicated by Leuffen et
al. (2013): the role of variation and divergence of data source information for the validity
of inference. Without including an analysis of the range of source information, triangula-
tion appears incomplete. I distinguish two types of variation in sources that researchers
!
27
can incorporate: variation of source quality and variation of content. Both types need to
be taken into account to complement aggregation strategies.
Conclusion
Together with the aggregation strategies discussed by Leuffen et al. (2013), this paper
provides systematic insights for how to triangulate. Applying both the aggregation strate-
gies to approximate a true value of interest and types of source variation for the purpose
of triangulation has allowed for more complete qualitative statements. Here, this com-
bined strategy enabled a better empirical understanding of TPA in Uganda and Tanzania.
As such, triangulation can indeed foster validity of inference, especially in qualitative re-
search environments where information is scarce. Still, further structured methodological
guidance is required to make triangulation more systematic.
!
28
5. References
Afrobarometer (2009). Afrobarometer Round 4. http://www.afrobarometer.org/.
Baldwin, K. (2013). Why vote with the chief? Political connections and public goods
provision in Zambia. American Journal of Political Science, 57(4), 794–809.
Archibald, M. M. (2015). Investigator triangulation: A collaborative strategy with poten-
tial for mixed methods research. Journal of Mixed Methods Research, Advance
online publication
Blaikie, N. W. (1991). A critique of the use of triangulation in social research. Quality &
Quantity, 25, 115–136.
Denzin, N. K. (1978). The research act: A theoretical introduction to sociological meth-
ods. New York, NY: McGraw-Hill.
Denzin, N. K. (2010). Moments, mixed methods, and paradigm dialogs. Qualitative In-
quiry, 16(6), 419–427.
Denzin N.K. (2012). Triangulation 2.0. Journal of Mixed Methods Research, 6(2), 80-88.
Erzberger, C. & Prein, G. (1997). Triangulation: Validity and empirically-based hypothe-
sis construction. Quality & Quantity, 31, 141–154.
Fielding, N. G. (2012). Triangulation and mixed methods designs: Data integration with
new research technologies. Journal of Mixed Methods Research, 6(2), 124–136.
Hesse-Biber, S. (2010). Qualitative approaches to mixed methods practice. Qualitative
Inquiry, 16(6), 455–468.
Holzinger, Katharina, Florian G. Kern, & Daniela Kromrey. 2016. “The Dualism of Con-
temporary Traditional Governance and the State: Institutional Setups and Political
Consequences.” Political Research Quarterly (forthcoming).
!
29
Holzinger, Katharina, Florian G. Kern, & Daniela Kromrey. 2015. “Traditional Govern-
ance and Conflict in Sub-Saharan Africa: A Micro-Level Cross-Country Compari-
son.” Manuscript presented at the European Conference on African Studies (ECAS),
Paris (July 2015).
Howe, K. R. (2012). Mixed Methods, triangulation, and causal explanation. Journal of
Mixed Methods Research, 6(2), 89-96.
Huang, R. (2012). RQDA: R-based qualitative data analysis. R Package Version 0.2-3.
http://rqda.r-forge.r-project.org/.
Jick, T. D. (1979). Mixing qualitative and quantitative methods: Triangulation in action.
Administrative Science Quarterly, 24, 602–611.
Johnson, R. Burke & Anthony J. Onwuegbuzie (2004). Mixed methods research. A re-
search paradigm whose time has come. Educational Researcher, 33(7),14-26.
Johnson, R. Burke, Anthony J. Onwuegbuzie, & Lisa A. Turner (2007). Toward a defini-
tion of mixed methods research. Journal of Mixed Methods Research, 1(2),112-133.
Johnstone, L. (2007). Weighing up triangulating and contradictory evidence in mixed
methods organizational research. International Journal of Multiple Research Ap-
proaches, 1(1), 27-38.
King, G., Keohane, R. O., & Verba, S. (1995). The importance of research design in po-
litical science. American Political Science Review, 89(2), 475–481.
Leuffen, D., Shikano, S., & Walter, S. (2013). Measurement and data aggregation in
small-n social scientific research. European Political Science, 12(1), 40–51.
Logan, C. (2013). The roots of resilience: Exploring popular support for African tradi-
tional authorities. African Affairs, 112(448), 353-376.
!
30
Mahoney, J. & Goertz, G. (2006). A tale of two cultures: Contrasting quantitative and
qualitative research. Political Analysis, 14(3), 227–249.
Marks, G. (2007). Editorial. Introduction: Triangulation and the square-root law. Elec-
toral Studies, 26, 1–10.
Moran-Ellis, J., Alexander, V. D., Cronin, A., Dickinson, M., Fielding, J., Sleney, J., &
Thomas, H. (2006). Triangulation and integration: Processes, claims and implications.
Qualitative Research, 6(1), 45–59.
Morse, Janice M. (1991). Approaches to qualitative-quantitative methodological triangu-
lation. Nursing Research, 40(2),120-123.
Morse, Janice M. (2010). Procedures and practice of mixed method design. Maintaining
control, rigor, and complexity. In Tashakkori, A. & Teddlie, C. (Eds.) SAGE hand-
book of mixed methods in social & behavioral research (339-352). Thousand Oaks,
CA: Sage Publications.
Stoker, L. (2011). Triangulation. In Badie, B., Berg-Schlosser, D., & Morlino, L (Eds.),
International encyclopedia of political science (2669–2671), Thousand Oaks, CA:
SAGE Publications.
Tarrow, S. (1995). Bridging the quantitative-qualitative divide in political science. Amer-
ican Political Science Review, 89(2), 471–474.
Turner, P. & Turner, S. (2009). Triangulation in practice. Virtual Reality, 13(3), 171–181.
!
31
Endnotes
1 By increased validity of inference I mean “the increased confidence in the implied
measurement outcomes of the research where there are convergent findings” (Moran-
Ellis et al. 2006, 47).
2 The Afrobarometer is a large public attitude survey on various socio-political issues in
Sub-Saharan Africa conducted by a pan-African research network every two to three
years since 1999.
3 The remaining two types – investigator triangulation and theory triangulation – are left
aside here, because (a) I use a common theoretical framework for all sources and hence
do not triangulate between theories; (b) Leuffen et al. (2013, p. 41) explicitly focus on the
triangulation of data and different methodological perspectives; and (c) investigator tri-
angulation would arguably happen either when data is collected, or when data is evaluat-
ed, but not at the aggregation stage of data investigated by Leuffen et al. (2013).
!
32
Tables
Q1: “How much influence do traditional leaders currently have
in governing your local community?”
Response
UGANDA
TANZANIA
None
0.18
0.46
A small amount
0.28
0.15
Some
0.24
0.13
A great deal
0.16
0.07
Don’t know
0.13
0.19
N = 2,431
N = 1,208
Table 1: Status quo of governing influence (Afrobarometer 2009), relative share
UGANDA
TANZANIA
N
actS
10
8
18
actT
10
9
19
exp
15
14
29
N
35
31
66
Table 2: Number and categories of in-depth interviews
!
33
UGANDA
TANZANIA
Source
status quo
preferred
status quo
preferred
Constitutional-
legal texts
“some”
-
“none”
-
Afrobarometer
“some-”
“large-”
“small-”
“small+”
Interviews
actS
“some”
“small”
“small-”
“none”
actT
“small+”
“some+”
“small”
“small-”
exp
“some+”
“some”
“none+”
“none+”
Table 3: Summary of scalings by source
!
34
Table 4: Strengths and limitations of triangulation strategies, findings
Strategy type
Strengths
Limitations
Note on divergence
Findings on TPA
Random Selection
Reduces bias for a
large number of
sources (only)
With small number
of sources, approx-
imation of true val-
ue is unlikely
Spread of values
across sources is ig-
nored, or random as
well
Not applicable
Arithmetic Mean
Informative if dif-
ferentials in source
quality are not as-
sessable, or sources
are of the same
quality
Assumption of
equal source quality
may be unrealistic,
leads loss of infor-
mation
Only spread of
source content is tak-
en into account
Status quo and pre-
ferred power of TPA
higher in UGA than
TAN; discrepancy in
UGA between lower
status quo and higher
preferred power
Weighted Average
Informative if
source quality can
be assessed, and if
source quality var-
ies; allows stronger
qualitative assess-
ment
Assessing source
quality is complex,
and requires further
measurement of rel-
ative source quality
weights; difficult for
varying types of
sources
Spread of source
content and source
quality is taken into
account
TPA less salient is-
sue in TAN than in
UGA; in TAN, status
quo similar to pre-
ferred level of power
if experts and
Afrobarometer weigh
more
Majority Strategy
Facilitates triangula-
tion of true value
and comparison
across sources
Sources might be
multi-modal or
without majority;
ignores source qual-
ity
Spread of values
across sources is ig-
nored
No discrepancy be-
tween status quo and
preferred level of
TPI power in UGA;
in TAN, preferred
level is lower than
status quo
Winner Takes It All
Allows detection of
“smoking gun” evi-
dence; informative
if sources are sys-
tematically biased
Wasteful; ignores
information in case
the goal of thick de-
scription
Spread in source
quality is incorpo-
rated, but focuses
only on the most
consistent sources
Afrobarometer find-
ings most consistent,
but ignore crucial
information, e.g. role
of Maasai in TAN
!
35
Figures
Figure 1: Scales of political power of TPA, (A) status quo and (B) preferred
Figure 2: Scaling example
