Content uploaded by Catherine Macphail
Author content
All content in this area was uploaded by Catherine Macphail
Content may be subject to copyright.
doi:10.1136/sti.2006.023689
2007;83;i70-i74; originally published online 27 Feb 2007; Sex. Transm. Inf.
Audrey E Pettifor, Catherine MacPhail, Stefano Bertozzi and Helen V Rees
programme: the case of loveLife, South Africa
Challenge of evaluating a national HIV prevention
http://sti.bmj.com/cgi/content/full/83/suppl_1/i70
Updated information and services can be found at:
These include:
References
http://sti.bmj.com/cgi/content/full/83/suppl_1/i70#BIBL
This article cites 13 articles, 6 of which can be accessed free at:
Rapid responses
http://sti.bmj.com/cgi/eletter-submit/83/suppl_1/i70
You can respond to this article at:
service
Email alerting
the top right corner of the article
Receive free email alerts when new articles cite this article - sign up in the box at
Notes
http://journals.bmj.com/cgi/reprintform
To order reprints of this article go to:
http://journals.bmj.com/subscriptions/
go to: Sexually Transmitted InfectionsTo subscribe to
on 29 August 2008 sti.bmj.comDownloaded from
Challenge of evaluating a national HIV prevention
programme: the case of loveLife, South Africa
Audrey E Pettifor, Catherine MacPhail, Stefano Bertozzi, Helen V Rees
...................................................................................................................................
See end of article for
authors’ affiliations
........................
Correspondence to:
Audrey Pettifor, Department
of Epidemiology, CB
#7435, McGavran-
Greenberg Building,
University of North Carolina
at Chapel Hill, Chapel Hill,
NC 27599-7435; apettif@
email.unc.edu
Accepted 13 February 2007
Published Online First
27 February 2007
........................
Sex Transm Infect 2007;83(Suppl I):i70–i74. doi: 10.1136/sti.2006.023689
Although 50% of all new global HIV infections occur among young people, our knowledge to date of the
impact of adolescent HIV prevention interventions in developing country settings is limited. During 1999, a
national HIV prevention programme for youth, called loveLife, was launched in South Africa. This paper
describes the challenges faced in trying to evaluate such a national programme and the types of evidence that
could be used to better understand the effect of programmes of national scale. A range of methods were
planned to evaluate the programme, including national household surveys and programme monitoring data.
Given the urgent need to scale-up programmes in an effort to reduce new HIV infections, a range of evidence
should be assessed to measure the effect of large-scale, complex behavioural interventions as an alternative
to randomised controlled trials.
A
s in much of Africa, young people in South Africa are
particularly at risk for HIV infection. In 2003, a nationally
representative household survey of close to 12 000 people
aged 15–24 years found that 15.5% of South African young
women were infected with HIV compared with 4.8% of young
men.
1
Despite the fact that 50% of all new global HIV infections
occur among young people, to date our knowledge of the
impact of adolescent HIV prevention interventions in develop-
ing country settings is limited. The few rigorous studies from
developed and developing countries have shown that, while it is
possible to improve knowledge, attitudes and to some extent
sexual behaviours, it has been more difficult to show an effect
on pregnancy, sexually transmitted infection (STI) or HIV
rates.
2–9
In order to have the greatest impact on stemming the
epidemic and to ensure efficient use of resources, it is critical to
provide evidence on the effect of adolescent focused prevention
programmes.
There is a need to bring prevention interventions to scale, and
with speed, to have a substantial impact on the HIV epidemic
among youth.
10 11
Nevertheless, national programmes do not
lend themselves to being evaluated using methods such as the
randomised controlled trial (RCT). Although the RCT is viewed
as the gold standard for proving the efficacy of interventions,
there has been debate in the literature about the appropriate-
ness of RCTs in evaluating public health programmes,
particularly large programmes that have multiple components
and complex causal pathways. Over the years several authors
have argued for alternative approaches and methodologies to
determine the effect of public health interventions in achieving
desired public health effects.
12–14
This paper aims to describe the
challenges faced when trying to evaluate a national pro-
gramme, specifically loveLife, and the types of evidence that
could be used in the absence of an RCT to better understand the
effect of the programme.
THE LoveLife PROGRAMME
In the context of high HIV prevalence levels and associated risk
behaviours among adolescents, a national HIV prevention
intervention for youth, called loveLife, was launched in South
Africa during 1999. loveLife is a national initiative combining
high-powered multimedia with comprehensive youth-friendly
reproductive health services in public clinics and countrywide
community outreach and support programmes. The interven-
tion aims to reduce HIV, other STIs and unwanted pregnancies
among South African youth. Specifically, its goal is to trigger
and sustain changes in sexual behaviour and related social
norms to halve the rate of new HIV infections among young
people. The target age range for loveLife is 10–17 years,
although many programmes reach individuals in older age
groups, and older youth are not excluded from participating in
programmes.
loveLife’s activities are broad ranging in scope, content and
level of engagement, and they operate at multiple levels: the
individual, peer group, family and community, and nationally
at a societal/cultural level. Media programmes, including
billboards, television, radio and printed materials, promote
HIV risk reduction and the concept of a positive lifestyle to
South African youth by providing limited factual information,
challenging social norms and stimulating public debate around
issues relevant to HIV risk, such as condom use, multiple
partners and gender norms.
To provide youth, parents, organisations and communities
with more substantive and face-to-face experiences that aim to
reduce HIV risk, loveLife also offers comprehensive, interactive
educational programmes. These programmes extend into
communities and are implemented by young people themselves
(loveLife’s community-based peer educators are called
groundBREAKERS and Mpintshis). loveLife’s outreach pro-
grammes provide service delivery, institutional support and
capacity building, and deliver what are called ‘‘loveLifestyle’’
experiences for young people. The programmes aim to inspire
and motivate young people to take control of their lives, set
goals, make healthy choices, and navigate and reduce their HIV
risk. An important element of the community outreach and
clinical service programmes are loveLife Y-Centers (youth
centres), franchises (youth-serving community organisations
affiliated with loveLife) and National Adolescent-Friendly
Clinic Initiative (NAFCI) clinics. These organisations act as
hubs for community outreach from which loveLife’s educa-
tional programmes are implemented directly in schools and
Abbreviations: NAFCI, National Adolescent-Friendly Clinic Initiative;
RCT, randomised controlled trial; STI, sexually transmitted infection
i70
www.stijournal.com
on 29 August 2008 sti.bmj.comDownloaded from
other community venues through engaging young people in a
range of recreational activities.
The theoretical framework for loveLife’s behaviour change
process draws on a number of behavioural theories including
diffusion of innovations, ecological theory and the theory of
reasoned action.
15–17
Diffusion is a process whereby members of
a social system communicate about an innovation over time.
15
This model focuses on social networks, opinion leaders and
change agents. loveLife’s programmes operate by using change
agents (the groundBREAKERS and Mpintshis) and opinion
leaders (eg well-known South Africans such as Nelson
Mandela) to communicate about new ideas and previously
unacceptable behaviours, such as talking to children about sex.
Further, by working in schools and with national media
organisations loveLife capitalises on existing social networks
to communicate the programme’s messages about behaviour
change and to challenge social norms that may act as barriers to
HIV prevention.
Ecological theory acknowledges that behaviour change is not
only an individual-level construct, but that influences also
operate at the dyad/small group, organisational, community
and cultural/societal levels.
17 18
A central tenet of ecological
theory is to tailor HIV intervention programmes to target the
level at which risk is manifest, rather than focusing exclusively
on the individual. loveLife’s programmes aim to address HIV
risk by working to change norms and attitudes around issues
that affect HIV risk, such as condom use, HIV testing and
disclosure, and gender power relations, at multiple levels.
Lastly, loveLife operates within the theory of reasoned action,
in which individuals take into account the implications of their
behaviour within a particular context before they decide to
change their behaviour.
16
The theory places importance on the
role of normative beliefs and locates young people’s behaviour
in the framework of both their attitudes and subjective norms
or social influences. loveLife’s programmes aim to change
societal norms and attitudes that place young people at risk of
HIV, particularly through promoting dialogue and debate about
youth sexuality issues.
Specifically, loveLife’s programmes use the following
approaches:
N
social networks and young people as change agents to
implement and diffuse programmes;
N
addressing the various levels at which HIV risk is manifest
(individual, small group, community, organisational and
societal);
N
working to change norms and attitudes that influence HIV
risk behaviour;
N
providing comprehensive, factual and personalised sexual
and reproductive health (SRH) and HIV/AIDS information;
N
providing positive lifestyle experiences for youth, which give
them skills and motivation to reduce their HIV risk
(motivation, art, music, debate, creative problem solving,
health and fitness programmes, HIV/STI/pregnancy risk
reduction skills);
N
providing comprehensive, quality SRH services for adoles-
cents.
CHALLENGES IN EVALUATING LoveLife
loveLife is an excellent example of a programme that cannot be
readily evaluated using an RCT. Nevertheless, given its national
scope, comprehensive nature and the investment made in it,
understanding its effect is critical. Given that loveLife is a
national intervention, all South African adolescents are
theoretically exposed. There is thus, no obvious control group,
and because the programme is comprehensive and involves a
number of national programmatic elements, randomisation of the
campaign is not possible. Although staged programmatic role-out
with intervention and control areas may have been a possibility
for some of the regional outreach and service components, the
programmes were specifically targeted at previously disadvan-
taged communities identified by partners in national and local
government. Sites chosen for programmes were selected based
on need and a desire for maximum impact, rather than the
requirements of an evaluation. Further, the intervention
implementers went to scale rapidly—a priority given the speed
at which the epidemic was growing in South Africa. The
programme, therefore, rolled out before a baseline evaluation
was conducted.
A further challenge in evaluating a complex programme such
as loveLife was determining how to define programme
exposure. In 2003 there were over 16 different programmes
designed to work synergistically. In the 2003 national survey,
65% of youth reported awareness of four or more programmes.
1
Thus, looking at one programme in isolation may not capture a
true representation of how the programmes work together. In
addition, programme exposure data are based on self-report
and thus subject to recall and social desirability bias. All these
issues contribute to the challenge of evaluating a complex
intervention of this scope with a predetermined set of factors
that provide a less than ideal evaluation situation, as is so often
the case in intervention implementation outside of the context
of study settings.
PROPOSED SOLUTIONS TO THE CHALLENGES
Given the above challenges and loveLife’s national and
community scope, two different study designs were originally
planned for intervention evaluation. First, a series of nationally
representative household surveys collecting both behavioural
and biological markers of HIV prevention were planned and the
first implemented in 2003.
119
The aims of this survey were to
determine:
N
the prevalence of HIV infection, associated risk behaviours
and exposure to the loveLife programme among young
people age 15–24 years in South Africa
N
whether there is a 50% decrease in HIV prevalence nationally
among young people age 15–24 years over a five-year period
(to be able to measure this within age groups and by sex);
N
whether young people exposed to loveLife have a lower
prevalence of HIV and related risk behaviours compared
with those young people who have not been exposed.
The focus on 15–24-year-olds was guided by the low HIV
prevalence in youth under 15; thus detecting changes in HIV
over time would require a very large sample size. Further, given
that loveLife started in 1999, it would allow us to determine the
intervention’s reach in older populations and measure its effect
in youth who may have been part of the target age group before
the survey but no longer were. The survey used a three-stage
disproportionate, stratified design using the 2001 census
enumeration areas as the primary sampling unit to identify a
nationally representative household sample of young people,
15–24 years of age, living in the nine provinces of South Africa.
Households within sampled areas were enumerated and one
eligible young person in each household was randomly selected
to take part in the interview, irrespective of whether they were
sexually active or not. Study participants were asked to
complete a comprehensive interviewer-administered question-
naire on HIV risk, protective behaviours and exposure to
loveLife. They were also asked to provide an anonymous oral
fluid specimen to test for HIV infection. It was originally
planned that this survey would be repeated over time to
measure changes in the desired outcomes, but challenges in
Evaluating national HIV prevention programmes i71
www.stijournal.com
on 29 August 2008 sti.bmj.comDownloaded from
accessing funding for intervention-specific surveillance type
data collection means that the survey will probably not be
repeated as planned.
Second, a quasi-experimental, repeated, cross-sectional,
community-based study was designed.
20
This design was
chosen as it was hypothesised that youth living in communities
with more intensive loveLife activities, specifically communities
with Y-Centers or NAFCI—the ‘‘hubs’’ of loveLife outreach and
programmatic implementation—would be at lower risk for HIV,
STIs and related risk behaviours compared with youth living in
communities without these intensive activities. The strength of
this study included allowing for a control arm and the
measurement of other biological markers, specifically gonor-
rhoea and chlamydia, which was not logistically or financially
possible in the national survey.
The study has three arms: 11 communities with Y-Centers, 11
with NAFCI clinics and 11 control communities (which all had
standard public sectors clinics) selected from matching health
districts. Households falling within 2 km of a study centre or
control clinic were enumerated and one young person aged
between 15 and 24 years from each household was randomly
sampled (again irrespective of sexual activity). Like the national
survey, participants took completed an interviewer-adminis-
tered questionnaire and provided anonymous samples to test
for HIV, and gonorrhoea and chlamydia. The baseline surveys
were conducted from August 2002 to January 2003. As with the
national survey, insufficient funds are available for the follow-
up survey to thus far be conducted.
In the light of loveLife’s evaluation plan, Habicht et al provide
a useful framework for assessing different types of evidence to
develop a more comprehensive picture of intervention impact,
particularly in the case where an RCT has not been conducted.
13
This framework provides a continuum of evidence for an
intervention’s effect moving from adequacy to plausibility to
probability. Adequacy designs ask whether the expected
changes occurred. They can evaluate performance, such as
how well programme activities met expected objectives (these
are often called process indicators: how many centres opened,
how many condoms were distributed, how many individuals in
the population used the programme), and can also look at
impact by assessing whether health or behavioural indicators
have improved among programme recipients or the target
population as a whole.
13
Although this design cannot causally
link the observed health or behavioural outcomes to the specific
intervention itself, it can supply information on whether the
services were provided, if services were used and if the target
population was reached. If these objectives were not met (ie no
youth accessed the programme) then one would not expect the
programme to have an effect on the outcome of interest.
In plausibility designs the evaluations attempt to document
impact and rule out alternative explanations. This is often done
by including a comparison group and by dealing with
confounding variables. Plausibility assessments encompass a
continuum from weak to strong; with stronger findings the
other possible explanations for the observed associations can be
discarded. Probability designs may provide the strongest
evidence that any observed effect was due to the intervention
and aim to ensure that there is only a small probability that
observed differences between intervention and control sites
were due to confounding or chance. The RCT falls into this
category.
EVALUATION OF LoveLife: INDICATORS OF
ADEQUACY AND PLAUSIBILITY
Providing evidence for the adequacy of loveLife’s impact on
desired outcomes is possible (table 1). There are good
monitoring systems in place to determine whether services
are available, accessible, of good quality and are being used by
the target population, and whether the target population is
being reached. Data monitoring from the fourth quarter of 2004
indicated that there were over 235 NAFCI clinics, 16 Y-Centers,
532 schools implementing loveLife programmes, 908
groundBREAKERS and that 153 543 young people participated
in ‘‘loveLifestyle’’ programmes nationally.
21
In the 2003 survey,
85% of all youth reported awareness of loveLife, ranging from
65% in rural farming areas of South Africa to 93% in urban
formal areas, and over a third reported participation in
loveLife’s programmes.
19
Outreach activities aim to target rural
and disadvantaged youth who have been shown to be less likely
to receive prevention interventions. In addition, programmes
seem to be reaching their target audience as they are designed
to be youth friendly and accessible to youth by being open on
weekends and afternoons.
Determination of the plausibility of loveLife’s programmes
having had an effect on observed changes in sexual behaviour
or HIV infection becomes more complicated. The first challenge
is determining the best way to classify exposure to the
programme. When loveLife started, programme exposure was
crudely defined as awareness of the programme (Have you
heard of or seen loveLife?); yet by 2003, 85% of youth were
aware of the programme and in the community survey baseline
these numbers were even higher.
120
Given that we hypothesise
that participation in loveLife’s programmes will probably have a
greater impact on HIV risk behaviour than awareness of the
programmes alone, we place more emphasis on examining
participation in programmes than awareness. To deal with the
multitude of possible programme exposures we have grouped
exposure based on type (eg outreach v media) and examined
programme effect in this manner rather than trying to look at
over 16 different programmes individually. Although this
definition does not get at the complexity inherent in each
programme, looking at programmes in isolation, when they
were designed to work synergistically, may not be accurate
either. Ideally, detailed information on the breadth (ie number
of programmes youth participate in) and depth (ie amount of
exposure time), the type of involvement and level of engage-
ment with the programmes should also be measured when
evaluating such programmes. Another important difficulty in
trying to isolate the effect of one programme operating at a
national level is that a multitude of other HIV prevention
programmes are also being implemented both at the commu-
nity and national level, thus any observed effect on behaviour
will most probably be the combined effect of all prevention
programmes that youth are exposed to. It is doubtful that the
different effects of a multitude of programmes will ever be fully
untangled.
One means to address sole reliance on self-reported exposure
data in the surveys involves using ecological measures of
exposure to loveLife programmes by analysing the impact of—
for example—living with a 2 km radius of a NAFCI clinic. The
community survey allows for the opportunity to examine the
effect of living in a community with a high intensity of loveLife
programmes as opposed to individual-level exposure to the
programme.
20
Methods to improve self-reported behaviors,
particularly those that are subject to social desirability bias,
are also an important element to consider for future surveys.
Once programme exposure is defined, the challenge of
examining associations between loveLife and selected out-
comes, and determining whether such associations are valid,
remains. Although we cannot draw causal conclusions from
these current studies given their cross-sectional nature, steps
can be taken to strengthen the plausibility that loveLife may
have had an impact on observed changes. These steps include
first, measuring potential confounders and controlling for them
i72 Pettifor, MacPhail, Bertozzi, et al
www.stijournal.com
on 29 August 2008 sti.bmj.comDownloaded from
using multivariable statistical methods. In multivariable
models using data from the 2003 national survey, both sexually
experienced males and females who reported participation in
loveLife had significantly less chances of being infected with
HIV (males: odds ratio (OR) 0.60 (95% CI 0.40 to 0.89);
females: 0.61 (0.43 to 0.85)) after controlling for socioeconomic
factors (education, rural/urban residence, having electricity in
the home, marital status, age); ever testing for HIV; personally
knowing someone who had died of AIDS; participation in
youth groups; and awareness of two other national prevention
campaigns (SoulCity and the Red Ribbon campaign).
19
Second,
assessing whether results are consistent when examining
associations between loveLife and different outcomes. In the
national survey, youth who participated in loveLife pro-
grammes, for example, were also more likely to report using a
condom at last sex (males: OR 2.2 (1.7 to 2.9); females: OR 2.1
(1.3 to 3.2)); were more likely to have talked to their parents
about HIV (males: OR 1.9 (1.6 to 2.1); females: OR 2.1 (1.5 to
2.9)); to report having changed their behaviour due to HIV
(males: OR 1.9 (1.4 to 2.7); females: OR 1.6 (1.3 to 2.0)); and to
report having a stronger sense of future optimism (males: OR
1.8 (1.4 to 2.4); females: OR 1.5 (1.3 to 1.9)).
22
Third we aim to
determine if dose–response associations are present in the
expected direction. Again, from the national survey we found
that youth who participated in two or more programmes
compared to no programme were less likely to be infected with
HIV (1 programme v 0: OR 0.68 (0.54 to 0.85); >2 v 0: OR 0.44
(0.33 to 0.58)) and more likely to use condoms consistently (1
programme v 0: OR 1.47 (1.18 to 0.85); >2 v 0: OR 2.28 (1.71 to
3.05)) compared with youth who participated in only one
programme compared with no programme participation.
Perhaps one of the biggest challenges with evaluating
national programmes stems from the cost of evaluation and
ensuring appropriate commitment to and financial planning for
rigorous evaluations. Evaluations such as the one described are
expensive, although considerably less expensive than an RCT.
Since the start of the loveLife evaluation, funding constraints
have resulted in many of the planned evaluation activities being
no longer feasible, in particular the two large surveys. Other
methods that could be used to strengthen the evidence of
loveLife’s effect, include the following.
N
Epidemic modelling to estimate reductions in HIV incidence
that would be expected with reductions in risk behaviour
and HIV prevalence associated with prevention exposure in
South African youth. This could involve developing a model
incorporating potential and observed relationships of pre-
vention exposure to levels of sex-specific and age-specific
risk behaviours and HIV prevalence; conducting epidemic
simulations to compare HIV infections expected in youth
nationally with and without prevention programmes, to
estimate infections averted; and exploring the potential
gains in HIV infections averted associated with expansion
and/or intensification of various prevention components.
N
Conducting economic costings of the implementation of
loveLife, overall and for programme components, by
geographic area and over the phases of programme
implementation for five years.
N
Spatial mapping to examine the variation in loveLife’s
spread geographically, and also to determine whether the
greatest change in HIV prevalence is observed in the areas
where there is the greatest exposure to loveLife.
Despite reductions in large-scale evaluation activities, loveLife
continues to collect programme monitoring and process
evaluation information to deal with the issues of programme
access, availability, coverage and utilisation. In addition,
smaller-scale surveys have been conducted to measure pro-
gramme exposure and knowledge of current programmes.
CONCLUSION
The RCT remains an important tool for providing evidence on
the effect of interventions. Nevertheless, in the context of an
urgent need to prevent new HIV infections, with the aim of
Table 1 Levels of information available for loveLife in demonstrating that the programme has achieved its desired aims
Tools to measure Potential indicators Possible outcomes
Provision
Are the services available? Monitoring Number of different loveLife programmes 235 NAFCI clinics
532 Schools participating
Are they accessible? Monitoring Hours of operation Open afternoons/weekends, provided in school and
CBOs
Adolescent friendly? All programmes adolescent friendly
Is the quality satisfactory? NAFCI—quality
assurance
assessments
Quality score 37% of NAFCI clinics attained 90% of standards for
adolescent quality care 98% attained 70% of
standards for adolescent quality care
On average clinics improved their score by 51%
post-NAFCI implementation
Utilisation
Are the services being
used?
Monitoring Number of youth services provided 153 543 youth participated in loveLife programmes
in Q4 2004
108 570 clinical services provided to youth ages
10–19 years in Q4 2004
Coverage
Is the target population
being reached?
Monitoring Number in target population accessing services 85% of youth reported being aware of the loveLife
programme
Evaluation Number of youth in household survey reporting
awareness of and exposure to loveLife programmes
34% report participation
Ecological measures of programme in area
Impact
Improvements in disease
patterns or health-related
behaviours?
National household
survey of HIV and
sexual behaviour
HIV prevalence Observed changes in HIV and behaviour based on
follow-up surveys
Community-based
survey
Sexual behaviours
Norms and attitudes with regard to HIV prevention
CBO, community-based organisation; NAFCI, National Adolescent-Friendly Clinic Initiative; Q4, fourth quarter of 2004.
Evaluating national HIV prevention programmes i73
www.stijournal.com
on 29 August 2008 sti.bmj.comDownloaded from
meeting the UN Millennium Development Goals for youth and
HIV, there is a need to role out interventions at scale and with
speed.
10
Organisations such as the WHO have advocated for a
‘‘Steady, Ready, Go!’’ approach for rolling out interventions on
a large scale based on a number of criteria.
10
Given this, there is
a need to expand and implement rigorous methods that will
provide information on the effect of large-scale, complex
behavioural interventions as alternatives to RCTs. Given the
need to identify successful prevention interventions, we must
look at multiple sources of data to understand the full picture in
relation to the effect of prevention programmes. While a
programme like loveLife may never be able to prove causation,
putting together pieces of the puzzle to understand what did
happen will provide valuable information, not only for loveLife,
but for other prevention programmes globally.
AUTHOR CONTRIBUTIONS
AEP developed and implemented the evaluation activities and wrote the
paper; CM analysed the data related to the evaluation and conceptua-
lised and edited the paper; SB helped to conceptualise the ideas in the
paper and edit the paper; and HVR was the principal investigator of the
evaluations and was involved in conceptualising and editing the paper.
Authors’ affiliations
.......................
Audrey E Pettifor, Department of Epidemiology, University of North
Carolina at Chapel Hill, USA
Catherine MacPhail, Helen V Rees, Reproductive Health and HIV Research
Unit, University of the Witwatersrand, Johannesburg, South Africa
Stefano Bertozzi, Division of Health Economics and Policy, National
Institute of Public Health, Cuernavaca, Mexico
The Kaiser Family Foundation (KFF) was the primary funder of the
evaluation work reported here. AEP, CM and HVR all received salary
support from KFF though loveLife for work on the evaluation. KFF is one of
the major funders of the loveLife programme.
Competing interests: None declared.
Adapted from a presentation given at the 16
th
International STD
Conference of the International Society for Sexually Transmitted Diseases
Research, 10–13 July 2005, Amsterdam.
REFERENCES
1 Pettifor A, Rees H, Steffenson A, et al. HIV and sexual behaviour among young
South African a national survey of 15–24 year olds. Johannesburg: Reproductive
Health Research Unit, 2004.
2 DiCenso A, Guyatt G, Willan A, et al. Interventions to reduce unintended
pregnancies among adolescents: systematic review of randomised controlled
trials. BMJ 2002;324:1426–30.
3 Wight D, Raab G, Henderson M, et al. The limits of teacher delivered sex
education: interim behavioural outcomes from a randomised trial. BMJ
2002;324:1430–3.
4 Grunseit A. Impact of HIV and sexual health education on the sexual behaviour of
young people: a review update. Geneva: UNAIDS, 1997.
5 Gallant M, Maticka-Tyndale E. School-based HIV prevention programmes for
African youth. Soc Sci Med 2004;58:1337–51.
6 Shuey D, Babishangire B, Omiat S, et al. Increased sexual abstinence among in-
school adolescents as a result of school health education in Soroti district,
Uganda. Health Educ Res 1999;14:411–9.
7 Kirby D. Reducing adolescent pregnancy: approaches that work. Contemp
Pediatr 1999;16:83–94.
8 Kirby D. Emerging answers: Research findings on programs to reduce teen
pregnancy. Washington DC: National Campaign to Prevent Teen Pregnancy,
2001.
9 Ross D, Changalucha J, Plummer M, et al. MEMA kwa Vijana, a randomised
controlled trial of an adolescent sexual and reproductive health intervention
programme in rural Mwanza, Tanzania: 3. Results: Knowledge, attitudes and
behaviour, Presented at the International Society for Sexually Transmitted
Diseases Research Congress, 27–30 July 2003, Ottawa, Canada.
10 World Health Organization. Steady … Ready … Go! The Talloires Consultation
to review the evidence for policies and programmes to achieve the global goals
on young people and HIV/AIDS. Geneva: World Health Organization,
Department of Child & Adolescent Health & Development, 2004.
11 Stover J, Walker N, Garnett G, et al. Can we reverse the HIV/AIDS pandemic
with an expanded response? Lancet 2002;360:73–7.
12 Victora CG, Habicht JP, Bryce J. Evidence-based public health: moving beyond
randomized trials. Am J Public Health 2004;94:400–5.
13 Habicht J, Victora C, Vaughan J. Evaluation designs for adequacy, plausibility,
and probability of public health programme performance and impact.
Int J Epidemiol 1999;28:10–8.
14 Des Jarlais DC, Lyles C, Crepaz N. Improving the reporting quality of
nonrandomized evaluations of behavioral and public health interventions: the
TREND statement. Am J Public Health 2004;94:361–6.
15 Rogers E. Diffusion of innovations, 5th edn., New York: Free Press 2003.
16 Fisher J, Fisher W. Theoretical approaches to individual-level change in HIV risk
behavior. In: Peterson J, DiClemente R, eds. Handbook of HIV prevention. New
York: Kluwer Academic/Plenum Publishers, 2000:3–55.
17 Waldo C, Coates T. Multiple levels of analysis and intervention in HIV
prevention science: exemplars and directions for new research. AIDS
2000;14:S18–S26.
18 Stokols D. Translating social ecological theory into guidelines for community
health promotion. Am J Health Promot 1996;10:282–98.
19 Pettifor A, Rees H, Kleinschmidt I, et al. Young people’s sexual health in South
Africa: HIV prevalence and sexual behaviours from a nationally representative
household survey. AIDS 2005;19:1525–34.
20 Pettifor A, Kleinschmidt I, Levine J, et al. A community based study to examine
the effect of a youth HIV prevention intervention on young people age 15–24
in South Africa: results of a baseline survey. Trop Med Int Health
2005;10:971–80.
21 loveLife. loveLife 2004: Report on progress and activities. Johannesburg:
loveLife, 2005.
22 Madikizela-Hlongwa L, Pettifor A, MacPhail C, et al. Awareness of and
participation in the loveLife youth HIV prevention campaign by youth in South
Africa: results from a nationally representative household survey. XV
International AIDS Conference, 11–16 July 2004, Bangkok.
Key messages
N
To have a substantial impact on the HIV epidemic among
youth, there is a need to bring prevention interventions to
scale, and with speed. Nevertheless, national pro-
grammes do not lend themselves to being evaluated
using methods such as the randomised controlled trial.
Alternative approaches and methodologies to determine
the effect of public health interventions in achieving
desired public health impacts are needed.
N
LoveLife is a large, national HIV prevention programme
for youth in South Africa. Given its scope, it does not lend
itself to evaluation using traditional methods such as a
randomised controlled trial. We highlight the challenges
of trying to evaluate loveLife, discuss the planned
evaluation strategy, and alternative evidence that could
be used to determine the impact of the programme.
N
Challenges of evaluating a national programme include: no
obvious control arms (all individuals are exposed),
programme implementers and key stakeholders implement-
ing a programme before a baseline evaluation is
conducted, no clear way to measure exposure to the
programme when multifaceted and complex casual path-
ways are involved, social desirability bias and multiple
other programmes simultaneously being implemented.
N
We support using an evaluation framework that exam-
ines multiple sources of evidence with regard to a
programme’s impact and provide examples of such data
from loveLife. Such frameworks may be necessary to
evaluate the effect of large scale prevention and
treatment programmes in many countries.
i74 Pettifor, MacPhail, Bertozzi, et al
www.stijournal.com
on 29 August 2008 sti.bmj.comDownloaded from
