Content uploaded by Yulia Kuzmina
Author content
All content in this area was uploaded by Yulia Kuzmina on May 15, 2019
Content may be subject to copyright.
Yulia A. Tyumeneva
1
, Yulia V. Kuzmina
2
THE EFFECT OF ONE EXTRA YEAR OF SCHOOLING ON PISA
RESULTS: A CASE OF COUNTRIES WITH DIFFERENT TRACKING
SYSTEMS
3
The purpose of our study is to compare the impact of an extra year of schooling on PISA
achievement across several national education systems and explore why that impact may differ
across systems. We first attempt to measure and compare the impact of an extra year of
schooling on PISA achievement in selected countries. Second, we conduct analyses of possible
interaction effects: whether the impact of an extra year of schooling differs for female vs. male
students and for students of higher and lower social class. Third, we explore whether splitting
students into general vs. vocational tracks changes the effects of an extra year of schooling on
achievement. The paper addresses the issue of PISA result interpretation for policy-making:
whether countries with low scores also have low school effectiveness and vice versa. Also
looking at the specific effects of tracking allows us to consider the academic-vocational problem
in a new way.
Keywords: effect of schooling, Programme for International Student Assessment (PISA), quasi-
experimental design, selection in education
JEL Classification: I21, C21.
1
National Research University Higher School of Economics The International Laboratory for
Educational Policy Research; E-mail: jtiumeneva@hse.ru
2
National Research University Higher School of Economics. The International Laboratory for
Educational Policy Research; E-mail: jkuzmina@hse.ru
3
The authors are grateful to Prashant Loyalka (Stanford University) and Martin Carnoy (Stanford University) for their help
during our research.
3
Introduction
For the people who make decisions concerning national education, data about school
efficiency are indispensable, especially if the information stems from comparable perspectives.
An inference about school effectiveness can be made on the basis of value-added data for some
fixed period of time. Usually data of this type come from longitudinal research. Unfortunately
longitudinal data are expensive and infrequent. In some cases an inference about school
effectiveness can also be reached using cross-sectional design. The OECD Programme for
International Student Assessment (PISA) is such a study.
Achievement in PISA shows the cumulative effect of age, learning experience,
curriculum, and the family environment of students rather than a pure schooling effect (OECD,
2006; OECD, 2010). To disentangle the effect of schooling from others a regression
discontinuity (RD) approach can be applied. Since the PISA sample includes students of the
same age (generally 15 years old) and enrolled in a higher or lower grade, the RD method is used
to assess the grade effect.
In trying to assess the effect of one extra year of schooling on PISA data, there is only
one serious drawback: the different educational tracks that pupils can take up to their 16th
birthday. Usually tracking takes the form of clear separate sectors in the education process,
typically specializing in general and vocational education. In many countries students have to
choose between the tracks at about 15 years of age. If the PISA outcomes occur after this
branching has happened, it is difficult to say which effect of schooling we assess with PISA
scores: general education, vocational education or a mix of both. This problem can guide many
of the current studies as most of them, to assess a grade-effect in a cross-sectional frame, treat
the cases without a division in different educational tracks.
Many authors show that vocational education affects academic achievement (and PISA
results as well) systematically and negatively (Hanushek & Wößmann, 2006; Ammermüller,
2005; Schuetz et al., 2005). So a direct comparison of student achievement from a country with
early tracking and a country where all 15-year-olds are included in the same general track is not
quite fair. This can lead to the under- or overestimation of school efficiency depending on the
intensity and scale of tracking in a country.
Also, politically, it is very important to know what the efficiency of different educational
programs is.
Unfortunately it is not possible to compare directly the grade effect for students from
different tracks because the selection is not random, but sometimes via individual talent and
skills, sometimes via parental pressure and lobbying. Furthermore, individual skills and talents
4
are also affected by many factors and only part of them relate to schooling. If one ends up in the
vocational track, it might be a part of the whole previous personal and educational story. It
means that any track has an indirect effect, acting through or together with many aspects of the
student’s background. If we used longitudinal data we could fix the starting point of some tracks,
calculate value-added effect for each, and then to compare the direct effect of every track.
Because PISA does not allow for a value-added approach, the separation of the direct and
indirect effect of the tracks still remains challenging when PISA data are used.
This research will be devoted to an evaluation of the absolute effect of schooling based
on data from PISA-2009. In this paper we are going to suggest a way to handle the issue of
tracking and to find the grade-effect for different schooling systems.
Literature review
The body of the previous studies on the grade effect, when the cross-sectional data are
involved, has focused on following points. First, an absolute grade effect has been assessed
(Ceci, 1991; Cascio & Lewis, 2006; Cahan & Davis, 1987; Cliffordson, 2010; Frenette, 2008;
Luyten, 2006). Second, factors associated with efficiency of schooling have been revealed (Heck
& Moriyama, 2010; Artman, 2006). Third, an optimal age to start school has been identified
(Fertig & Kluve, 2005; Mayer & Knutson, 1997; Sprietsma, 2010). Fourth, the grade effect has
been compared to the effect of aging (Alexander & Martin, 2004; Cahan & Cohen, 1989; Cahan
& Davis, 1987; Cliffordson, 2010; Crone & Whitehurst, 1999).
Our study will be devoted mainly an evaluation of the absolute grade effect of schooling.
The most widespread methods to handle cross-sectional data to reveal the grade-effect are
instrumental variable (IV) and regression discontinuity (RD) analysis (Imbens & Lemieux,
2008). These methods allow us to avoid a misinterpretation of the grade effect assessed, when
there was a non-random earlier start of schooling. Also IV analysis helps to separate age and
grade effects from each other. The RD method, if it applies to assess the grade effect, supposes
that in every country the start of schooling is based on the age of child. Also the IV method
exploits variation in a student's age relative to age entry cutoffs for primary school (these cutoffs
are different for each country). A combination of these methods with propensity score matching
helps to understand to what extent tracking changes the effect of an extra year of schooling on
academic achievement (Schneider et al., 2007; Brodziak, 2009).
Applying pure RD design on samples without tracking, the achievement gain between
students with an extra year of schooling, and those without have been assessed. Typically the
5
measured grade effect varied from 0,18 SD (Cascio & Lewis, 2006) to 0,53 SD (Luyten, 2006)
and depended on students’ age, race and cognitive domain. Ceci in a literature review on the
grade effect pointed out a variation in the size of the grade effect from 0,25 to 6 IQ points (Ceci,
1991).
Judging by PISA data about 0,5 SD is the typical effectiveness of one extra year of
schooling. Frenette (2008), using the strict RD analysis on Canadian PISA data, showed that the
reading and math domains are more sensitive to schooling (grade effect in these domain was
(0,41 SD), than science (0,33 SD). However the results varied for students with different SES.
For TIMSS
4
scores (when the national TIMSS sample included students from different
grades) one extra year of schooling also had a positive impact, though with a large variance
between countries, schools and subjects (Luyten, 2006). Besides, the grade effect has depended
on the level of academic achievement of primary school students: when achievements were high,
the grade effect was small. Again it has been found that the effectiveness of one extra year of
schooling was lower for science than for math for all countries included (except Iceland): the
grade effect for math varied from 0,25 SD (England) to 0,8 SD (Norway); for science the range
was from 0,19 SD (England) to 0,53 SD (Norway).
It is interesting to note that the assessed effectiveness has not been associated with
countries’ ranks in PISA or TIMSS league. In other words, national performance in the
international programs and an effectiveness of a year of schooling stem from different sources:
PISA/TIMSS scores reflect the cumulative effect of very different factors, while the grade effect,
as we said, shows precisely the schooling impact on national achievement.
Among the factors which change the size of the grade effect, researchers have pointed out
gender (for girls the grade effect was higher than for boys), and the number of books in the home
(this factor affected grade effect positively) (Luyten, 2006). These two factors were not stable
across countries assessed. Another strong factor affecting the grade effect was the teaching
practice which included the quality of teaching, student support, professional training for
teachers, and how much attention the school pays to improving instruction. In general this factor
could change the grade effect by half (Heck & Moriyama, 2010).
When a large proportion of students in each country does not comply with the birth date
cutoff rule, the student’s relative age as an instrumental variable for grade level has been used
instead of a sharp RD design (e.g., Fertig & Kluve, 2005).
All of the above studies, in assessing the grade effect deal with only the general
educational track. Indeed, the educational systems in the countries Sweden, Norway, England,
4
Trends in International Mathematics and Science Study (Третье международное исследование по оценке качества
математического и естественнонаучного образования).
6
Canada, Iceland and some others provide only one track for the cohort of students who are in the
research sample. As a result the researchers did not have to take into account learning on
different tracks. Methodological differences in these studies have mainly stemmed from the
specific research questions.
We did not manage to find any studies on the grade effect which took the different
educational tracks into consideration as a factor biasing the assessed effectiveness of schooling.
Certainly, vocational tracks in education have become the subject of attention for many
researchers. They treated the vocational pathway in four different ways. First, they compared the
effect of tracking on achievement by a pretest, controlling for prior ability before tracking, and
post-test for achievement after tracking (Alexander & Cook, 1982; Jencks & Brown, 1975).
Second, tracking (vocational vs. general) can be considered as an outcome of previous
educational background (Sprietsma, 2010). Third, tracking was treated as the independent factor
for future educational carriers (Hanushek &Wößmann, 2006). Fourth vocational education was
considered as a specific subject to reveal the principal features of this type of training in terms of
teacher experience and expectations, curriculum, parental involvement - all of which can cause
specific outcomes for vocational students.
In terms of our research questions we cannot rely fully on the methods used in the above-
mentioned studies. We assess the effect of schooling when this period is divided into different
tracks and we can avoid possible bias caused by the selection process. The other particularity of
our study is that we cannot pretest student skills before their segregation into different programs.
PISA data show that students from the vocational tracks have systematically lower
academic results than students from the general track (Hanushek & Wößmann, 2006;
Ammermüller, 2005; Schuetz et al., 2005). The factors affecting these differences are very
complex. Some combination of interacting factors like student academic achievement and
aspirations, teachers’ and parents’ expectations can lead to different tracks, and thus selection is
only part of the story. Being allocated to different programs students are exposed to different
curriculums, peer influence, and social expectations (Gangl et al., 2003; Manning & Pischke,
2006). Even if there are no final conclusions about the effect of differentiation on student
achievements, we have enough information to see a potential bias when comparing the
effectiveness of educational systems with different tracks and systems without tracking.
There are several programs and a selection process based on non-measured characteristics
of students for many countries. A methodological way to take this factor into account is crucial
for an accurate comparison between schools, tracks and countries.
7
This work is focused on revealing the grade effect in the countries addressing selection
bias problem. Additionally we included the countries without tracking to compare the grade
effect with the first group of countries.
This study answers the following research questions:
1. What is the effectiveness of one extra year of schooling in Russia and how does this
compare with other countries that showed diverse PISA results?
2. What is the grade effect on students from general education in Russia and in other
countries with early tracking?
3. To what extent does the grade effect depend on the social and demographic
characteristics of students?
Method
Sample
In PISA the target population is fifteen-year-old students who attend formal schooling
and can be enrolled in seventh to twelfth grade, although a very high percentage of students are
in the ninth and tenth grades in the most countries.
Countries that were included in analysis: Russia, Germany, Czech Republic, Slovakia,
Hungary, Canada and Brazil.
Repeaters were removed.
Table 1. The description of country samples
N
Total sample
9 grade
10 grade
9
10
General*
vocatio
nal
General*
*
Pre-
vocat
vocatio
nal
N
N
Russia
5308
60%
28%
100%
81%
19%
Slovakia
4555
36%
57%
100%
29%
42%
29%
Czech R.
6064
49%
47%
100%
39%
61%
Germany
4979
55%
33%
100%
100%
Brazil
20127
37%
36%
100%
100%
Hungary
4605
67%
22%
82%
18%
86%
14%
Canada
23207
14%
84%
100%
100%
*from all 9th graders **from all 10th graders
Variables
Outcome variables were math, reading and science performance in PISA 2009.
Treatment was being in 10th grade up to the moment of PISA testing.
8
Instrumental variable was the age of the student. In the selected countries all the students in the
sample were born between January and December 1993. The age was measured as the month of
birth minus the month of the cutoff which was identified for each country.
The other students’ and schools’ characteristics were taken into account:
- gender of student (a binary variable equal to 1 if the student was female and 0 otherwise),
- socioeconomic status (SES) of student, which was derived from three variables related to
family background: the higher parental occupation status, the higher parental education
expressed as year of schooling and the index of home possession, SES score was
obtained as component scores for the first principal component with zero being the score
of an average OECD student and one being the standard deviation
- school location (4 dummy variables: village, small town, town, city; large city is the
reference category
5
).
Analysis Strategy
We identify the causal effect of an extra year of schooling on student achievement using
an instrumental variables strategy based on the regression discontinuity design. In the regression
discontinuity design (RDD), the probability of receiving a given treatment jumps at the cutoff
point along a continuous variable (Hahn et al., 2001). The cutoff point, when established by
policymakers, can often be used as a source of exogenous variation in treatment assignment
(Imbens and Lemieux, 2007). For example, in the sharp RDD, where the probability of receiving
the treatment changes from zero to one at the cutoff point, an average treatment effect can be
identified by comparing the outcomes of “treated” students to the right of the cutoff point with
the outcomes of “control” students to the left of the cutoff point. In the fuzzy RDD, where the
probability of receiving the treatment jumps by less than one at the cutoff point, a local average
treatment effect (LATE) can be identified by using variation in the treatment assignment because
of the cutoff as an instrument for the treatment variable.
6
In contrast to the sharp RDD, the fuzzy
RDD identifies a treatment effect for an unidentifiable group of compliers (Hahn et al., 2001).
Each country in our analytical sample has established a specific birth cutoff to determine
when students were old enough to attend primary school.
Table 2 also shows more clearly that a certain proportion of students in each country did
not comply with the birth cutoff rule. For most countries, the proportion of students that did not
5
For the Slovakia sample “city” has been reference category because there was no category “large city”
6
The probability jumps by less than one at the cutoff point in the fuzzy RDD because individuals do not comply with the
treatment (or control) condition to which they are assigned.
9
with comply the birth cutoff rule was around 20% or less.
7
Because of imperfect compliance
around the birth cutoff, we estimate the LATE of an extra year of schooling in each country
using student’s relative age (age relative to the cutoff point in each country) as an instrumental
variable for grade level.
Table 2. The birth cutoffs and proportion of non-compliers in each country
Country
Birth cutoff
Non- compliers
10th graders who
were born after
birth cutoff (%
from all 10th
graders)
9th graders who
were born before
birth cutoff (%
from all 9th
graders)
Russia
October, 1st
52%
11%
74%
Canada
October, 1st
18%
14%
18%
Slovakia
September, 1st
17%
7%
26%
Czech
September, 1st
20%
1%
38%
Brazil
September, 1st
29%
24%
38%
Germany
July, 1st
12%
12%
11%
Hungary
June,1st
20%
6%
25%
We assume that relative age is a pre-treatment variable that plausibly affects student
PISA scores through the grade level but not through any other (observed or unobserved) pre-
treatment covariate. Given the general level of compliance with the birth cutoff rule in most
countries, relative age should also be correlated reasonably well with grade level.
We also examine whether an extra year of schooling affected certain subgroups of
students (namely female students and students of lower socioeconomic status) more than others.
We ran IV regressions with two interaction terms (grade effect*SES and grade effect*gender) to
estimate the impact of attending an extra year of schooling on different types of students.
The above mentioned models were estimated for the whole national samples for all tracks
combined. After that IV analysis results for general and vocational tracks separately. In
situations with early tracking it was very important to select the students for this kind of analysis
correctly.
If we compare all 9th graders with 10th graders from the general track, we risk over-
estimating the grade effect, because we are not able to take into consideration those 9th graders
who are likely to attend the vocational track and be poor achievers in PISA. Instead we analyzed
the restricted sample: 10th graders from the general track and 9th graders with high probability of
being in the general track in 10th grade.
7
In the case of Russia (and to a lesser extent Brazil), however, the percentage of non-compliers was rather large.
10
These calculations included:
1) First, the regression coefficients were calculated on the sample of all 10th graders (from
both tracks) using logistic regression with the dependent variable of being in general
track and independent variables of the student’s SES and gender.
2) Second, the probability of continuing education in general schools was calculated for all
9th graders. For this calculation the coefficients from the first step were used.
As a result only 9th graders with a high probability of being in general track and 10th
graders from general schools were selected for the RD analysis. To estimate the value of the
grade effect more accurately in each country with general and vocational tracks we choose 9th
graders with three different probabilities of being in the general track and compare the results for
each subsample.
IV analysis for this restricted sample was carried out in the same manner for the whole
sample.
Finally, in all regressions, we accounted for the clustered nature of our sample by
constructing Huber-White standard errors corrected for school-level clustering (relaxing the
assumption that disturbance terms are independent and identically distributed within schools).
Results
Descriptive statistics
Canada and Germany have the highest results in math, reading and science among the
selected countries. These countries have the highest level of socioeconomic status also. Brazil
has the lowest average results in 3 domains and the lowest average SES.
In every country 9th graders have significantly lower results than 10th graders in all
domains. According to our descriptive analysis for the sample without repeaters 9th graders have
lower results than 10th graders in every selected country regardless of whether we take into
account the division into different tracks or consider all 10th graders combined. The highest
difference between scores of 9th and 10th graders was found in Germany, it was more than 50
points or 0,5 standard deviations
8
. In Canada the difference between results of 9th and 10th
graders was estimated as 0,1-0,2 standard deviations and it was the lowest difference among all
selected countries.
There are clear differences in the results between general and vocational schools in every
country where this takes place (see Table 3). Results of 10th graders on general track are better
8
In PISA the average score among OECD countries is 500 points and the standard deviation is 100. About two-
thirds of students across OECD countries score between 400 and 600 points (for details see OECD, 2012).
11
than those of vocational students. In Russia and Slovakia 10th graders from vocational schools
are weaker than 9th graders in general schools. The largest difference between the results of
general and vocational 10th graders is estimated in Slovakia for every PISA achievement domain.
For reading and math performance this difference is more than 130 points and for science it is
127 points. In Czech Republic and Hungary the difference between general and vocational
students in all PISA domains is more than 100 points (1 standard deviation in PISA scores). In
Russia 10th graders from general schools have higher results than vocational students although
among other countries the difference between their scores 9th and 10th grades is lowest.
Also it should be noted that 10th graders from the general track have a higher level of SES
than students from vocational track and there is a higher proportion of female students (see
Appendix, Table 1).
Table 3. PISA scores for each country, grade and track
Countries
Program
Math
Read
Science
9
10
9
10
9
10
Russia
general
466(80)
506(78)
459(85)
499(82)
481(87)
511(84)
vocational
444(76)
428(79)
452(78)
All
466(80)
496(82)
459(85)
487(86)
481(87)
501(86)
Slovakia
general
492 (81)
580(72)
463(79)
558(60)
481(84)
568(69)
vocational
444(73)
421(72)
441(78)
prevocational
501(71)
499(65)
503(73)
All
492 (81)
508(89)
463(79)
494(84)
481(84)
504(88)
Czech
general
482 (84)
604(69)
469(83)
590(58)
494(89)
609(67)
vocational
497(77)
481(75)
504(76)
All
482 (84)
523(88)
469(83)
508(86)
494(89)
530(87)
Hungary
general
516(76)
538(73)
519(71)
548(68)
527(69)
549(66)
vocational
408(60)
427(60)
408(63)
437(60)
427(62)
456(60)
All
499(84)
526(80)
502(80)
536(75)
511(77)
539(71)
Germany
general
517 (87)
575 (79)
504(84)
557(71)
528(90)
579(78)
Brazil
general
397(75)
432(77)
430(82)
469(82)
418(76)
454(76)
Canada
modular
525(85)
536 (83)
516(83)
535(85)
519(84)
539(85)
Causal (IV) analysis’ results for all programs combined
Differentiating the educational programs (general or others) was not taken into account
and the grade effect was estimated for the whole sample of 9th and 10th graders without repeaters
for every country. The entire results of IV analysis for all programs are presented in Table 4.
12
Table 4. The grade effect for PISA 2009 (all programs combined)
Countries
Model 1
Model 2
Model 3
Model 4
Grade effect
(s.e.)
Grade effect
(s.e.)
Grade effect
(s.e.)
Grade
effect*SES
(s.e.)
Grade effect
(s.e.)
Grade
effect*female(s.e.)
Russia
Math
1,98 (11,8)
6,48 (10,8)
9,41(11,1)
14,13(16,1)
-2(15,1)
17,55(23)
Read
10,49(12,7)
14,48(12,21)
17,46(12,67)
14,38(19,1)
18,66(17,76)
-8,64(25,44)
Science
11,72(12,57)
17,26(12,51)
18,54(12,73)
6,22(18,92)
20,91(18,28)
-7,54(24,35)
Slovakia
Math
-10,8**(4,4)
-13,94(9)
-13,03(8,8)
17,03***(5,6)
-8,38(10,8)
-10,9(10,6)
Read
8,36*(4,32)
-2,04(8,39)
-1,37(8,19)
12,63*(6,76)
1,26(10,72)
-6,48(10,15)
Science
2,09(4,41)
-1,69(9,22)
-0,93(9)
14,44**(7,3
6,58(11,13)
-16,24(10,29)
Czech
Republic
Math
29,69***(3,5)
23,38***(7,14)
23,36***(7,12)
4,47(6,11)
20,7**(9,23)
5,39(8,98)
Read
18,32***(3,55)
15,12**(6,2)
15,11**(6,19)
2,42(5,58)
10,89(7,98)
8,51(8,94)
Science
23,45***(3,63)
20,43***(7,11)
20,43***(7,11)
0,94(6,07)
18,74***(8,72)
3,4(9,08)
Germany
Math
39,46***(4,47)
33,45***(3,5)
36,85***(4,9)
-10,16**(4,5)
41,79***(6,08)
-15,45**(7)
Read
31,84***(3,65)
26,79***(3,68)
29,63***(4)
-8,49**(3,93)
29,79***(5,27)
-5,57(6,5)
Science
35,5***(4,01)
29,34***(4,41)
34,63***(4,68)
-15,8***(4,4)
38,76***(5,83)
-17,43***(6,61)
Brazil
Math
25,89***(5,4)
28,47***(5,04)
34,65***(6,8)
7,31(4,6)
32,26***(6,8)
-6,93(9,7)
Read
24,41***(5,74)
27,96***(4,8)
33,5***(5,98)
6,55(4,54)
32,26***(7,4)
-7,84(10,71)
Science
16,88***(5,48)
19,53***(4,42)
25,53***(6,01)
7,09*(4,25)
25,36***(6,39)
-10,62(9,55)
Canada
Math
37,28***(8,11)
31,56***(8,72)
31,9***(9,4)
-0,73(10,5)
29,38***(12,7)
4,32(16,15)
Read
27,47***(8,23)
24,94***(7,84)
23,98***(9,09)
2,03(10,17)
31,1**(12,08)
-12,22(16,42)
Science
30,23***(8,33)
24,79***(8,06)
21,56**(9,13)
6,88(11)
25,81**(12,22)
-2,02(16,73)
Hungary
Math
12,12**(4,8)
10,96**(4,38)
11,04**(4,42)
4,49(4,13)
12,9*(7,12)
-3,46(8,49)
Read
14,78***(4,61)
13,88***(4,18)
14,04***(4,27)
8,75**(3,9)
15,63**(6,76)
-3,09(7,81)
Science
14,47***(4,33)
13,28***(4,01)
13,4***(4,08)
6,62*(3,79)
17,64***(6,68)
-7,72(7,98)
In the first model the grade effect without covariates was estimated. As we can see from
Table 4 the grade effect in Russia is non-significant for all domains. For Slovakia the grade
effect is significant only for math and reading. The grade effect for math in Slovakia is negative.
13
Other countries demonstrated significant positive grade effects. In the first model the relationship
between the effectiveness of schooling for all three domain is not consistent. While in Germany
and Czech Republic the grade effect for reading is the lowest and for math performance it is the
largest, in Slovakia we can see opposite pattern with the highest grade effect in reading and the
lowest in math. Brazil has the biggest grade effect in math and the smallest in science. Grade
effect in Hungary is nearly the same for all PISA domains.
In the second model SES, gender and school location was controlled. In Slovakia and
Russia grade effect was non-significant for every domain. For other countries the grade effect
remained significant and the size of it was close to the grade effect in the model without
covariates.
In Germany and Czech Republic the pattern remains the same: the grade effect for math
performance is the highest and for reading it is the lowest in all three domains. In Brazil the
grade effect for science is the lowest. In Canada again the grade effect in math is higher than in
reading and science. In Hungary the grade effect in math is lowest among other domains,
although the differences are minimal.
In the third model interaction term between SES and grade effect was added. The
coefficient of this term was significant in four countries: Germany, Slovakia, Hungary and
Brazil. In Germany this coefficient was negative for three domains showing that the size of the
grade effect increases with decreasing SES. In science performance, for instance, it is 15,8 points
(nearly half of the mean grade effect in Germany), meaning that difference in results for 9th and
10th graders with high SES is smaller than for students with low SES. In other countries a
contrary pattern was found. In Slovakia the grade effect increases for student with high SES for
math, reading and science achievements. The size of coefficient of the interaction between 12,6
(for reading performance) and 17 points (for math). In Hungary the interaction term between
SES and the grade effect was significant in science and reading performance and it is less than
10 points (0,1 standard deviation). In Brazil the coefficient of the interaction term was significant
(at level 0,1) only for science performance.
In the fourth model interaction term between gender and the grade effect was added. The
coefficient of this term was significant only in Germany for math and science achievements. This
coefficient is negative meaning that girls have a lower grade effect in science and math than
boys. The size of the grade effect is nearly half of average grade effect in Germany.
The results of the estimation of the grade effect without taking into account different
educational tracks show that the grade effect is higher in countries where all 9th and 10th graders
are on the same track (Germany, Canada and Brazil). The low value of the grade effect in
countries with vocational and prevocational students in the PISA sample suggests it can partially
14
depend on the low results of vocational students. That is why it is important to estimate the grade
effect for general schools to compare it with the grade effect for all programs.
Causal (IV) results for general track
In three countries (Slovakia, Czech Republic and Russia) all 9th graders are in the general
schools, but some of them can pass to vocational school after finishing 9th grade. That is why our
first aim was to select 9th graders who have a high probability of continuing general education in
order to compare them with 10th graders of general schools.
The first step was to calculate the probability of being in a general program for 10th
graders in all these three countries (Russia, Czech Republic and Slovakia). According to the
results of a logistic regression girls and high SES students have higher probability of continuing
education in general schools. In Slovakia and Czech Republic SES is the most meaningful
predictor, in Russia gender is more important than SES (See Appendix, Table 2).
The next step was calculating the predicted probability of continuing education for 9th
graders in general track (See Appendix, Table 3).
In Slovakia and Czech Republic the average probability of being in the general track is
not very high in comparison with Russia where the average value of the predicted probability
was estimated as 0,79. For Slovakia average value of the predicted probability to continue
education in general program was 0,28; it is the lowest result among the three countries.
The third step was to select 9th graders with a high probability of continuing a general
education. At this stage some issues can arise. The value of the grade effect will depend on
which value of probability we estimate as “high”. At the beginning we selected 9th graders with
probability higher than the average value in each country (more than 0,8 in Russia; more than 0,3
in Slovakia; and more than 0,4 in Czech Republic). Then we tried some other indicators of
“high” probability to analyze how the grade effect can change depending on different values of
probability of continuing general education. In Russia and Czech Republic we have chosen two
other values of predicted probability: the first is lower than the average value, the second is
higher than the average value. In Slovakia we have chosen two indicators also; but both of them
were lower than the average value (0,2 and 0,25) because of the small number of 9th graders that
would remain in the sample if we choose probability more than 0,3. It may be explained by the
small number of 9th graders in the Slovakian sample and the small number of 10th graders in
general schools in comparison with Russia and even Czech Republic. The values of the
probability and the number of 9th graders with this value in each country is given in Table 5.
15
Table 5. The number of 9th graders with a different value of probability of continuing
education on a general track (without repeaters)
Russia
Slovakia
Czech Rep
Probability
>0,7
> 0,8
>0,85
>0,2
>0,25
> 0,3
>0,3
> 0,4
>0,5
Number
2174
1713
1073
889
665
511
1570
1164
843
% (from 9th
or 10th
graders)
74%
58%
37%
60%
45%
35%
55%
41%
29%
In Hungary the selection process to estimate the grade effect in general schools was
easier. We selected 9th and 10th graders from general schools and we did IV analysis for this
restricted sample.
For each selected country which differentiated the educational tracks after the 9th grade
we assessed the grade effect using four models (as for all programs) for three subsamples. In
each subsample 10th graders of general schools and 9th graders with different values to continue
general track were included (See Table 6).
16
Table 6. The grade effect for general track (PISA 2009)
Coun
tries
Probabilit
y to be in
general
Model 1
Model 2
Model 3
Model 4
Grade effect
(s.e.)
Grade effect
(s.e.)
Grade effect
(s.e.)
Grade
effect*SES
(s.e.)
Grade effect
(s.e.)
Grade
effect*female(s
.e.)
Russia
Math
>0,7
15,6(12,9)
25**(11,9)
25,41**(11,9)
5,47(18,4)
29,56*(16,7)
-7,85(25,5)
>0,8
16,3(12,8)
22,44*(13,3)
22,59*(13,3)
2,46(15,5)
14,93(22,8)
10,7(28,9)
>0,85
11,37(13,4)
29,05*(15,9)
27,35*(16,1)
20,5(17,7)
57,48(45,7)
-33,32(47,3)
Read
>0,7
16,58 (13,6)
31,1**(13,1)
31,9**(13,3)
11,04(19,9)
53,2***(18,2)
-38,2(27,8)
>0,8
6,63(13,2)
26,56*(15)
27,55*(14,9)
15,88(17,1)
53,9*(22,8)
-38,98(29,8)
>0,85
-8,43(14,2)
29,08(18,9)
27,52(18,8)
18,8(21)
98,39*(46)
-81,26*(46)
Science
>0,7
19(13,8)
29,6**(13,3)
30,31**(13,4)
10,58(19,9)
47,52*(18,7)
-31,08(25,92)
>0,8
15,32(13,57)
23,82(14,96)
24,57*(14,91)
11,95(16,6)
40,22*(24,1)
-23,39(28,41)
>0,85
8,18(14,26)
28,79(17,52)
26,72(17,32)
25,02(19,3)
100,9**(47,3)
-84,48*(46,49)
Slovakia
Math
>0,2
59,9***(6,5)
51,6***(9,5)
51,9***(10,3)
-0,89(10)
77,5***(13,5)
-42,5***(13,6)
>0,25
53,14***(7)
49,8***(10,4)
47,1***(12,6)
5,2(12)
79,1***(15,3)
-46,8***(14,7)
>0,3
50,59***(7)
50,54***(10,6)
46,9***(14,5)
5,5(14)
77,7***(14,6)
-44,3***(14,8)
Read
>0,2
69,6***(6,2)
53,45***(8,4)
55,1***(9,1)
-4,53(8,8)
77,5***(13,7)
-39,4***(13,3)
>0,25
62,5***(6,4)
48,94***(9)
48***(10,5)
1,8(10,1)
70,9***(14,5)
-35,2***(13,6)
>0,3
60,1***(6,5)
45,33***(9,3)
40,2***(11,8)
7,9(11,4)
66,4***(14)
-34,3***(13,9)
Science
>0,2
64,3***(6,7)
55,04***(9,2)
57,47***(9,9)
-6,76(8,74)
83,9***(13,5)
-47,2***(13,6)
>0,25
57,88***(7)
51,89***(9,8)
53,6***(11,6)
-3,2(10,5)
79,9***(14,4)
-44,9***(14,5)
>0,3
54,2***(7,2)
49,89***(9,8)
51,3***(13,2)
-2,2(12,5)
77***(13,4)
-44,2***(14,6)
Czech Republic
Math
>0,3
95,5***(6,5)
90,05***(6,5)
102,8***(7,3)
-28,2**(7,8)
103***(10,3)
-19,4*(10,4)
>0,4
86,8***(5,3)
86,24***(7,1)
102,5***(8,4)
-29,5***(9)
98,7***(11,4)
-18,46(11,3)
>0,5
82,5***(5,6)
83,65***(7,7)
107,8***(11)
-34,6***(11)
92,7***(12,6)
-13(12,3)
Read
>0,3
75,8***(5,2)
76,12***(5,6)
90,3***(7,27)
-31,5***(8,3)
82,27***(9,7)
-9,19(11,34)
>0,4
67,8***(5)
70,36***(6,2)
85,97***(8,3)
-28,3***(8,7)
76,6***(10,6)
-9,24(11,6)
>0,5
61,6***(5,2)
65,48***(6,7)
85,26***(11)
-28,3***(11)
66,38***(14)
0,7(12,55)
Science
>0,3
95,5***(5,2)
90,05***(6,5)
102,8***(7,3)
-28,2***(7,8)
103***(10,3)
-19,4*(10,4)
>0,4
81,7***(5,5)
83,51***(8,3)
101,1***(11)
-31,9***(10)
94,8***(12,7)
-16,68(12,85)
>0,5
76,1***(5,7)
78,91***(8,8)
102***(14,3)
-33,06*(13,4)
87,3***(13,6)
-11,95(13,88)
Hung
ary
Math
From
general
schools
14,7***(4,6)
12,26***(4,29)
7,74(13,22)
-5,16(15,4)
6,66(6,9)
12,91*(7,5)
Read
17,7***(4,3)
15,78***(4,03)
15,44***(4,1)
4,24(3,84)
16,24**(6,46)
-0,79(7,4)
Science
15,4**(7,01)
6,09(6,39)
5,98(6,57)
12,69*(6,98)
16,27 (11,12)
-16,84(13,45)
*** p<0.01, ** p<0.05, p* <0.1
The first model was a model without covariates and assessed the grade effect only. For
Russia the grade effect was non-significant for any level of probability and for each PISA
achievement domain. In Slovakia and Czech Republic the grade effect is significant and positive
for any level of probability and in all three domains. It should be noted that the value of the
grade effect decreases when the probability of continuing general education for 9th graders
increases.
According to the results of the first model, in Czech Republic the grade effect for general
schools is the highest among other selected countries. We can see the same pattern for all
programs combined: the grade effects for math and science performance are nearly the same, and
17
they are lower for reading than for math and science. For math and science the grade effect is
about 90 points (0,9 standard deviation), for reading it is about 70 points (0,7 standard
deviation).
Slovakia has a big grade effect too. For general schools in Slovakia the grade effect is
higher than for any other country (except Czech Republic) with or without tracking. The Grade
effect in Slovakia is nearly the same for all three domains and it is about 60 points (0,6 standard
deviation).
The grade effect in Hungary for general schools does not differ very much from the grade
effect for all programs and it is about 15 points for all PISA achievement domains.
The second model was a model with covariates. In Russia the grade effect becomes
significant when we control for SES, gender and school location. Although for science the grade
effect is significant only at a low level of probability (0,7) of continuing general education. The
grade effect for general schools in Russia is higher than for all programs but not very large. It is
lower than 30 points.
In Slovakia the grade effect becomes lower when SES, gender and school location are
controlled for, although these changes are significant only for reading performance. For this
domain the grade effect decreases by nearly 15 points in comparison with the estimated grade
effect in the model without covariates. In the model with covariates the grade effect is about 50
points (0,5 standard deviation) and does not change significantly depending on PISA domain or
9th graders’ probability of continuing general education.
In Hungary the grade effect for science achievement becomes non-significant after taking
into account student and school variables.
The results of the models with interaction terms for general schools differ from results of
these models for all programs.
The grade effect in Czech Republic is significantly lower for general students with high
SES. It is true for any level of probability and each domain of PISA achievements. The grade
effect decreases by nearly 30 points when SES increases by 1 point. In Hungary the grade effect
is higher for students with high SES only for science achievements.
In Slovakia the value of the grade effect does not change with an increase in a student’s
SES but it is different for boys and girls. Girls from general schools have a smaller gain in results
for one year of schooling in math, reading and science than boys. The difference in the grade
effect between boys and girls is about 40 points (0,4 standard deviation).
In Russia the grade effect does not differ across different SES and gender.
18
Causal (IV) results for vocational track
The grade effect for the vocational track was estimated only for Czech Republic and
Slovakia because of the small number of vocational students in Russia and Hungary. In Hungary
the sample included only 527 9th graders and 139 10th graders on vocational programs. In Russia
the sample totals only 295 10th graders on vocational track.
To estimate the grade effect for the vocational track the predicted probability that 9th
grader moves to the vocational track was assessed. We used the same method to estimate
predicted probability as for general orientation but the outcome variable was “being on the
vocational track”.
The mean probability of being on the vocational track for 9th graders in Slovakia is 0,29. In
Czech Republic the predicted probability of being on the vocational track is much higher than in
Slovakia; it is nearly 0,6 (See Appendix, Table 4).
As for the estimation of the general track we chose three levels of predicted probability of
being in a vocational school and ran IV analysis for three subsamples (See Appendix, Table 5).
According to IV results for the vocational track, the grade effect for the vocational track
is much lower than for general programs.
Table 7. The grade effect for vocational schools
Countries
Probabil
ity to be
in
vocation
al
Model 1
Model 2
Model 3
Model 4
Grade effect
(s.e.)
Grade effect
(s.e.)
Grade effect
(s.e.)
Grade
effect*SES
(s.e.)
Grade effect
(s.e.)
Grade
effect*female
(s.e.)
Slovakia
Math
>0,2
-50,9***(5,4)
-62,2***(12,9)
-55,6***(13,9)
14,59(11,8)
-51,2***(14,6)
-32,46*(13,1)
>0,3
-46,5***(5,7)
-55,5***(14)
-44,7***(16,3)
20,5(15,3)
-49,5***(15,1)
-25,1*(14,43)
>0,4
-44,1***(6,1)
-47,4***(14,5)
-39,2**(18,1)
13,8(18,9)
-46,53***(15)
-6,93(19,1)
Read
>0,2
-37,1***(5,4)
-49,4***(11,5)
-48,4***(11,9)
2,25(10,8)
-41,9***(12,2)
-22,41(14,1)
>0,3
-24,1***(5,1)
-41***(12,6)
-36,8***(13,9)
7,99(13,1)
-39,1***(12,9)
-8,17(14,6)
>0,4
-16,5***(6)
-35***(13,5)
-34,66**(15,8)
0,58(15,6)
-37***(13,2)
16,6(19,4)
Scien
ce
>0,2
-40,1***(5,7)
-50,8***(13,3)
-45***(13,5)
13(14,1)
-38,2***(13,7)
-37,2***(14)
>0,3
-32,8***(6,1)
-41***(14,6)
-27,53*(15,8)
25,5(18,3)
-34,1**(14,6)
-28,9*(15,7)
>0,4
-29,2***(6,5)
-32,7**(15,4)
-18,7(18,1)
23,5(22,9)
-31,09**(14,9)
-13,1(21,4)
Czech Republic
Math
>0,5
17,5***(3,9)
5,8(8,3)
3,1(8,6)
-11,2(8,1)
6,98(10,3)
-2,81(10,9)
>0,6
22,2***(4,1)
7,7(8,5)
4,45(9)
-10(8,89)
8,03(10,25)
-5,98(17,6)
>0,65
24,3***(4,2)
7,17(8,6)
2,19(9,5)
-12,6(9,6)
7,19(10,3)
-0,05(11,5)
Read
>0,5
9,09**(3,9)
-0,6(7,2)
-3,3(7,5)
-11,2(8)
-0,6(9,1)
-0,04(10,9)
>0,6
15,7***(4,04)
1,28(7,5)
-1,95(8,01)
-10(8,9)
0,25(9,1)
2,58(11,43)
>0,65
18,3***(4,2)
0,56(7,5)
-4,41(8,4)
-12,6(9,8)
-0,66(9)
3,23(11,7)
Scien
ce
>0,5
10,7***(4)
2,85(7,5)
-1,46(8)
-18,1**(8,2)
5,45(9,5)
-6,06(10,5)
>0,6
15,3***(4,2)
4,9(7,7)
-0,66(8,59)
-17,3*(9)
6,8(9,53)
-4,8(11,09)
>0,65
17,3***(4,3)
4,86(7,8)
-3,4(8,9)
-20,9**(9,6)
5,99 (9,5)
-2,99(11,3)
*** p<0.01, ** p<0.05, p* <0.1
In Slovakia the grade effect for the vocational track is negative for all domains and any
value of predicted probability. It becomes smaller if the predicted probability increases. The
19
most negative difference between results for 9th and 10th graders on the vocational track was
found for math performance, it is nearly 50 points (0,5 standard deviation).
In Czech Republic the grade effect for the vocational track is significant and positive. It is
almost the same as for all programs and it is about 20 points (0,2 standard deviation) for math,
reading and science performance (See Table 7).
In Slovakia the negative grade effect for girls on the vocational track is higher for girls
for math and science achievements. In Czech Republic grade effect is lower for high SES
students but only for science performance.
Discussion
The objective of this paper was to estimate the effectiveness of one extra year of
schooling based on PISA data. Despite having examples of similar attempts the existence of
several educational programs in some countries complicate this task seriously. By and large the
difference in PISA performance which students with almost the same age but in consecutive
grades demonstrate can be regarded as the effect of one year of schooling. However for this
interpretation the following demands have to be met.
First of all, the effect of maturing can lead to higher scores in a higher grade. We can see
this pattern from our analysis as well as from PISA reports (OECD, 2007, 2010). Even if the age
effect has been shown for younger students rather than for senior (Alexander & Martin, 2004;
Cahan & Cohen, 1989; Cahan & Davis, 1987; Cliffordson, 2010; Crone & Whitehurst, 1999), it
should be controlled.
The second problem was that more advantaged students start their schooling earlier than
students. In other words the difference between students who have been taught one year more
and those who have been taught one year less to the moment of PISA testing can be
overestimated because of non-random allocation of students around cut-off date (Cascio &
Lewis, 2006; Cliffordson, 2010; Frenette, 2008; Luyten, 2006; Heck & Moriyama, 2010;
Brodziak, 2009). We can see an unequal distribution of social and cultural resources between
students with earlier or later start of schooling. To take the age effect into account and to control
for the bias due imperfect compliance with birth date rule we used instrumental Variable
Analysis and Regression Discontinuity design.
The third problem was non-random selection into different educational tracks in the
countries where the several programs exist. A large body of research has shown that school and
parental expectations and pressure, selective support for individual abilities of students as well as
the specialization effect and peer effect tracking can influence significantly academic
20
achievements and skills (e.g., Hanushek & Wößmann, 2006; Ammermüller, 2005; Schuetz et al.,
2005; Gangl et al., 2003; Manning & Pischke, 2006). We can see from our data that students
from vocational schools have lower results than students from general programs. To handle this
selection bias a restricted sample of students was analyzed. It consisted of only 9th graders with a
high probability of being in general track and 10th graders from general track. This propensity
score matching procedure allowed us to compare similar groups of students in terms of their
SES, motivation, aspirations, abilities and other unmeasured characteristics.
Returning to our research questions it is possible to outline the main outcomes of this
study.
What is the effectiveness of one extra year of schooling in Russia and how does this
compare with other countries that showed diverse PISA results? One extra year of schooling in
Russia for students from all tracks combined turns out to be insignificant for PISA results (in
reading). It implies that for PISA achievements it does not matter whether students have been
taught for 9 or 10 years up to the moment of PISA testing. In comparison with other countries
included in the analysis, it has the lowest grade-effect, although an unfavorable pattern has been
found for all ex-socialist countries: Hungary, Slovakia and Czech Republic show a lower grade-
effect than Germany, Canada and even Brazil.
In addition to a shared socialist past, Russia, Hungary, Slovakia and Czech Republic have
been linked with the practice of early educational tracking. In Hungary this educational
diversification occurs earlier than in other countries studied – in the 9th grade. Taking into
consideration the negative effect of the vocational track on academic achievements repeatedly
confirmed (Hanushek &Wößmann, 2006; Ammermüller, 2005; Schuetz, Ursprung & Wößmann,
2005), it makes sense that in this case we are dealing with exactly this effect. If we want to
compare the different national systems fairly, we need to allow for the presence of several
educational tracks in some countries.
What is the grade effect on students from general education in countries with early
tracking? Assessing the grade effect on students from the general education track, it can be seen
that the grade effect is higher than the grade effect for all tracks combined. Thus, in Russia when
we control for the social-demographic characteristics of students (Model 2), we find a significant
grade effect for math and reading performance. In other countries with two educational tracks for
15-year-olds (Hungary, Slovakia and Czech Republic) the grade effect of the general track is
also higher than for all tracks combined. It means that vocational tracks in all these countries
have lower effectiveness than general tracks. In this case it is true even if all latent characteristics
are controlled for. Being on a vocational track implies by itself a low effectiveness of schooling
regardless of the initial skills and motivation of students.
21
In Czech Republic the gap between the grade effect for all tracks combined and the grade
effect for the general track is the largest of the countries selected; in Hungary this gap is
smallest. Obviously, this gap corresponds to the difference in the quality of education on
different tracks. In Russia this gap, which is approaching one third of a standard deviation,
shows a visible flaw in primary and secondary vocational education.
To what extent does the grade effect depend on the social and demographic
characteristics of students? As a rule, the relationship between SES and the grade effect differs
depending on the educational system. This analysis only confirms these previous conclusions.
Among the countries selected only Germany shows a negative relationship between SES and the
grade effect. It is probable that in Germany, the school system takes into account the needs of
disadvantaged students. In Czech Republic such a negative relationship has been found only in
the general education track.
In Slovakia and Hungary a positive relationship between SES and the grade effect has
been found: students with high SES benefit from schooling more than disadvantaged students.
In Russia, Brazil and Canada the grade effect remains the same for students regardless of
SES. Does it show the equality of educational possibilities in these countries? This is likely,
although, in the case of Russia equality can only be found in the general track.
Generally in Russia the following picture can be drawn. The age to start schooling
depends to a large extent on parental expectations and the preliminary skills of the child. More
skilled children go to school earlier and, up to the moment of PISA testing, they learn longer
than their less skilled peers. The effect of latent family presence can be traced to the link
between PISA achievement and the SES of the student. This family trail will affect the selection
of students after 9th grade, when roughly one fifth of students leave general education and start
the vocational track. From this moment we have two indicators for the grade effect. The first
shows that students who remain in general education benefit from one year of schooling
significantly: their reading literacy gains. The second indicator shows the grade effect for all
students, those who leave general education and those who remain. Here it does not matter
whether they have been taught for nine or ten years: there is no grade effect. The simple
conclusion results from these two indicators and it concerns the grade effect for students who
leave the general track after 9th grade: one year of schooling affects their achievements
negatively. In other words the grade effect for a fifth of Russian students is negative.
Unfortunately, it is impossible to specify which year of schooling exactly this is: whether
this year has been spent in vocational school or it was a year of general schooling. As has been
shown, it is not possible to specify the moment when the effect of vocational school occurs. If
the student is recondemned to leave general school after 9th grade based on past personal history,
22
the anticipation of this event (from parents, teachers and students themselves) can affect the
efficiency of learning from primary school. To speak more accurately about the grade effect in
Russia, we need to keep in mind different groups of 15-year-olds and consider varied grade
effects related to them: the significantly positive grade effect for students who remain in general
school after 9th grade; the negative effect for students who leave the general school; and the
neutral, that is no effect, for all 15-year-olds for both groups combined.
Summary and conclusions
1) PISA scores cannot be regarded as indicators for effectiveness of national educational
systems. Firstly, the duration of schooling differs across different countries and the duration
influences the PISA results. Secondly, the age to start formal education has its own effect on
academic success and the age of starting school varies across countries. Finally, being in the
vocational schools decrease PISA results and, as we can see, the proportion of general and
vocational students in PISA sample is different across counties.
2) Regression Discontinuity design generally allows for the estimation of effectiveness of
national educational systems based on the cross-sectional PISA data if we control the possible
bias by additional analyses like IV and propensity score matching. In this case the grade effect
is interpreted as a value-added estimation for schooling.
Considering the case of Russia, the grade effect is not significant for students from all
programs combined. Also it should be noted that the grade effect in Russia was lowest among
countries analyzed (except Slovakia). The effect of one extra year of schooling for students from
the general track is higher than for students from all programs but still lowest among countries
analyzed. In Russia the grade effect does not depend on student’s gender and SES while in other
countries the effectiveness of schooling was differ depending on gender and SES.
23
References
Alexander, K. L., & Cook, M. A. (1982). Curricula and coursework: A surprise ending to a
familiar story// American Sociological Review, 47, 626-640.
Alexander, J. R. M. & Martin, F. (2004). The end of the reading age: grade and age effects in
early schooling// Journal of School Psychology, 42, 403-416.
Ammermüller, A. (2005). Educational Opportunities and the Role of Institutions. ZEW
Discussion Papers 05-44, ZEW - Zentrum für Europäische Wirtschaftsforschung / Center
for European Economic Research.
Angrist, J. D., Imbens, G. W., Rubin, D.B. (1996). Identification of Causal Effects Using
Instrumental Variables// Journal of the American Statistical Association, 91(434), 444-455.
Brodziak, I. (2009). School Matters: Perspectives On Differences In Student Achievement In
Mexico. Dissertation submitted to the school of education and the committee on graduate
studies of Stanford University in partial fulfillment of the requirements for the PhD degree.
Stanford University.
Cahan, S. & Cohen, N., (1989). Age versus Schooling Effects on Intelligence Development//
Child Development, 60, 1239-1249.
Cahan, S. & Davis, D., (1987). A Between-Grade-Levels Approach to the Investigation of the
Absolute Effects of Schooling on Achievement// American Educational Research Journal,
24, 1-12.
Cascio, E. & Lewis, E. (2006). Schooling and the Armed Forces Qualifying Test: Evidence from
School‐Entry Laws// The Journal of Human Resources, 41(2), 294‐318.
Ceci, S. J. (1991). How much does schooling influence general intelligence and its cognitive
components? A reassessment of the evidence// Developmental Psychology, 27, 703–722.
Cliffordson, C. (2010). Methodological issues in investigations of the relative effects of
schooling and age on school performance: the between-grade regression discontinuity
design applied to Swedish TIMSS 1995 data// Educational Research and Evaluation, 16,
39-52.
Crone, D. A. & Whitehurst, G. J. (1999). Age and Schooling Effects on Emergent Literacy and
Early Reading Skills// Journal of Educational Psychology, 91, 604-614.
Fertig, M. & Kluve, J. (2005). The Effect of Age at School Entry on Educational Attainment in
Germany, IZA Discussion Paper 1507.
24
Ertl, H. (2006). Educational Standards and the Changing Discourse on Education: The Reception
andConsequences of the PISA Study in Germany// Oxford Review of Education, 32, 5,
619-634.
Frenette, M. (2008). The Returns to Schooling on Academic Performance: Evidence from Large
Samples Around School Entry Cut-off. Analytical Studies Branch Research Paper Series,
317.
Gangl, M., Müller, W. & Raffe, D. (2003). Conclusions: explaining cross-national differences in
school-to-work transitions. In: Walter Müller, Markus Gangl (Eds.): Transitions from
Education to Work in Europe. The Integration of Youth into EU Labour Markets. Oxford.
Hanushek, E. & Wößmann, L. (2006). Does Educational Tracking Affect Performance and
Inequality? Differences-in-Differences Evidence Across Countries // Economic Journal,
116, 63-76.
Hanushek, E. & Wößmann, L. (2008). The Role of Cognitive Skills in Economic Development.
// Journal of Economic Literature, American Economic Association, vol. 46(3), 607-68.
Heck, R. H. & Moriyama, K., (2010). Examining relationships among elementary schools’
contexts, leadership, instructional practices, and added-year outcomes: a regression
discontinuity approach. School Effectiveness and School Improvement// An International
Journal of Research, Policy and Practice, 21, 377-408.
Imbens, G. & Angrist, J. (1994). Identification and Estimation of Local Average Treatment
Effects// Econometrica, 61(2), 467-476
Imbens, G. & Lemieux, T. (2008). Regression Discontinuity Designs: A Guide to Practice //
Journal of Econometrics, 142(2), 615-635.
Jencks, C. L., & Brown, M. (1975). The effects of high schools on their students. Harvard
Educational Review, 45, 273-324.
Luyten, J. W. (2006). An empirical assessment of the absolute effect of schooling: regression-
discontinuity applied to TIMSS-95// Oxford Review of Education, 32, 397-429.
Luyten, H., Peschar, J., Coe R. (2008). Effects of Schooling on Reading Performance, Reading
Engagement, and Reading Activities of 15-Year- Olds in England// American Education
Research Journal, 45(2), 319-342.
Manning, A. & Pischke, J. (2006). Comprehensive Versus Selective Schooling in England and
Wales: What Do We Know? CEE Discussion Papers 0066. Centre for the Economics of
Education, LSE.
Martin, M.O., Mullis, I.V.S., & Foy, P. (with Olson, J.F., Erberber, E., Preuschoff, C., & Galia,
J.). (2008). TIMSS 2007 international science report: Findings from IEA’s Trends in
25
International Mathematics and Science Study at the fourth and eighth grades. Chestnut
Hill, MA: TIMSS & PIRLS International Study Center, Boston College.
Mayer, M. E. & Knutson, D., (1997). Does Age at Enrollment in First Grade Affect Children’s
Cognitive Test Scores? Harris School of Public Policy Studies.
OECD (2006). Assessing Scientific, Reading and Mathematical Literacy: A Framework for
PISA 2006, OECD Publishing.
OECD (2007). PISA 2006: Science Competencies for Tomorrow’s World, Vol. 1, OECD
Publishing, Paris.
OECD (2009). Top of the Class High Performers in Science in PISA 2006. OECD Publishing,
Paris.
OECD (2010). PISA 2009 Results: What Students Know and Can Do – Student Performance in
Reading, Mathematics and Science (Volume I). OECD Publishing., Paris.
OECD (2012). Lessons from PISA for Japan, Strong Performers and Successful Reformers in
Education, OECD Publishing, Paris.
OECD (2012). PISA 2009 Technical Report. OECD publishing, Paris.
Santos, M. E. (2009, August). Human Capital and the Quality of Education in a Poverty Trap
Model. Oxford Poverty & Human Development Initiative, Working Paper: 30.
Schneider, B., Carnoy, M., Kilpatrick, J., Schmidt, W. H., & Shavelson, R. J. (2007). Estimating
causal effects using experimental and observational designs. Washington, DC: American
Educational Research Association.
Schuetz, G., Ursprung, H. & Wößmann L. (2005). Education Policy and Equality of
Opportunity. CESifo Working Paper Series 1518, CESifo Group Munich.
Simola, H. (2005). The Finnish Miracle of PISA: Historical and Sociological Remarks on
Teaching and Teacher Education. Comparative Education, 41, 4, 455-470.
Sprietsma, M. (2010). Effect of relative age in the first grade of primary school on long-term
scholastic results: international comparative evidence using PISA 2003// Education
Economics, 18, 1-32.
Suggate, S. P., (2009). School entry age and reading achievement in the 2006 Programme for
International Student Assessment (PISA) // International Journal of Educational Research,
48, 151-161.
Thistlethwaite, D., Campbell, D. (1960). Regression-Discontinuity Analysis: An alternative to
the ex post facto experiment // Journal of Educational Psychology, 51 (6), 309-317.
Appendix
Table 1. Descriptive statistics for SES, gender and schools location (without repeaters)
Countries
Orientation
SES
Female
Schools in village
Schools in
small town
Schools in town
Schools in city
9
10
9
10
9
10
9
10
9
10
9
10
Russia
general
-0,21(0,8)
0,11(0,8)
50%
59%
17%
8%
19%
14%
17%
24%
34%
35%
vocational
-0,22(0,8)
31%
5%
11%
0%
45%
Slovakia
general
-0,09(0,8)
0,39(0,8)
44%
59%
19%
0%
25%
17%
34%
64%
22%
19%
vocational
-0,48(0,7)
39%
1%
13%
82%
4%
Prevocat.
-0,1(0,7)
66%
1%
10%
68%
21%
Czech
general
0,03(0,7)
0,48(0,7)
45%
64%
12%
2%
29%
31%
42%
55%
7%
7%
vocational
-0,15(0,6)
49%
5%
23%
48%
15%
Hungary
general
0,01(0,88)
0,11(0,89)
50%
60%
1%
0,4%
15%
19%
40%
40%
24%
24%
vocational
-0,81(0,7)
-0,89(0,7)
39%
45%
6%
5%
9%
16%
39%
38%
28%
27%
Germany
general
0,19(0,9)
0,46(0,9)
50%
57%
3%
1%
24%
21%
50%
56%
16%
16%
Brazil
general
-1,1(1,2)
-0,88(1,2)
58%
60%
3%
2%
12%
14%
35%
36%
37%
37%
Canada
modular
0,4(0,8)
0,5(0,8)
51%
52%
19%
16%
22%
23%
24%
22%
22%
28%
Table 2. Regression coefficients for dependent variable “to be in general” for 10th graders
Variables
Russia
Slovakia
Czech Rep.
Beta
Exp (Beta)
Beta
Exp (Beta)
Beta
Exp (Beta)
SES
0,629
1,875
1,047
2,849
1,489
4,434
Female
1,19
3,288
0,298
1,347
0,821
2,273
Constant
0,993
2,698
-1,096
0,334
-1,073
0,342
*All coefficients are significant on the level <0,01
3
Table 3. Predicted probability to continue the education in general track for 9th graders
Russia
Slovakia
Czech Rep
Min
0,31
0,01
0,02
Max
0,97
0,85
0,94
Mean
0,79
0,28
0,37
25 percentile
0,69
0,16
0,19
75 percentile
0,88
0,36
0,53
Table 4. Predicted probability to be in vocational track
Russia
Slovakia
Czech R.
Min
0,03
0,01
0,05
Max
0,69
0,98
0,98
Mean
0,21
0,29
0,6
25 percentile
0,12
0,13
0,41
75 percentile
0,31
0,43
0,79
Table 5. Number of 9th graders with different value to continue education in vocational orientation (without repeaters)
Russia
Slovakia
Czech Republic
Probability
>0,15
>0,2
>0,3
10th graders
from
vocational
>0,2
> 0,3
>0,4
10th graders
from
vocational
>0,5
>0,6
> 0,65
10th graders from
vocational
Number
1859
1324
798
295
966
704
475
698
2064
1749
1542
1575
% (from 9th or
10th graders)
63%
45%
27%
18%
65%
48%
32%
28%
72%
61%
54%
59%
Contact details:
Yulia A. Tyumeneva, National Research University Higher School of Economics, The
International Laboratory for Educational Policy Research, senior researcher.
E-mail: jtiumeneva@hse.ru
Tel.: 8-910-407-51-40
Any opinions or claims contained in this Working Paper do not necessarily
reflect the views of HSE.
