Taking PISA Seriously: How Accurate are Low-Stakes Exams?

Abstract

PISA is seen as the gold standard for evaluating educational outcomes worldwide. Yet, being a low-stakes exam, students may not take it seriously resulting in downward biased scores and inaccurate rankings. This paper provides a method to identify and account for non-serious behavior in low-stakes exams by leveraging information in computer-based assessments in PISA 2015. Our method corrects for non-serious behavior by fully imputing scores for items not taken seriously. We compare the scores/rankings calculated by our method to the scores/rankings calculated by giving zero points to skipped items as well as to the scores/rankings calculated by treating skipped items at the end of the exam as if they were not administered, which is the procedure followed by PISA. We show that a country can improve its ranking by up to 15 places by encouraging its own students to take the exam seriously and that the PISA approach corrects for only about half of the bias generated by the non-seriousness.

Appendix:

This appendix delves into more detail on a number of peripheral facts and issues. In the first part we present some non causal regressions on who are non serious and which questions tend to be taken non seriously.

We use the data on questions in the Science clusters only in the body of the paper. Our reason for doing so is that all students take two Science clusters, but may not be tested in Math or Reading. One might ask whether the patterns in other parts of the exam are similar. As a check we look at the fraction of non serious items across subjects in the third part of the A. Their similarity reassures us that our focus on the Science clusters is warranted.

In the fourth part of the A, we discuss in more detail the behavior patterns of serious and non-serious students in terms of time spent and accuracy of response as a function of question position.

In the fifth part, we discuss the exact variables we use in the imputation procedure. In the sixth part, we explain some details behind the decomposition for partially serious students and present the results for them.

A.1 What Drives Being Non-serious?

We have seen in Section 3 that serious and non-serious students behave very differently. The next question is, what factors correlate with being non-serious? We explore this in two levels. First, we look at the correlates of individuals being non-serious. After this, we look at correlates of the question not being taken seriously.

Table 7 Summary statistics

A.1.1 Summary Statistics

In this section we explain the definition of various factors which are potentially correlated with non-serious behavior. Table 7 gives the descriptive statistics for these factors. Scores in the component parts of the exam (reading, math and science) are scaled so that 500 is the mean and the standard deviation is 100 for all OECD countries together. Clearly, OECD countries do better than average as the mean math and science scores overall are 464 and 474 respectively. Students are on average in the 10th grade and half the students are female. The variable “anxiety” is an index we constructed by taking questions that asked about this subject (where the ranking was from a “1” to a “4” in terms of strength of the viewpoint where 1 strongly disagree and 4 is strongly agree) and taking a simple average of the response. The median is 2.8 suggesting a fair degree of anxiety on the part of students. Similarly for “ambition” where the median response is 3.2.⁴² The variable skipping class/arriving late uses the response for the three questions in ST062 about skipping, its intensity and arriving late and adds them up. A 1 is never in the last two weeks, a 2 is 1 or 2 times and a 3 is 3 or 4 times, and a 4 is 5 or more times. On average, such behavior exists but is not endemic.

The median time spent learning out of school is 16 hours per week, while time spent learning in school is 27 hours per week. Students spend more than 40 hours a week on school related work. The standard deviations are roughly 15 and 11 suggesting that a fair number of students are spending well over 60 to 70 hours a week on such work. Standardized test frequency and teacher developed test frequency is the response to question SC034. A response of 1 means there were no such tests and a response of 5 means the tests were given more than monthly. The median value is 2 or the frequency was 1-2 times a year. The variable “Stakes of standardized (teacher developed) tests comes from the answers to SC035. The question is composed of 11 yes/no sub-questions (where a yes is a 1 and a 0 is a no) regarding the purpose of these tests. We label each purpose as low, medium or high stakes for the students giving them a weight of 1, 2 and 3 respectively. Of the 11 sub-questions, 5 are low, 3 are medium and 3 are high stakes. We then add these weighted responses up to get our index. As the maximum value the index could have taken is 20, the median of 10 and 13 suggest the stakes are high, especially of teacher developed tests.

A.1.2 Who is Non-serious?

The factors that correlate with a student being non-serious are explored in Table 8. Column 1 shows the results for all countries. The dependent variable is 1 if the student is non-serious. In columns 1 to 3, being non-serious is defined as meeting at least one of criterion 1, 2 or 4. In column 4, being non-serious is defined as meeting criterion 3. We make this distinction because the patterns explored in the previous section differ across these two groups. We also look at high-stake countries, ones where the standardized tests given in school are high-stakes⁴³, as well as low-stake countries as the patterns in the two might be different. If, for example, students are fed up with exams in high-stakes countries while not in low-stakes countries, we might expect a higher probability of being non-serious in PISA in high-stake countries. One might want to do these regressions country by country, but with 58 countries, this would be overkill as this is not the main object of this paper. Also note that we are not claiming any causal effects, merely pointing out some correlations in the data.

Table 8 Factors related to being non-serious

To begin with, we ask whether better students are more or less likely to be non-serious. Columns 1-3 suggest that higher math scores (a proxy for ability) are associated with a student being less likely to be non-serious, except when we use criterion 3, suggesting that students with missing items are a different breed.⁴⁴ Students with high socioeconomic status (ESCS) and in lower grades are more likely to be non-serious. Again the sign in column 4 is reversed. This suggests that poor able students use criterion 3 when they are non serious while the rest use criterion 1, 2 or 4.

Students from richer countries are more likely to be non-serious, though the shape is that of an inverted U with a turning point at about $33,000 for per capita GDP. However, this pattern is again reversed in column 4 where the pattern is U shaped with a turning point at about $38,500.

Gender matters: women are less likely to be non-serious in columns 1-3, but are more likely to be non-serious (by quitting in the middle of the exam) in column 4 suggesting that women “blow off” the exam in different ways than men. As might be expected, being anxious or ambitious is associated with being less likely to be non-serious, while being undisciplined, i.e., having a pattern of skipping class or arriving late, is associated with being non-serious.

One might speculate that students who are over-worked and over-tested, especially with high-stakes exams, have test fatigue and passively resist taking yet another test, and therefore are more likely to not take PISA seriously. There is some evidence in favor of this. First, countries with high-stakes exams do seem to make students work harder. The data reveals that on average students spend 1.3 hours more per week in class and 3.1 hour more on out-of-school learning in all subjects in high-stakes countries relative to low-stakes ones. Working harder seems to be associated with not taking PISA seriously. In column 1, spending more time on studies out of school is significant for all countries together, but the effect seems to be coming from high stakes countries. Time spent in school is positive but not significant for all countries together, but is significant for low stake countries.⁴⁵ Having more tests (standardized or teacher-developed) does seem to correlate positively with being non-serious overall, though the coefficients are not significant. This might be because the effects differ in high stake and low stake countries. Having more standardized tests raises the likelihood of being non-serious in high-stakes countries (column 2) but does the opposite in low-stakes ones (column 3).

When teacher-developed tests are being given, raising the stakes seems to make students more likely to be serious, not less, suggesting that such testing may be less likely to result in test fatigue. Students from better schools, as reflected in the log of the school science score, are also less likely to be non-serious in low stakes countries, but more likely to be non-serious in high stakes countries. This makes sense if better schools push students more in high stakes countries resulting in fatigue. In Table 9 and Table 10, we present correlates of each non-seriousness criterion for cutoff level of 10% (as defined in Section 3) and 6%, respectively. The results are consistent across different cutoff levels.

Our results here should be seen as preliminary as there is no causation implied, merely correlation. The patterns described above are suggestive and might be worth exploring in future work.

Table 9 Factors related to being non-serious for each criterion (for cutoff level of 10%)

Table 10 Factors related to being non-serious for each criterion (for cutoff level of 6%)

A.1.3 Which Questions are Not Taken Seriously?

We define a non serious question as those that were not reached, for which there was no response, were missing, or on which too little time was spent. Non-reached and no-response items were looked at by the student who then chose not to answer the item despite having time left. Had he taken the exam seriously, he would have answered to the best of his ability. The student did not even look at missing items. But he had time left. In general, students have ample time to do the exam. Not even bothering to even read the question is again an indication of non seriousness. One might argue that no-response items, i.e., those that were skipped in the middle of the exam, should be treated differently as this was a computer based exam and students could not go back. Assuming they knew this, their choosing to skip again indicates the question is not taken seriously. Questions on which too little time was spent (as explained in criterion 4 for defining non serious students) are those where the response time is below a threshold which is country specific and for which the proportion correct is lower than that for normal time items for the same person. This is to prevent us from mistakenly labeling a question as non serious when in fact the student knew the answer immediately and so spent little time on it.

We explore the effects of question characteristics on the probability of a question being skipped, i.e., being not-reached or no-response. We also do the same for the probability of too little time being spent on a question. In both cases we run a linear probability model with individual fixed effects as well as question characteristics. Figure 7 shows the predicted probability of skipping a question and the predicted probability of spending too little time on a question for each cluster as a function of the difficulty of the question.⁴⁶ In all clusters, as the difficulty of the question increases, the probability of skipping increases though there is a slight decrease as questions become very difficult(top panel). In the bottom panel, we see that the probability of spending too little time is roughly flat: first increasing, then decreasing and finally increasing again. Students seem to try to answer if the question is easy but as it gets difficult, they seem to give up. There are also differences between clusters. Consistent with the “fatigue” hypothesis, questions are more likely to be taken non seriously in the second and fourth clusters.

Fig. 7

Pr(skip) and Pr(spend too little time) w.r.t. cluster and difficulty. In the figure, lowess-smoothed lines are presented. Predicted probabilities are obtained from a linear probability model with individual fixed effects as well as question characteristics such as cluster, sequence, difficulty and the type of the question. For each cluster, the predicted probability at each level of question difficulty takes the mean value of the predicted probability at that level of difficulty. In the figure, lowess-smoothed lines are presented

Table 11 Factors affecting Pr(Skip) and Pr(Spend too little time) (Individual Characteristics)

In Fig. 8, we explore whether question type affects the probability of skipping or spending too little time as a function of question order. For all questions, the probability of skipping rises with order, or sequence, in a cluster and jumps down at the beginning of the new cluster and more so after the break, which is consistent with “fatigue”. The graph of complex multiple choice questions for the probability of skipping lies between the open response and simple multiple choice questions. This makes sense as it is easy to guess an answer for simple multiple choice questions so that they are less likely to be skipped.

Fig. 8

Pr(skip) and Pr(spend too little time) w.r.t. sequence and the type of the question. Predicted probabilities are obtained from a linear probability model with individual fixed effects as well as question characteristics such as cluster, sequence, difficulty, and the type of the question. For each question type, the mean value of the predicted probability at each order is presented

Non-serious behavior in terms of spending too little time weakly falls with the order within a cluster for all question types. However, there is a large jump up at the beginning of the second and fourth clusters. The above pattern suggests that for open response questions at least, as the exam proceeds, students substitute towards skipping with a reset at the end of each cluster. Hence we see a fall with sequence within a cluster and a jump up in each new cluster. While skipping is more likely for open response questions, spending too little time is less likely for such questions relative to other question types.

In order to understand the effects of individual characteristics on the probability of being skipped or spending too little time, we run individual characteristics on estimated individual fixed effects from our linear probability model, see Table 11. The results are in line with those of Table 8.

So far we ran choice regressions as if they were independent. However, the appropriate model is a multinomial choice one as the student has three mutually exclusive and exhaustive options for each question: skip, answer with too little time or answer with normal time. We used the linear probability model as it allowed us to incorporate individual fixed effects, which we could not do with logit. With logit, we can control for individual characteristics, but as we are unlikely to have information on all possible characteristics, we might have omitted variable bias.

Table 12 presents the results of a logit regression where the baseline choice is spending normal time answering the item. In the regression, we control for the question characteristics and the individual characteristics used in the previous tables. The first and second columns show the factors affecting the probability of skipping and the probability of spending too little time, respectively. The position within a cluster is positively correlated with the probability of skipping and negatively correlated with the probability of spending too little time, consistent with students switching from spending too little time to skipping as the exam progresses. If a question is in the second, third or fourth cluster relative to being in the first cluster, it is more likely to be skipped and this likelihood is much higher in the second and fourth clusters as they are the last clusters in each science session. Open response and complex multiple choice questions move students towards skipping and away from spending too little time. However, as the difficulty of the questions increase, the students become more likely to skip and spend too little time. The coefficients on individual characteristics are roughly in line with those in Table 8. The math score of the student is negatively correlated with the probability of skipping and the probability of spending too little time. Female students are less likely to skip or spend too little time. Ambitious students are less likely to skip. Consistent with our previous findings, students from richer countries are more likely to skip and spent too little time, though the shape is that of an inverted U with a turning point at about $43,000 for per capita GDP. We control for standardized test frequencies and teacher developed test frequencies to investigate whether there is any evidence that students are fed up with testing, and as a result do not take them seriously. We find that as the frequency of the standardized tests increases, students likelihood of skipping and spending too little time significantly increases which is consistent with the “fatigue” effect. However, the teacher-developed tests do the exact opposite. This suggests that students view them very differently.

Table 12 Factors affecting Pr(Skip) and Pr(Spend too little time) (Logit results)

A.2 Fraction of Non-serious items Across Subjects

Table 13 shows the fraction of no-response items and the fraction of non-reached items for the subject of science, reading and math. The fraction of no response items for the reading and math tests are a bit higher on average than science. Moreover, the fraction of no-response and non-reached items are highly correlated across subjects. For example, the correlation between the fraction of no-response items for science and for reading is 0.98, showing that non seriousness is common across subjects of the test as might be expected.

Table 13 Fraction of non-serious items across subjects

Table 14 Time per science cluster (minutes)

A.3 Time Spent, Accuracy and Position

Table 14 shows time per science cluster across positions for serious and non-serious students. Note that time spent on the cluster falls with the position of the cluster and then jumps back up after the break at the end of cluster 2 and this is more so for non-serious students. There is substantial heterogeneity between non-serious students according to the criterion used. Students with no-response or too-little-time items, not surprisingly, spend less time per cluster than serious students regardless of cluster position. However, the opposite holds for those with non-reached or missing items but only for the first and third clusters. For the second and fourth clusters their time spent is 30-40% less than that of serious students. It is also worth noting that for these students, time is still not a constraint: on average they have more than 15 minutes left. This suggests that “fatigue” sets in faster for non-serious students.

The upper part of Table 15 shows proportion correct for all items (not just answered ones) across positions. Serious students have higher proportion correct than each category of non-serious students. Accuracy falls in the second cluster compared to the first one, and this is more so for non-serious students, reminiscent of the patterns for time spent. However, non-serious students will have a lower proportion correct on all items by definition as they skip many items. If we want to know what their accuracy is we should divide by the number of answered questions as done in the lower part of Table 15. The numbers show that even with this correction non-serious students have lower accuracy than serious ones. In addition, the degree to which accuracy falls across clusters is now similar (around 2%) for both serious and non-serious students. This is consistent with non-serious students’ performance experiencing a substantial drop in the second cluster primarily because they skip more items there.

Table 15 Proportion correct in science clusters

A.4 Variables Used in Imputation

PISA data has a rich array of information from the student and school questionnaires in the survey. In the imputation we use variables constructed from these surveys by PISA. We choose the variables that seem relevant. A list of the variables used is contained in Table 16. Binary variables are clearly identified. All others are continuous indices. Details of these are available in the PISA technical report, OECD (2015b), Chapter 16. The imputation also uses the individual’s scores for all other items and other students’ scores for all items as in the standard MICE imputations. We also include country fixed effect in the imputations.

Table 16 Variables used in imputation

A.5 Decomposition for Partially Serious Students

We call fully serious students those who neither skip items nor spend too little time on any item. These fully serious students, together with what we call partially-serious students, make up what we have termed serious students. For fully serious students, the number correct will be the same before and after imputation by definition. The increase in fraction correct for serious students (Y_s) therefore only comes from imputations for partially serious students who did skip a few items or spent too little time on a small enough number of items so that they were not classified as non-serious. There are PS partially serious students. Next we will decompose Y_s into its component parts.

$$ \begin{array}{@{}rcl@{}} Y_{s} &=&\frac{\sum \limits_{i\in S}I_{i}}{\sum\limits_{i\in S\cup NS}T_{i}} \\ &=&\frac{\sum\limits_{i\in PS}\left( I_{i}\right) }{\sum\limits_{i\in PS}NI_{i}}\frac{\sum\limits_{i\in PS}NI_{i}}{\sum\limits_{i\in PS}T_{i}} \frac{\sum\limits_{i\in PS}T_{i}}{\sum\limits_{i\in S\cup NS}T_{i}} \\ &=&A_{ps}E_{ps}P_{ps} \end{array} $$

A_ps is the increase in the fraction correct for non-serious items among partially serious students. E_ps is the fraction of non-serious items among all items for partially serious students, which measures the degree of non-seriousness. P_ps approximately measures the proportion of partially serious students in a country as partially serious students on average have the same number of total items as other students. The values of Y_ps, A_ps, E_ps and P_ps for each country are provided in Table 17.

Table 17 Decomposed factors for partially serious students

Similar to the decomposition for non-serious students, we divide both sides by the geometric mean and get

$$ y_{ps}=\frac{Y_{ps}}{\bar{Y}_{ps}}=\left( \frac{A_{ps}}{\bar{A}_{ps}}\right) \left( \frac{E_{ps}}{\bar{E}_{ps}}\right) \left( \frac{P_{ps}}{\bar{P}_{ps}} \right) =a_{ps}e_{ps}p_{ps} $$

(7)

Take the logarithm on both sides of Eq. 7 gives:

$$ \ln (y_{ps})=\ln a_{ps}+\ln e_{ps}+\ln p_{ps} $$

(8)

Next we run the regression of $\ln a_{ps}$, $\ln e_{ps}$, $\ln p_{ps}$ separately on $\ln y_{ps}$, that is,

$$ \begin{array}{@{}rcl@{}} \ln a_{ps} &=&\alpha_{2}\ln y_{ps}+\epsilon_{a} \\ \ln e_{ps} &=&{\upbeta}_{2}\ln y_{ps}+\epsilon_{d} \\ \ln p_{ps} &=&\gamma_{2}\ln y_{ps}+\epsilon_{p.} \end{array} $$

Let the OLS estimates be denoted by $\hat {\alpha }_{2},\hat {\upbeta }_{2},\hat { \gamma }_{2}$. Similarly we can show that $\hat {\alpha _{2}}+\hat {{\upbeta }_{2}}+ \hat {\gamma _{2}}=1\$and the coefficients $\hat {\alpha _{2}}$, $\hat {{\upbeta }_{2}}$, $\hat {\gamma _{2}}$ measure the contribution of partially serious students’ ability, extent of non-seriousness and proportion to a country’s increase in fraction correct. Figure 9 plots the scatter plot and regression lines above for partially serious students.

Fig. 9

y_ps Versus its Components for Partially Serious Students