Introduction

The ability to appropriately deal with modern information and communication technology (ICT)—often referred to as ICT literacy—has become increasingly important in everyday life (ETS [Educational Testing Service] 2002) and is assumed to be key competence in the 21st century for successful participation in society (Binkley et al. 2012; European Commission 2008; Fraillon et al. 2013). Lennon et al. (2003) defined ICT literacy as “the interest, attitude, and ability of individuals to appropriately use digital technology and communication tools to access, manage, integrate, and evaluate information, construct new knowledge, and communicate with others in order to participate effectively in society.” (p. 8).

Recently, several ability tests have been developed that objectively measure ICT competence in simulated environments, for example assessments in ICILS 2013 (Fraillon et al. 2014) or the CavE-ICT Test (Engelhardt et al. 2017). Results in ICILS revealed a considerable variation in ICT literacy between and within participating countries and showed that still a substantial percentage of adolescents had only below-average basic ICT literacy (Fraillon et al. 2014).

The Programme for International Student Assessment (PISA)—an influential large-scale assessment carried out by the Organisation for Economic Co-operation and Development (OECD 2016)—included questions on ICT availability, ICT familiarity, and ICT use in the optional ICT Familiarity Questionnaire since the first cycle in 2000. As one major change in 2015 the PISA 2015 main study moved from paper-and-pencil administrations to computer-based administration. This change strengthened the meaning of ICT literacy in the assessment context.

In this study, we present first empirical evidence on the dimensional structure of a new cognitive-motivational aspect of ICT literacy—ICT Engagement—that was newly introduced to PISA 2015. Further, as another source for the validity of test scores interpretations in these new scales we will report gender differences and investigate the relationships in the nomological network with ICT usage scales and performance in PISA 2015 literacy tests in mathematics, reading, and science. The analyses are based on data from Switzerland and Germany.

Theoretical background

Motives for ICT usage

ICT literacy is typically reflected by the self-reported frequency and diversity of typical ICT use at school and outside of school. Results from previous studies indicate that young people use ICT more often at home than at school (Zhong 2011). Adolescents use ICT for various activities outside of school ranging from playing computer games to preparing presentations for school and using social networks. In particular, they use ICT more often for entertainment and social interaction than for information and learning–related purposes (Fraillon et al. 2014; OECD 2010).

In the literature several models trying to explain interindividual differences in ICT usage were developed, among them the influential model of media attendance (MMA), a social-cognitive theory of internet uses and gratifications (LaRose and Eastin 2004; LaRose et al. 2001). Senkbeil and colleagues recently adapted the MMA model and defined six motives for self-regulated ICT activities that are hierarchically structured. On the highest level, the authors distinguished between hedonic, instrumental, and social interaction motives (Senkbeil and Ihme 2017; Senkbeil et al. 2016). On the second level, two sub-factors for hedonic motivation “Entertainment” and “Escapism” were postulated. For instrumental motivation the authors differentiated “Information seeking” from “Learn and work”. Finally, for social interaction motivation two sub-factors “Social exchange” and “Self-presentation” were proposed. Senkbeil and colleagues could establish corresponding measurement models and provided further empirical evidence for the validity of test score interpretations (Senkbeil 2017; Senkbeil and Ihme 2017). In particular, Senkbeil (2017) analyzed data from the ICILS study 2013 and reported small positive significant correlations between the two subscales of “Instrumental motivation” (“Information seeking” and “Learn and work”) with ICT skills (r = .12 and r = .14, respectively). Moreover, in this study the author found a small negative correlation between the two subscales of “Social interaction motivation” (“Social exchange” and “Self-presentation”) and ICT literacy (r = − .21 and r = − .14, respectively) (Senkbeil 2017). In a related publication with two smaller convenience samples Senkbeil and Ihme (2017) found even higher correlations between instrumental motivation and ICT skills (r = .23). Interestingly in the latter publication, the scores in “hedonistic motivation” and “social interaction motivation” were also negatively correlated with ICT literacy (r = − .14 and r = − .15, respectively), however they were not significant. The authors interpret these results as a hint that “not all ICT activities, in particular hedonic and social interaction activities, enhance ICT knowledge and skills” (p. 155).

Relations between computer usage and academic performance

The relationship between academic performance and computer use (mostly at school) was under the scope of the investigation for a long time. In the literature inconsistent results were reported. Whereas some authors found partly positive relationships (Weaver 2000; Weller 1996), Wenglinsky (1998) reported that students who spent more time on computers in school actually performed slightly worse than those who spent less time on them. Similarly, according to Papanastasiou (2002) in the Third International Mathematics and Science Study (TIMSS) computer use was also negatively associated with high student achievement in some countries. Papanastasiou et al. (2003) explored this phenomenon in more detail for the prediction of science performance based on data from PISA 2000 study and found that “it is not computer use itself that has a positive or negative effect on the science achievement of students, but the way in which computers are used.” (p. 325; for similar results for Mathematics see Papanastasiou and Ferdig 2006). The ICILS 2013 study also found a negative correlation between ICT use at school and ICT competence in some countries, including Germany (Bos et al. 2014).

In a recent study, Zhang et al. (2016) investigated how computer familiarity (reflected by computer access and use measures) is related to the students’ achievement in NAEP (National Assessment of Educational Progress) assessments for Writing, Mathematics, and Technology and Engineering Literacy. The authors found positive relationships between home computer access and student performance in all three assessments. Further, they reported that the use of different types of specific mathematics-related computer programs was negatively related to students’ mathematics achievement. In conclusion, previous results seem to be inconsistent and suggest that, at least in some studies, the use of technology is related to academic performance, depending on how technology is used at home and at school, and on how trained the teachers are in using technology (Wenglinsky 1998).

ICT Engagement

As a relevant cognitive-motivational aspect of ICT literacy, a new construct ICT Engagement has recently been introduced by Zylka et al. (2015), which is theoretically based on self-determination theory (Deci and Ryan 2000). ICT Engagement is assumed to be “a crucial individual factor for developing and adapting ICT skills in a self-regulated way” that “facilitates learning and acquiring new knowledge and skills through the life span by using ICT in both formal and informal learning environments” (Goldhammer et al. 2017, p. 332).

In its original conceptualization of ICT Engagement Zylka et al. (2015) theoretically distinguished between three different facets: ICT-related interest, self-concept related to the use of ICT, and social exposure to ICT. The results of the confirmatory factor analyses in their study—based on a questionnaire with 48 items—confirmed the assumed factors, and suggested further to distinguish a positive and a negative self-concept on using ICT as well as to separate interest in computers and interest in mobile devices factor.

In the next step, this ICT Engagement questionnaire was completely revised, theoretically extended by the aspect of “Perceived autonomy related to ICT use”, and was introduced as a new part of the optional international ICT Familiarity questionnaire in PISA 2015 main study. The revised ICT Engagement questionnaire involves the following factors: ICT interest, Perceived ICT competence, Perceived autonomy related to ICT use, and ICT as a topic in social interaction (Goldhammer et al. 2017).

ICT interest represents a “content-specific motivational disposition” and describes “individuals’ long-term preference for dealing with topics, tasks, or activities related to ICT” (Goldhammer et al. 2017, p. 341). It is supposed to initiate self-regulated actions involving ICT. Moreover, Christoph et al. (2015) found that ICT interest mediated the relation between a positive ICT self-concept and ICT-related performance outcomes.

Perceived ICT competence refers to beliefs about “his or her own knowledge about ICT and about how to use ICT skills” (Goldhammer et al. 2017, p. 342) and is assumed to maintain the self-regulated activities involving ICT and thus to facilitate the development of ICT skills.

Perceived autonomy related to ICT use “reflects the individual’s perceived control and self-directedness in ICT-related activities” (p. 424). In particular, engaged ICT users are supposed to experience a feeling of control using ICT and to attribute ICT related success to their own abilities instead of external factors. These attribution patterns foster a positive self-concept and increase the probability of showing self-regulated ICT-related behavior in future. Perceived autonomy related to ICT use is—with regard to the self-determination theory—a crucial factor for life-long-learning in the ICT context, “since a strong and stable feeling of control supports individuals to continuously deal with ICT in a self-directed way” (Goldhammer et al. 2017, p. 343) and to keep up with the latest technological developments.

ICT as a topic in social interaction addresses “the extent to which students make ICT a subject of interpersonal communication and interaction” (Zylka et al. 2015, p. 151) and therefore represents the “connectedness or belongingness to others when dealing with ICT” (Goldhammer et al. 2017, p. 343). In contrast to the ICT interest, this aspect of ICT Engagement refers to social exchange with other persons in informal learning contexts in which ICT skills can be developed through discussions on ICT related topics. This theoretical framework is described in detail by Goldhammer et al. (2017).

Gender differences in ICT literacy

Previous research pointed out that—although few significant gender differences for ICT usage at school were found—the differences are much more pronounced for ICT usage at home (BECTA 2008). In particular, great body of literature has presented empirical evidence that boys report higher frequency of ICT use outside of school and greater experiences of using ICT at home than girls (BECTA 2008; Eurydice 2005; OECD 2007; Sanders 2005; Valentine et al. 2005). Girls tend to use ICT more for school work and online social networking, while boys use ICT more often for leisure purposes (in particular for playing computer or console games) (BECTA 2008).

With regard to attitudes, confidence, and self-efficacy most studies found that girls’ confidence is lower in comparison to boys (BECTA 2008; Colley and Comber 2003). Accordantly, females in general were found to report more negative ratings on their ICT-related self-concept than their male peers (Birol et al. 2009; Janneck et al. 2013; Sáinz & Eccles 2012).

With respect to ICT literacy, girls outperformed the boys in most countries in the ICILS 2013 (Fraillon et al. 2014; for detailed analysis see Gebhardt et al. 2019; Punter et al. 2017) and ICILS 2018 (Gerick et al. 2019). Interestingly, in ICILS 2013 no significant gender differences were found for ICT-related self-efficacy expectations with regard to basic ICT skills (e.g. creating a new document). In contrast, boys reported higher values for ICT-related self-efficacy expectations referring to advanced skills (e.g. erasing a computer virus or programming) (for similar results see also Eurydice 2005). Volman et al. (2005) also reported greater gender differences in subjective ICT attitudes than for objective ICT literacy. These results indicate that girls consistently underestimate their ICT literacy.

With regard to ICT Engagement, Christoph et al. (2015) investigated the relationships between several aspects of the ICT Engagement in its original conceptualization, computer skills, and computer knowledge in a mediation model. In this study, boys perceived themselves as more competent in computer activities and reported higher levels in computer self-concept, computer interest, and ICT-related social engagement. Furthermore, the authors found different correlation pattern for boys and girls, suggesting that compared to boys, the correlations between computer self-concept and computer interest as well as to ICT-related social engagement were lower for girls.

Research questions and hypotheses

In the present study, we pursue the following goals related to ICT engagement as assessed by the PISA 2015 ICT Familiarity questionnaire:

  1. 1)

    First, we present different sources of validity supporting the construct interpretation of test scores by

    • Investigating the dimensional structure of ICT Engagement for the Swiss sample

    • Investigating the relationship between ICT Engagement and different types of ICT usage in a SEM-based latent regression model

    • Investigating gender differences in ICT Engagement via multi-group models

    • Exploring correlations with performance in PISA 2015 literacy tests in mathematics, reading, and science

  2. 2)

    Second, we replicate the results obtained for the Swiss sample with the German sample.

Related to these goals we tested the following five hypotheses. Supporting theses hypotheses provides empirical evidence for the construct interpretation of test scores obtained from the ICT Engagement subscales.

H1: We aimed to empirically confirm the theoretically expected four-dimensional structure of the ICT Engagement scales using a SEM approach.

H2: We assumed that ICT usage outside of school is more strongly self-regulated than ICT usage at school that is likely initiated by teachers. Therefore, we expected that subscales of ICT Engagement would explain a higher amount of variance in ICT usage outside of school than in ICT usage at school.

H3: With respect to ICT usage outside of school, we further assumed that subscales of ICT Engagement would explain a higher amount of variance in ICT usage for entertainment than with school-or-learning-related ICT usage, due to the fact that ICT usage for entertainment is mainly self-chosen, whereas school-or-learning-related ICT usage is at least partly extrinsically motivated (e.g. by homework requirements). Furthermore, we explored differences in the explanation of different motives for ICT usage by different subscales of ICT Engagement.

H4: As outlined above in most previous studies girls reported lower values in ICT related attitudes and ICT related self-concept. Since ICT Engagement reflects cognitive-motivational aspects of ICT literacy, facilitating intrinsic ICT related activities, we expected that girls would score lower on all ICT Engagement scales.

H5: Finally, in order to show that the construct interpretations are not only valid for one selected country, we aimed to replicate these results for the German sample in PISA 2015. We have selected Germany for this comparison, due to the great cultural proximity between the two countries and partly very similar language (at least in some parts of Switzerland). Thus, a replication of the results allows to generalize the arguments for the validity of the construct interpretation in the revised ICT Engagement subscales to another educational system. The obtained results inform about construct equivalence across countries and thereby further justify cross-country comparisons PISA is aiming at.

Methods

Sample

The following data analyses are based on pubic data from PISA 2015 main study from Switzerland and Germany. In total in Switzerland 5860 15-year-old students were tested; 2807 were female (47.9%). In Germany 6504 students participated with 3197 being female (49.2%). The amount of missing data for the ICT Engagement scale differed between items and countries: In Swiss data it ranged from 8.8 to 12.2%, whereas for Germany it varied between 18.1 and 21.3%. Closer inspection of the data for the German sample revealed that 4593 persons (70.6%) did not have any missing value and 1105 (approx. 17%) did not provide any valid answer on ICT Engagement scales.

As described in the documentation of PISA 2015 for Germany (Reiss et al. 2016) the main reason for the relatively high percentage of missing values was that answering the optional part of the student questionnaire was not mandatory in most German federal states. In particular, in federal states with voluntary participation, parental declarations of consent were necessary. Due to these different regulations, the answer rates for the questionnaire data varied between 82 and 97% in states with mandatory participation and ranged from 62 to 91% in states with voluntary participation. On average, approx. 85% of questionnaire data is available for Germany. Thus, the reported missing rates for the ICT Engagement seem to be comparable to the reported missing rates in the PISA documentation.

Instruments

ICT usage and ICT Engagement were assessed in a questionnaire via Likert rating scales. With regard to ICT usage scales the students were required to indicate how often they use digital devices for various activities outside of school and at school, respectively, on a 5-point-rating-scale (1-never or hardly ever; 2-once or twice a month; 3-once or twice a week; 4-almost every day; 5-every day). Digital devices were defined as “digital devices, including desktop computers, portable laptops, notebooks, smartphones, tablet computers, cell phones without internet access, game consoles, and internet-connected television” in the introduction part of the questionnaire.

The instruction for the ICT Engagement scales was “Thinking about your experience with digital media and digital devices: to what extent do you disagree or agree with the following statements?” with four answer options (1-strongly disagree; 2-disagreee; 3-agree; 4-strongly agree). The complete list of items for ICT Engagement and ICT usage are presented in Tables 1 and 2. Descriptive statistics for the scales are presented in Table 3. The internal consistencies (Cronbach’s α) ranged from 0.61 to 0.89 and were sufficiently high with two exceptions for the subscales “Use for entrainment” (α = 0.61) and “Practical usage” (α = 0.64).

Table 1 Items for the assessment of ICT Engagement
Full size table
Table 2 Items for the assessment of ICT Usage outside of school and at school
Full size table
Table 3 Descriptive statistics for the scales for ICT Engagement and ICH Usage for the Swiss and German sample
Full size table

Statistical analyses

The measurement models of ICT Engagement and ICT usage were estimated via structural equation models in the software Mplus 8.3 (Muthén and Muthén 1998–2017a) using the weighted least squares mean and variance (WLSMV) estimator for categorical indicators. This procedure is recommended for the analysis of measurement models with ordinal data (Kline 2011) and uses pairwise present data (Asparouhov and Muthén 2010; Muthén and Muthén 1998–2017b). Further, a latent regression model was estimated in order to investigate how much variance of different motives for ICT usage can be explained by the ICT Engagement scales. For model fit evaluation traditional cut-off values for Root Mean Square Error of Approximation (RMSEA) and Confirmatory Fit Index (CFI) were applied. RMSEA values smaller than .08 and CFI greater than .90 suggest an acceptable model fit, whereas RMSEA values smaller than .05 and CFI greater than .95 indicate a well-fitting model (Hu and Bentler 1999).

For the comparison of girls’ and boys’ scores in ICT Engagement, multi-group models were estimated in Mplus (Muthén and Asparouhov 2002). Therefore, the measurement invariance assumption was tested. To make sure that the construct of interest is measured in both groups in the same way and that the latent means in both groups are comparable, scalar invariance level is required (i.e. factor loadings and intercept—or thresholds for categorical indicators—are kept equal for both groups) (Meredith 1993). For categorical indicators the online Mplus User’s Guide (Muthén and Muthén 1998–2017b) suggests evaluating only the configural-scalar comparison, by arguing that factor loading and thresholds for categorical indicators should only be freed or constrained in tandem. Therefore, only configural and scalar invariance were considered in the reported analyses. For the evaluation of model fit in the invariance analysis we considered the traditional Likelihood-Ratio Test, which is based on the Chi squared distribution. However, for large sample sizes the Likelihood-Ratio Test tends to flag statistical significance even, if the observed deviations are not practically meaningful. Therefore, we further considered the criterion of CFI difference (ΔCFI) suggested by Cheung and Rensvold (2009), which is less vulnerable to sample size. These authors suggest that “a value smaller than or equal to − 0.01 indicates that the null hypothesis of invariance should not be rejected.” (Cheung and Rensvold 2009 p. 251).

For the calculation of the correlations between ICT Engagement and performance in PISA 2015 reading, mathematics, and science tests we conducted the respective SEM analysis for five plausible values and averaged the results.

Results

In the next sections we present different sources of validity supporting the construct interpretation of test scores, referring to the validity framework suggested by American Educational Research Association (AERA), American Psychological Association (APA), & National Council on Measurement in Education (NCME) (2014).

Evidence based on internal structure

To investigate the dimensional structure of the PISA ICT Engagement scales (H1) we have estimated a four-dimensional measurement model by modelling each postulated aspect as a separate latent factor (see Fig. 1). The fit for this model was acceptable (χ2 = 7650.9; df = 179; CFI = 0.929; RMSEA = 0.087). Latent correlations between the four aspects ranged from 0.44 to 0.79, indicating that these four latent factors assessed distinct aspects of ICT Engagement. We observed a high latent correlation between ICT competence and perceived ICT autonomy. Therefore, we estimated an alternative, more parsimonious, model with ICT competence and perceived ICT autonomy building one latent factor. This model with three latent factors had a worse model fit than the measurement model with four factors (χ2 = 8890.2; df = 182; p < .001; CFI = .918; RMSEA = .093). Therefore, we used the measurement model with four latent factors in further analyses. Please note that this model included four theoretically meaningful residual correlations between items that referred to very similar contexts or wording (e.g. playing games or internet). Surprisingly, the residual correlation between two statements referring to the personal value of internet (F1_2 and F1_5) was negative.

Fig. 1
figure 1

Measurement model for ICT Engagement. The coefficients in front of the forward slash refer to values for the Swiss sample, whereas the values after the forward slash refer to the results for the German sample. The presented values are based on the scalar SEM with fixed factor loadings and thresholds (Model fit χ2 = 14,925.18; df = 413; p < .001; CFI = 0.933; RMSEA = 0.08)

Full size image

Evidence based on relations with other variables

Measurement model for ICT usage

In the second step we have estimated a measurement model for ICT usage postulating four different aspects of ICT usage outlined above (see Fig. 2). The model fit was good (χ2 = 5383.2; df = 258; CFI = 0.946; RMSEA = 0.06). Latent correlations between the four aspects ranged from 0.22 to 0.77, indicating that these four latent factors assessed distinct aspects of ICT usage. As expected, the correlations between ICT usage at school and School-or-learning-related use at home (r = .77 for Switzerland) and ICT Use for Entertainment at home and Practical usage at home (r = .68 for Switzerland) were particularly high.

Fig. 2
figure 2

Measurement model for ICT Usage. Values in front of the forward slash refer to values for the Swiss sample, whereas the values after the forward slash refer to the results for the German sample. The presented results are based on the scalar SEM with fixed factor loadings and thresholds (Model fit: χ2 = 10,850; df = 608; CFI = 0.938; RMSEA = 0.055)

Full size image

Although in this SEM many residual correlations were exploratory included, it is important to point out that all of these correlations were theoretically meaningful and were related to very similar contexts. For example, a residual correlation between the items U2_1 and U4_1 refers to usage of email at home versus using emails at school. The results indicate that ICT usage can be—at least partly—interpreted as a reflective construct that is measured by heterogeneous behavioral indicators.

Explaining variance in ICT usage

In Table 4 we present the latent correlations between ICT Engagement and ICT usage. In the next step, we estimated a latent regression SEM in which the subscales of ICT Engagement explain variance in different aspects of ICT usage (see Fig. 3). The latent regression SEM M3 (see Fig. 3) had an acceptable fit (χ2 = 18,257.7; df = 904; CFI = 0.905; RMSEA = 0.058). In terms of ICT usage outside of school, as expected ICT Engagement scales explained more variance in Use for Entertainment (74.1%, SE = 2.8%) and Practical usage (21.8%, SE = 1.3%) than for learning-related activities outside of school (10.8%, SE = 1%). In line with our hypotheses, ICT Engagement scales explained much smaller amount of variance in ICT usage at school (8.7%, SE = 0.9%).

Table 4 Latent correlations between ICT Engagement and ICT usage
Full size table
Fig. 3
figure 3

Structural part of the final latent regression model with ICT Engagement factors explaining variance in ICT Usage factors. The coefficients in front of the forward slash refer to values for the Swiss sample, whereas the values after the forward slash refer to the results for the German sample. For a greater clarity only significant regression coefficients (p < 0.001) are shown in the figure. Model fit for Switzerland: χ2 = 18,281; df = 904; CFI = 0.905; RMSEA = 0.058. Model fit for Germany: χ2 = 17,869; df = 904; CFI = 0.899; RMSEA = 0.058. n.s. stands for not significant

Full size image

Surprisingly, for almost all motives of ICT usage ICT as a topic in social interaction had the highest predictive values (with the exception of Practical usage). ICT interest and Perceived ICT autonomy related to ICT use were only predictive for leisure-related ICT usage (i.e. Entertainment and Practical usage), whereas Perceived ICT competence unexpectedly did not contribute significantly to explaining the variance in motives for ICT usage – when controlling for other aspects of ICT Engagement. One possible explanation for this finding might be the high latent correlation between Perceived ICT competence and Perceived ICT autonomy.

Gender differences in ICT Engagement

In Table 5 model fit for configural and scalar measurement invariance and results of the model comparison test are presented. Unfortunately, we had to reject the model with scalar invariance due to significant results in the Likelihood-Ratio Test and ΔCFI = 0.016.

Table 5 Testing measurement invariance between girls and boys for ICT Engagement scales for Switzerland
Full size table

When setting the latent mean in the male group to 0 as reference, latent means for female students were significantly lower in Perceived ICT competence (M = − 0.47; p < 0.001), Perceived ICT autonomy (M = − 0.58; p < 0.001), as well as 0ICT as a topic in social interaction (M = − 0.80; p < 0.001). In contrast, no significant differences between females and males were found for ICT interest (M = − 0.05; p = 0.07). These results should be interpreted with caution, since scalar invariance could not clearly be established.

Correlation between ICT Engagement, ICT usage, and performance in PISA literacy tests

In addition, we explored the correlations between ICT Engagement, ICT usage, and the performance in PISA literacy tests (see Table 6). In general, the correlations between the different aspects of ICT Engagement and performance in PISA literacy tests were rather low. As the two highest correlations we found a positive relationship between ICT autonomy and performance in Mathematics (r = .17) as well as an unexpected negative correlation between “ICT as a topic in social interaction” and reading performance (r = − .20).

Table 6 Correlations between ICT Engagement scales, ICT Usage, and competencies in PISA for Switzerland and Germany
Full size table

The correlations between ICT usage and the performance in PISA literacy tests were also low. “Practical usage” correlated significantly with performance in Mathematics and Science (r = .14 and r = .13, respectively). Moreover, we found small but significant negative correlations between “ICT use at school” and “School-or-learning-related use at home” and all three PISA 2015 literacy tests, ranging between r = − .13 and r = − .21.

Replication of the results for the German sample

As outlined above, one important aim was to replicate the results using data from the German PISA 2015 study. For greater clarity, the results for Switzerland and Germany are presented in all tables and figures simultaneously. In the first step, we have tested measurement invariance between the Swiss and German sample for the empirical structure of ICT Engagement. The results revealed a significant Likelihood-Ratio Test (see Table 7). As outlined above for large sample sizes the Likelihood-Ratio Test tends to flag statistical significance, even if the observed deviations are not practically meaningful. Therefore, we further consulted the ΔCFI criterion, which is not affected by sample size. Since the difference in CFI between the configural and scalar model was only 0.002, the application of ΔCFI criterion indicated that scalar invariance between both countries could be established. In Fig. 1 the coefficients for the Swiss and German sample based on the scalar model are presented. Most correlations between the latent factors differed only slightly between the countries and can be interpreted as comparable. One exception was the correlation between Perceived ICT competence and Perceived ICT autonomy, which was somewhat smaller in the Swiss (r = .70) than in the German sample (r = .84).

Table 7 Testing measurement invariance between Switzerland and Germany for ICT Engagement scales
Full size table

Measurement model for ICT usage

In the next step, we tested the measurement invariance between Switzerland and Germany for ICT usage. Although the results revealed a significant difference in the Likelihood-Ratio Test (see Table 8), we again relied our decision on the ΔCFI criterion. Since ΔCFI = 0.004 was smaller than 0.01, we assumed that scalar invariance between both countries could be established for the ICT usage scales. In Fig. 2 the coefficients for the Swiss and German sample based on the scalar model are presented.

Table 8 Testing measurement invariance between Switzerland and Germany for ICT Usage scales
Full size table

Explaining variance in ICT Use

As can be seen in Fig. 3 the percentage of explained variance did not differ much between the two countries. The largest difference was found for the factor “Use for Entertainment” with an approximately 11% higher amount of explained variance in the Swiss (R2= .74) than in the German sample (R2= .63).

Gender differences in ICT Engagement

Similarly to analyses for the Swiss sample, scalar invariance could not be established for male and female students due to a significant Likelihood-Ratio Test and ΔCFI = 0.022 (see Table 9). In the scalar model we found significantly lower scores for the female students in Perceived ICT competence (M = − 0.65; p < 0.001), Perceived ICT autonomy (M = − 0.77; p < 0.001), as well as ICT as a topic in social interaction (M = − 0.94; p < 0.001). In contrast to Switzerland, we found also significant differences in interest (M = − 0.19; p < 0.001) with male students showing higher values. Although these results should be interpreted with caution, the closer inspection of the results revealed a tendency to slightly stronger gender differences for the German sample in comparison to Switzerland.

Table 9 Testing measurement invariance between girls and boys for ICT Engagement scales for Germany
Full size table

Correlation between ICT Engagement, ICT usage, and performance in PISA literacy tests

As can be seen in Table 5 the correlation patterns were very similar between the two countries. Some minor discrepancies were observed for the correlation between “ICT as a topic in social interaction” and reading with a larger negative correlation (r = − .20) in Switzerland than for the German sample (r = − .12). Moreover, the negative correlation between “ICT usage at school” and performance in Mathematics was larger for the German (r = − .23) than for the Swiss sample (r = − .13).

Discussion

The goal of this study was to provide first evidence for the construct interpretation of the ICT Engagement scale, which was used in PISA 2015 for the first time. Specifically, we investigated the internal structure by dimensional analyses and investigated the relation of ICT engagement aspects to other variables. Overall, the obtained results support the construct interpretation of the four ICT engagement subscales.

With respect to the first hypothesis H1, we could confirm the theoretically suggested structure for the scores in the ICT Engagement questionnaire, although some empirically driven and theoretically meaningful residual correlations were included in the model. In line with the second hypothesis H2, ICT engagement scales explained more variance in ICT usage outside of school than in ICT usage at school. Apparently, ICT Engagement scales explained the highest amount of variance in ICT Use for Entertainment, followed by Practical use. Further in line with our expectations, only little variance was explained in learning-related use at home. Thus, hypothesis H3 could be confirmed.

One could have expected that the perceived ICT autonomy shows higher correlations with “Use at school” or “Learning-related ICT usage at home” than the other ICT Engagement scales, because it is supposed to maintain the continuous ICT related activities. However, the empirical analyses do not support this assumption. Surprisingly, for almost all motives of ICT usage ICT as a topic in social interaction had the highest predictive values (with the exception of Practical usage). In particular, the correlation between ICT as a topic in social interaction and Use for Entertainment is remarkably high (r = .78 for Switzerland; r = .75 for Germany). The result that ICT as a topic in social interaction was predictive for learning-related ICT activities outside of school can be partly explained by the fact that the ICT usage scale measures collaborative learning activities as one relevant aspect.

In terms of gender differences, we could partly confirm H4. In agreement with previous literature, we found significantly lower values for girls in all ICT Engagement scales—except ICT interest—for both countries, whereas gender differences in ICT interest were significantly lower for girls in the German sample only.

Furthermore, in exploratory analyses for both Germany and Switzerland we found low to moderate correlations with the performance in PISA literacy tests. As a rather unexpected finding in our analyses we found a negative correlation between “ICT as a topic in social interaction” and reading performance in PISA 2015 that will be discussed below. Finally apart from a few exceptions that were shortly outlined above, we were able to replicate most results for the second independent German sample, in particular by establishing measurement invariance for the measurement models for ICT Engagements and ICT usage. Therefore, H5 could be confirmed.

In summary, these results show that the newly introduced ICT Engagement scales measure reliably. Further, (a) the empirical modeling of the theoretically derived four-dimensional-factor structure, (b) the relations to ICT usage in the latent regression model, and (c) the reported gender differences provide evidence for the validity of test scores interpretations of the ICT Engagement scale with four subscales.

An interesting finding was that although Practical Use and Use for Entertainment were highly correlated, there were substantial differences in explained variance of these two constructs (22% vs. 74%). To prove whether the results would change if only one of these two criteria is taken into account, we have estimated additional models, where either the latent factor for Entertainment or the latent factor for Practical use was excluded from the measurement model for ICT usage. The results revealed that the amount of explained variance did not substantially change in these reduced models.

In terms of ICT usage we found for both countries small but substantial negative correlations between ICT use at school and school-or-learning-related use at home and performance in PISA 2015 literacy tests. This result is consistent with previously reported findings from other large scale assessments like PISA 2000 (Papanastasiou et al. 2003), TIMSS 1995 (Papanastasiou 2002) and NAEP (Zhang et al. 2016). In accordance with the argumentation line proposed by Zhang et al. (2016) we assume that more frequent use of computers at school is likely associated with remedial purpose for students with lower school performance.

Another remarkable finding were consistently negative correlations between “ICT as a topic in social interaction” and reading performance in PISA 2015 both in the German and in the Swiss sample. Since this aspect of ICT Engagement refers to social exchange with other persons in informal learning contexts, it is possible that students with lower reading performance use ICT more often for social exchange for school-related issues e.g. in order to get help from classmates. Another explanation might be that students with lower reading skills solve occurring ICT related problems less often by searching for written information but rely more often on social interaction instead. However, these correlational results should not be overstated, because the correlations between ICT Engagement and achievement in PISA reading tests were only small.

The postulated measurement model for ICT usage motives is similar to the motivational structure proposed by Senkbeil (2017). However, there are several noticeable differences between these models. First, we consider both ICT usage outside of school and ICT usage at school, whereas Senkbeil et al. only addressed motives for ICT usage outside of school. Second in contrast to Senkbeil et al., we did not assume a hierarchical structure and estimated a simpler measurement model with four distinct but correlated latent factors. Third, in our measurement model “school-or-learning-related motives” was supposed to be one of the three main motives for ICT usage outside of school, whereas in Senkbeil’s model the corresponding sub-factor “Learn and work” was nested under the higher-order factor “Instrumental motivation”. Fourth, the sub-factors “Escapism” and “Self-presentation” were not addressed in PISA 2015 framework. Finally, since we did not specify a distinct latent factor for “social exchange motivation”, items referring to social exchange were assigned to different factors, which explains some of residual correlations (e.g. between items U1_3 and U1_4; U1_4 and U1_5; U1_4 and U3_5).

This study has some limitations. First, Cronbach’s alpha values were rather small for two ICT usage subscales “Use for entertainment” and “Practical usage”. One plausible interpretation for these only moderate reliabilities might be that the ICT usage scale does not represent a trait-like disposition, but an index variable consisting of different, not necessarily homogeneous items that describe different motives for ICT usage. That might partly lead to lower inter-item correlation, which in turn can result in sub-optimal Cronbach’s alpha values. This explanation is consistent with the finding that the scale for ICT usage at school is more reliable than the subscales for ICT usage outside of school. It is possible that activities for ICT usage at School are rather limited and more consistent among students. In contrast, it is likely that there is much more room for action in the ICT usage outside of school and that certain activities are carried out to varying degrees across students, which statistically may lead to lower inter-item correlations between different motives. Note that while low internal consistencies certainly impair the application of these scales for individual diagnostic purposes, they are explicitly considered in SEM analyses and are less problematic when these scales are used for descriptions of relationships on the population level.

Second, the results of the gender comparison should be interpreted with caution, because the scalar invariance assumption was not empirically supported by the observed data. However, the fact itself that measurement invariance was explicitly tested can be seen as strength of this study. Third, this cross-sectional study does not allow any causal statements, because all scales were assessed at one measurement point.

An important strength of this paper is the replication of the analyses for Germany having the similar testing language (at least for one part of Switzerland). For future PISA cycles, it would be worthwhile to examine the relationship between ICT Engagement and reading literacy in more detail by investigating the latent correlations separately for linear reading versus hypertext reading that requires navigation across different websites. In particular, one would expect higher relationships between ICT Engagement and hypertext reading than for linear texts.

Further, it might be interesting to study the relationship between ICT Engagement and objective assessments of ICT literacy as further evidence for the validity of test score interpretations in the ICT Engagement subscales. Unfortunately, in PISA 2015 main study no achievement data on actual ICT competence is available and as far as we know it is also not planned to assess ICT competencies via assessment tests in the next PISA cycles.

Conclusion

In this work, we investigated the internal structure of ICT Engagement—as one relevant motivational aspect of ICT literacy. Furthermore, we investigated the relationships of ICT Engagement in the nomological network with ICT usage motives and gender. As a main result, we could confirm the four-dimensional structure of ICT Engagement in the public PISA 2015 data set using a SEM approach. Further, we presented different sources of validity supporting the construct interpretation of test scores in the ICT Engagement scale based on two independent, but culturally comparable, samples from Switzerland and Germany. We could confirm that ICT Engagement scales explained more variance in ICT usage outside of school than in ICT usage at school. Moreover, we found significantly lower values for girls in all ICT Engagement scales—except ICT interest—for both countries, whereas gender differences in ICT interest were significantly lower for girls in the German sample only. Overall, the obtained results support the construct interpretation of the four ICT Engagement subscales.