Next Article in Journal
Challenges When Evaluating Cognitive Bias Modification Interventions for Substance Use Disorder
Previous Article in Journal
Test@Work Texts: Mobile Phone Messaging to Increase Awareness of HIV and HIV Testing in UK Construction Employees during the COVID-19 Pandemic
 
 
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

The Social Support Networks Scale (SSNS) for Family Caregivers of Children with Cancer: A Psychometric Evaluation

by
Filiberto Toledano-Toledano
1,*,
José Moral de la Rubia
2,
René Reyes Frometa
3,
Fabiola González Betanzos
3,
Laura Villavicencio Guzmán
4 and
Marcela Salazar García
4
1
Evidence-Based Medicine Research Unit, Hospital Infantil de México Federico Gómez National Institute of Health, Mexico City 06720, Mexico
2
Facultad de Psicología, Subdirección de Posgrado, Universidad Autónoma de Nuevo León, Monterrey 64460, Mexico
3
Faculty of Psychology, Universidad Michoacana de San Nicolas de Hidalgo, Michoacán 58030, Mexico
4
Developmental Biology Research Laboratory, Hospital Infantil de México Federico Gómez National Institute of Health, Mexico City 06720, Mexico
*
Author to whom correspondence should be addressed.
Int. J. Environ. Res. Public Health 2020, 17(21), 7820; https://doi.org/10.3390/ijerph17217820
Submission received: 16 September 2020 / Revised: 20 October 2020 / Accepted: 20 October 2020 / Published: 26 October 2020

Abstract

:
Currently, information about the psychometric properties of the Social Support Networks Scale (SSNS) for family caregivers of children with cancer is not yet available; therefore, there is no empirical evidence of its validity and reliability to support its use in this population. The aim of this study is to determine a factorial model of the SSNS, estimate its internal consistency reliability, describe its distribution, and check its concurrent validity. A convenience sample of 633 family caregivers of children with cancer hospitalized in a National Institute of Health in Mexico City was collected. The SSNS, a sociodemographic variables questionnaire, and three instruments that evaluated family functioning, quality of life, and resilience were applied. The five-factor model had a poor data fit and lacked discriminant validity. The sample was divided. In a subsample of 316 participants, exploratory factor analysis suggested a four-factor model. When testing the four-factor model through confirmatory factor analysis, religious support was independent of family support, friend support, and lack of support. In the other subsample of 317 participants, the one-factor model for religious support had a good fit, and the correlated three-factor model, with the remaining factors, showed an acceptable fit. Reliability ranged from acceptable (Guttman’s λ2 = 0.72) to good (λ2 = 0.88). Socio-family support and its three factors were correlated with family functioning, resilience, and quality of life. Religious support was correlated with four factors of resilience and quality of life. A scale of socio-family support with three factors and an independent scale for religious support are defined from the SSNS, and they showed internal consistency and construct validity.

1. Introduction

1.1. Pediatric Cancer as a Health Problem

Data from the World Health Organization (WHO) indicate that cancer is the second leading cause of death worldwide, where approximately one in six deaths is due to cancer in the total population, that is, including all ages [1]. On the other hand, in the population of children and adolescents, cancer is the main cause of death worldwide [1]. Nevertheless, in high-income countries, 80% of children diagnosed with cancer are cured and survive cancer [1]. The most common types of cancer are leukemia, brain cancer, lymphoma, and tumors such as neuroblastoma and Wilms’ tumor [1]. Mexico’s official figures indicate that the group from 0–4 years had the highest cancer incidence rate, adolescents between 15–19 years had the highest mortality rate, and the survival rate in children and adolescents was 57.5% [2].
In this context, the research literature has indicated that pediatric chronic diseases represent a life-changing event presenting an important challenge for families, their health, and coping [3]. These diseases have physical, psychological, socioeconomic, and behavioral effects that decrease quality of life and family functioning and increase the burden of the caregiver [4], all of which are aspects that define and structure the profile of adversity and vulnerability of families of children with chronic diseases [5].

1.2. The Family Caregiver of a Child with Cancer

In particular, the family caregiver of a child with cancer experiences various feelings and profound changes in the family’s daily life; children with cancer need to spend substantial time in hospitals, leading to the emotional, physical, mental, and socioeconomic exhaustion of the caregiver [6]. In addition, parents experience life stress associated with constant hospital admissions and patient care [7].
In the research literature, a family caregiver is defined as a person who has a significant emotional bond with the patient. This caregiver is a family member who is part of the patient’s family life cycle, offers emotional-expressive and instrumental support, and provides comprehensive care and assistance during the chronic illness, acute illness, or disability of a child, adult, or elderly person [8].
Family caregivers can be affected physiologically and psychosocially by experiencing depression [8], anxiety [9], and posttraumatic stress [10]. All of these psychosocial variables have consequences that involve risk factors for developing psychopathology and caregiver burden [4].
However, despite the adversity, risk, and vulnerability experienced by families of children with cancer, there are protective factors that can reduce the negative impact of care. These factors can generate adaptive responses in families and in their psychosocial functioning to overcome the psychosocial consequences of caring for children with cancer. Among the protective factors are coping styles [3], resilience [11,12], and, most critically, perceived social support, that is, social support networks through family, friends, neighbors, religion, and governmental and nongovernmental institutions [13].

1.3. Effect of Social Support on Family Caregivers of Children with Cancer

Social support refers to a social network’s provision of psychological and material resources intended to benefit an individual’s ability to cope with stress. It is often differentiated in terms of three types of resources: instrumental, informational, and emotional [14]. The type of information given leads persons to think that they are esteemed, loved, and cared. This support is usually provided in a situation of illness or during a stressful event and protects people in crisis from a variety of psychopathological states [15]. It is usually defined as the existence or availability of people on whom we can rely, and who let us know that they care about, value, and love us [16].
In this regard, family, friends, and neighbors may bolster the social support perceived by family caregivers, influencing their daily life through cohesion, open communication, and creating a positive environment to care for the child with cancer [17].
With respect to the theoretical links between social support, health, and well-being, empirical evidence shows that being able to rely on a social support network in times of adversity contributes to overall health; on the other hand, when people perceive a lack of support or a diminishing social support network, they can become emotionally unstable and pessimistic [18]. In particular, the lack of social support in contexts of illness can affect the physical and mental health of family caregivers and have emotional and behavioral consequences [19,20,21,22,23,24].

1.4. Instruments for Measuring Social Support

The international literature has reported a set of measurement instruments that evaluate social support in various contexts, populations, and cultures. Most of them have been subjected to validation processes and cross-cultural adaptation. To the best of our knowledge based on bibliographic reviews, none have been validated and adapted in the context of Mexican family caregivers of children with cancer; therefore, it is not possible to adequately extrapolate these measurement instruments to this population because there is no empirical evidence of its validity and reliability to support its use in this population.
Several studies have confirmed that the psychometric properties of instruments measuring social support are consistent over time. One study in Spain evaluated the validity and reliability of the Duke-UNC-11 instrument, which measures functional social support [25]. In this study, 656 people were interviewed in a health center. The intraclass correlation coefficient of the 11 items was greater than 0.50 between the two formats of administering the questionnaire. Its internal consistency was excellent (α = 0.92) for the self-report format and good (α = 0.80) when administered by an interviewer; it had a two-factor structure, and the multiple linear regression explained 30% of the variance in social support through family functioning, education level, perceived internal control of health, mental health, and perception of disease severity. Therefore, it showed reliability and construct validity in that population.
In another study developed to validate the Medical Outcomes Study-Social Support Survey (MOS-SSS) questionnaire at a Spanish cancer institute, the instrument was applied to a population of 400 outpatients. Exploratory factor analysis (EFA) extracted three factors: emotional-informational support, affective support, and instrumental support. The overall internal consistency was excellent (α = 0.94). The authors concluded that the MOS-SSS is a valid instrument to measure perceived social support [26].
Other researchers, Dambi et al. [27], conducted a psychometric evaluation of the Multidimensional Scale of Perceived Social Support Scale (MSPSS) in family caregivers of cancer patients in Zimbabwe. In this study, 126 families were interviewed. The results of the EFA and the confirmatory factor analysis (CFA) showed evidence of structural validity for the two-factor model. In addition, the overall internal consistency was excellent (α = 0.90).
In addition, a study was conducted in the United States with the objective of psychometrically evaluating the Family Support Scale (FSS) developed by Littlewood et al. [23]. The FSS was applied to 255 family caregivers. The results of the EFA, using principal components and varimax rotation, showed a four-factor model: partner or spouse support, formal professional support, informal community support, and family support. The explained variance was 50%, and the overall internal consistency was acceptable (α = 0.79). The authors confirm that the FSS is a valid and reliable instrument for measuring social support in family caregivers [23].

1.5. Instruments for Measuring Social Support in Mexico

In Mexico, research regarding social support and the evaluation thereof have been developed through the design and development of valid, reliable, and culturally relevant measurement instruments. Among these instruments is the Scale of Social Support in Mexican Adults, which was designed using a natural semantic networks technique in its qualitative phase. In the quantitative phase, EFA obtained a scale of 26 items distributed in four factors: emotional, companionship, validation, and practical-instrumental. These factors explained 56% of the variance, and its overall reliability coefficient was excellent (α = 0.93). Therefore, it was determined to be a valid, reliable, and culturally relevant scale to evaluate social support in the general Mexican population [18].
Martínez et al. [28] conducted a study in Mexican patients with HIV with the aim of validating the Medical Outcomes Study (MOS) of Sherbourne and Stewart [29]. The Spanish translation by De la Revilla et al. [30] was used. A total of 313 adult patients from a National Institute of Health of Mexico City participated. Through the principal component method and varimax rotation, EFA revealed two factors: emotional and informational support (α = 0.97) and tangible support (α = 0.89). Both factors explained 72% of the variance. In addition, the two-factor structure presented a good data fit according to the CFA [28].
Another instrument is the Quality of Networks Scale (QNS) by Nava-Quiroz [31]. The QNS emerged as a result of a literature review on social support. The main findings of the review indicated a set of theoretical and methodological limitations in the published studies. One limitation was that most instruments had been developed with the objective of assessing the contribution of the social environment to the field of health and had not considered measuring or evaluating social support networks. In addition, most of these instruments had been designed based on the author’s subjective view about the construct but lacked content and social support construct validation [31].
Based on these gaps in the research literature, Nava-Quiroz [31] designed, constructed, and validated the QNS, which evaluates perceived network quality focusing on cohesion, communication, trust, help, and mutual respect. After developing the items from a panel of experts, the scale was applied to 215 psychology students. The EFA, through principal component analysis and varimax rotation, revealed three conceptually clear factors: family bond with 15 items (α = 0.94), friendship and trust with 14 items (α = 0.91), and neighbor-peers with 6 items (α = 0.69). The scale, reduced to 35 items, had excellent overall reliability (α = 0.92), and these three factors explained 48.1% of the variance [31].
In Mexico, studies have been conducted to test the psychometric properties of the QNS in the general population [32]. In this study, 200 inhabitants of the city of Morelia with an age range of 21 to 70 years participated. After a theoretical and empirical study, López-Peñaloza [32] added 10 items to the QNS that aim to measure more specific aspects of Mexican culture. With the 45 items, she obtained five factors: support from friends (α = 0.91), family support (α = 0.90), lack of support (α = 0.77), religious support (α = 0.86), and support from neighbors (α = 0.78), which explained 52.03% of the variance. The overall internal consistency was excellent (α ± 0.92). The resulting new scale was called the Social Support Network Scale (SSNS). These results of the psychometric properties of the SSNS have been useful in subsequent studies on family functioning and its relationship with social support networks [33].
Although the QNS and SSNS have been applied to university students, the general population and parents, to date, there is no evidence that they have been used and validated in populations of family caregivers of children with cancer. It should be noted that the SSNS has broader cultural content for the Mexican context and better metric properties than the QNS; therefore, the SSNS was chosen in the present study.

1.6. Problem Statement

Research regarding the impact of social support on family caregivers is promising, but one of its limitations is having reliable measurement instruments that have been validated in this specific population. The SSNS scale can be useful for this purpose and has the advantage of having been developed in the Mexican cultural context. However, it should be considered that this test was originally validated for use in the general population; therefore, its use in other populations, such as family caregivers of children with cancer, would compromise the validity and reliability of its results, due to the lack of psychometric data.
In response to this need, the aim of the present study was to analyze psychometric properties of the SSNS. To this end, we stated four objectives: (1) to test the five-factor model proposed by López-Peñaloza [32]; (2) to estimate the internal consistency reliability of the scale and its factors; (3) to describe their distributions; and (4) to prove the concurrent validity of the SSNS in relation to family functioning, quality of life, and resilience among family caregivers of children hospitalized with cancer in a National Institute of Health in Mexico City.
The expectations were as follows: the five-factor model would have a good or acceptable fit to the data, as evidence of structural construct validity; the scale would show excellent internal consistency reliability, with the five factors ranging from excellent to acceptable [32,33]; the distributions of the scale and its factors would follow a nonnormal distribution with negative skewness because most of the participants usually report that they receive very much support and a minority of people usually report that they receive very little support [32,34]; and social support would be positively correlated with family functioning [17,33], resilience [35], and quality of life [36], as evidence of concurrent construct validity.

2. Materials and Methods

2.1. Participants

Nonprobabilistic sampling was performed. The convenience sample consisted of 633 family caregivers of children with cancer hospitalized at a National Institute of Health in Mexico City. The inclusion criteria for this study were as follows: being a family caregiver of a child with chronic illness hospitalized in the institution, being older than 18 years, and signing an informed consent form. The sample met the minimum criteria of participants (n > 200) to perform EFA [37] and CFA [38]. An initial CFA was performed with the total sample of 633 caregivers. The sample was then subdivided into 2 groups. In sample A, with 316 caregivers, the factorial structure was explored, and in sample B, with 317 caregivers, the new factorial model was tested. The analyses for reliability, description of the sample, and convergent validity were performed for the total sample of 633 participants. The Results section describes the total sample.

2.2. Measuring Instruments

A sociodemographic variables questionnaire for research on family caregivers of children with chronic diseases (Q-SV) by Toledano-Toledano et al. [39]: this questionnaire contains 20 questions that evaluate social, family, and clinical variables of the patient, for example, age and sex of the patient and caregiver, diagnosis, amount of time hospitalized and time since diagnosis, family role of the caregiver (mother, father, or other family member), education, occupation, marital status, years married, number of children, family type, family life cycle, support networks, religion, and monthly household income.
Social Support Network Scale (SSNS) by López-Peñaloza [32]: this is a self-report instrument in which each item is scored on a 5-point Likert scale, ranging from 1 (strongly disagree) to 5 (strongly agree). The items of the original scale are distributed in 5 factors: support from friends (15 items: 2, 5, 6, 10, 13, 17, 18, 27, 30, 31, 34, 38, 39, 42, and 45), family support (15 items: 1, 3, 7, 8, 9, 12, 14, 20, 21, 33, 37, 40, 41, 43, and 44), lack of support (7 items: 15, 22, 23, 25, 26, 29, and 35), religious support (4 items: 4, 16, 28, and 36), and support from neighbors (4 items: 11, 19, 24, and 32). Its overall internal consistency was excellent, α = 0.92 [32].
Measurement scale of Resilience in Mexicans (RESI-M) by Palomar and Gómez [40]: this instrument has been validated in family caregivers of children with cancer, by Toledano-Toledano et al. [11]. It is scored on a 4-point Likert scale, ranging from 1 (strongly disagree) to 4 (completely agree). The 43 items are distributed across 5 factors: strength and self-confidence (19 items: 1–19), social competence (8 items: 20–27), family support (6 items: 28–33), social support (5 items: 34–38), and structure (5 items: 39–43); overall internal consistency was excellent, α = 0.93 [11,12].
Family Functioning Scale (FFS) by García-Méndez et al. [41]: this instrument scores items on a 5-point Likert scale, ranging from 1 (never) to 5 (always). The 45 items are grouped into 4 factors: positive family environment (7 items: 12, 14, 15, 17, 18, 21, and 22), cohesion/rules (5 items: 1, 6, 9, 10, and 19), hostility/avoidance of conflict (5 items: 2, 3, 5, 16, and 20), and command/problems in expression of feelings (5 items: 4, 7, 8, 11, and 13). Good overall internal consistency was found, α = 0.89 [41].
World Health Organization Quality of Life Assessment (WHOQOL-BREF): this instrument was developed by the WHOQOL Group [42], belonging to the WHO program on mental health, and consists of 26 items that are scored using a 5-point Likert scale, ranging from 1 (very dissatisfied) to 5 (very satisfied). It was adapted for Mexico by González-Celis and Sánchez-Sosa [43]. The items are distributed across 4 factors: physical health (7 items: 3, 4, 10, 15, 16, 17, and 18), psychological (6 items: 5, 6, 7, 11, 19, and 26), social relationships (3 items: 20, 21, and 22), and environment (8 items: 8, 9, 12, 13, 14, 23, 24, and 25). In addition, item 1 evaluates overall perceived quality of life, and item 2 evaluates overall perceived health status; good overall internal consistency was found, α = 0.89 [43].

2.3. Procedure

Family caregivers were interviewed by the first author of this work, and the interviews were conducted from September 2019 to February 2020 in the children’s rooms in the oncology ward of Federico Gómez Children’s Hospital of Mexico at the National Institute of Health. All the family caregivers interviewed were invited to participate voluntarily, the objectives of the research were explained, and all their questions related to the study were answered. Those family caregivers who decided to participate signed an informed consent form and answered the questionnaires themselves in a single session.

2.4. Ethical Considerations

This study is a part of the Research Project HIM/2015/017/SSA.1207 “Effects of mindfulness training on psychological distress and quality of life of the family caregiver”, which was approved on December 16, 2014, by the Research, Ethics, and Biosafety Commissions of the Hospital Infantil de México Federico Gómez National Institute of Health, in Mexico City. While conducting this study, the ethical rules and considerations for research with humans currently enforced in Mexico [44] and those outlined by the American Psychological Association [45] were followed. All family caregivers were informed of the objectives and scope of the research and their rights according to the Helsinki Declaration [46]. The caregivers who agreed to participate in the study signed an informed consent letter. Participation in this study was voluntary and did not involve payment.

2.5. Statistical Analysis

Statistical calculations were performed with SPSS version 24 (IBM; Armonk, NY, USA), AMOS version 16 (IBM; Armonk, NY, USA), and Excel 2007 (Microsoft; Redmond, WA, USA). For the first objective of determining the factorial structure, the five-factor model of López-Peñaloza [32] was contrasted with the total sample of 633 participants. Since the population was different from the population in which the SSNS was developed and the model was poorly fit to the data and presented problems of discriminant validity between 2 factors, it was decided to explore the factorial structure and contrast the resulting model.
For these analyses, the recommendations that suggest randomly dividing the sample into 2 groups were followed [47]. In a subsample of 316 participants, the exploratory part of the analysis was performed, and in the other subsample of 317 participants, the confirmatory part of the analysis was performed.
The number of factors for the EFA was determined using the new empirical Kaiser criterion [48] and Horn’s [49] parallel analysis. The factors were extracted using the maximum likelihood (ML) estimation method in accordance with the method used for the CFA. Oblique rotation was applied to interpret the factors by the Promax method. All items with loads <0.40 in the structural matrix were eliminated. In addition, the one-dimensionality of each factor was analyzed using the same procedure; in cases where the items revealed a second factor, they were eliminated one by one, starting with the item with the greatest weight in the second factor.
In the CFA, the parameters were estimated using the ML method. Since multivariate normality was not strictly fulfilled, the bias-corrected percentile (BCP) method was used to estimate 95% confidence intervals and the significance of parameters. The likelihood-ratio chi-square test (χ2) was used to evaluate the fit of each model to the data. However, this index is sensitive to sample size. Consequently, other fit indices were taken into account, such as relative chi-square (χ2/df), standardized root mean square residual (SRMR), goodness of fit index (GFI), comparative fit index (AGFI), and Tucker–Lewis index (TLI), as well as the root mean square error of approximation (RMSEA) and its 90% confidence interval (90% CI). Acceptable data fit was defined according to the following criteria: χ2/df < 3, RMSEA < 0.08, SRMR < 0.10, GFI > 0.90, AGFI > 0.85, TLI > 0.90, and CFI > 0.90. Close data fit was defined according to the following criteria: χ2/df < 2, RMSEA < 0.05, SRMR < 0.05, GFI > 0.95, AGFI > 0.90, TLI > 0.95, and CFI > 0.95 [38,50].
To improve the model fit, a data-driven strategy was employed [51], and the model was redefined based on the modification indices that indicated a significant reduction in the chi-square value. It was verified that the modifications were justified; for example, in the cases in which covariance existed between the errors, redundancy in the content was analyzed, and the item with the smallest factorial weight was eliminated. This first analysis, still exploratory in nature, was performed in the same subsample A, with 316 participants, in which the EFA was performed. In addition, in this sample, 3 models were compared: the original five-factor model, the model derived from the four-factor EFA, and the model modified with the CFA data.
This last model was cross-validated by performing CFA for the 317 participants in the second sample. In addition, the convergent validity of each measurement model (degree of certainty that the items are measuring the same construct) was estimated through 3 criteria: measurement weights (l), composite reliability (McDonald ω), and average variance extracted (AVE). An acceptable level of convergent validity was attained when the average of l ≥ 0.50; ω ≥ 0.70; and AVE ≥ 0.37 for 4 items, ≥0.28 for 6 items, and ≥0.25 for 7 or more items. Good convergent validity implied l ≥ 0.50; ω ≥ 0.80; and AVE ≥ 0.50 for 4 items, ≥0.40 for 6 items, and ≥0.25 for 12 or more items [52]. The discriminant validity between factors (degree of certainty in 2 factors not sharing the same indicators) was established by a shared variance (r 2) < 0.81 and < AVE of each factor [53].
For the second objective, estimating reliability, the Guttman’s lambda-2 coefficient (λ2) was used because the assumption of tau-equivalence was not met, as revealed by unequal measurement weights within each factor. Values of λ2 < 0.50 were considered unacceptable reliability, between 0.50–0.59 poor, between 0.60–0.69 questionable, between 0.70–0.79 acceptable, between 0.80–0.89 good, and ≥0.90 excellent [54].
In relation to the third objective, describing the distributions, measures of central tendency (mean and median), variation (standard deviation and semi-interquartile range), shape of the distribution (coefficients of asymmetry and kurtosis based on central moments), and noncentral position (quartiles and deciles) were calculated. Normality was verified by the D’Agostino–Pearson and Kolmogorov–Smirnov tests with Lilliefors correction for calculating probability. Friedman’s test was used to compare averages of the factors. The effect size was estimated by Kendall’s W coefficient, and pairwise post hoc comparisons were performed via the Dunn–Bonferroni test.
For the fourth objective, concurrent validity, the correlations were calculated via the Spearman’s correlation coefficient. As in the previous objective, nonparametric tests were used because the distributions of the scores in the 2 scales (socio-family support and religious support) and their factors did not follow a normal probability distribution [55].

3. Results

3.1. Sociodemographic Characteristics of Family Caregivers of Children with Cancer

Table 1 shows the frequencies and percentages of the sociodemographic variables belonging to the 633 caregivers who participated in the study. Additionally, Table 1 provides information on sociodemographic and clinical variables of cancer patients.

3.2. Determining the Factorial Structure

Testing the Hypothetical Five-Factor Model

First, the hypothetical five-factor model, developed by López-Peñaloza [32], was tested. When estimating the parameters by ML and their significance by BCP in the total sample of 633 participants, the fit to the data was poor for most of the indices (Table 2). All parameters were significant, except for the correlation between lack of support (F3) and religious support (F4): rF3,F4 = 0.016, BCP CI 95% (−0.097, 0.132), p = 0.777. Neighbor support had questionable reliability (λ2 = 0.606) and lacked convergent validity (AVE = 0.269 and ω = 0.576). Another additional problem with this model was the single correlation between friend support (F1) and religious support (F4), rF1,F2 = 0.953, BCP 95% CI (0.901, 1.005), p < 0.001. Consequently, the shared variance between these two factors was greater than the variance extracted from each factor (rF1,F22 = 0.908 > AVEF1 = 0.414 and AVEF4 = 0.474); therefore, there was no discriminant validity between the two factors (Table 3).

3.3. Exploration of the Factorial Structure

In subsample A, with 316 participants, there were four referenced eigenvalues greater than 1 and less than the empirical eigenvalues. Table 4 shows the arrangement obtained by extracting four factors and eliminating the items with factorial weights below 0.40 or defining a second factor when factorized individually. Horn’s parallel analysis indicated six factors, but two factors had only two items with loads greater than 0.40 in the matrix. With the 33 items retained, both criteria to determine the number of factors converged into four (Table 4).
The fit of the four-factor model specifying 33 items, derived from the EFA, was improved with respect to the original model (Δχ2 [446] = 1096.632 p < 0.001; ΔTLI = 0.064, and ΔCFI = 0.064) but did not reach an acceptable fit (Table 2). This model revealed that religious support is independent of the other three factors (family support r = 0.034, BCP 95% CI (−0.109, 0.212), p = 0.606; support from neighbors r = −0.002, BCP 95% CI (−0.146, 0.175), p = 0.972; and lack of support r = 0.090, BCP 95% CI (−0.085, 0.248), p = 0.304). Items with high correlated residuals and lower weights were eliminated to improve the fit. Following this procedure, family support was reduced to six indicators (items 1, 8, 9, 14, 21, and 43), friend support was reduced to six indicators (items 18, 30, 31, 34, 38, and 39), lack of support was reduced to four indicators (items 15, 23, 25, and 26), and religious support was reduced to four indicators (items 4, 16, 28, and 36) (Figure 1). In this way, all indices achieved an acceptable fit (Table 2). When the religious support factor remained independent (i.e., from family support r = 0.015, BCP 95% CI (−0.121, 0.164), p = 0.769; support from neighbors r = −0.013, BCP 95% CI (−0.174, 0.154), p = 0.787; and lack of support r = 0.098, BCP 95% CI (−0.046, 0.307), p = 0.210), we decided to define a three-factor model for socio-family support or trusted relationships with 16 items and a single-factor model with four indicators for religious support. Consequently, the SSNS was divided into a 16-item socio-family support scale (SFS-16) or support from trusted relationships and a 4-item religious support scale (RSS-4).
In validation subsample B, with 317 participants, the three-factor socio-family support model with 16 items (SFS-16) had an acceptable fit (Table 2 and Figure 1), and the single-factor model for religious support with four items (RSS-4) showed a good fit (Table 2 and Figure 2). The three factors of the first model had convergent validity (good for FAS and FRS and acceptable for LS) and discriminant validity (Table 3). The religious support scale (RSS-4) presented acceptable convergent validity (Table 3).
Notably, if the number of factors in the validation subsample with 317 participants is determined, the number of factors converges to four, not only by the new empirical Kaiser criterion and Horn’s parallel analysis but also by other criteria such as optimal coordinates, minimum average of the squared partial correlations, and Ruscio and Roche’s comparison data approach (Figure 3).

3.4. Reliability of the SSRS in Its Final Version (SFS-16 and RSS-4)

Table 3 shows the internal consistency reliability analyses for the total items in each model and the factors in the original model and the two final models (SFS-16 and RSS-4). Calculations were made for the total sample of 633 participants. The reliability for the 16 items of the socio-family support scale (SFS-16) and the 4 items of the religious support scale (RSS-4) were acceptable. The reliability of the family support factors (FAS-6) and friends support factors (FRS-6) were good, and that for lack of support was acceptable (LS-4).

3.5. Description of the Distributions

Table 4 shows the descriptive statistics of the 16-item socio-family support scale (SFS-16) and its three factors (family support with six items (FAS-6), friend support with six items (FRS-6), and lack of support with four items (LS-4)), as well as the religious support scale with four items (RSS-4). The scores were obtained as the ratio between the sum of the items and the number of items added (average score); therefore, the potential range varies along a continuum from 1–5. The distributions of the scores in the two three-factor scales showed asymmetry and leptokurtosis for the SFS-16 and FAS-6. None followed a normal distribution. By dividing this range into five equal intervals corresponding to the five ordered response categories, it can be interpreted that scores between 1–1.79 indicate (average) strong disagreement with receiving support, that scores between 1.8–2.59 indicate disagreement, that scores between 2.6–3.39 neither agree nor disagree, that scores between 3.4–4.19 agree, and that scores between 4.2–5 strongly agree.
The medians for the SFS-16 and the religious support scale reflect support (agree). The median of the family support factor reflected strong support (strongly agree), that of friend support was ambiguous (neither agree nor disagree), and congruently, the median rating for lack of support was disagreement (Table 5).
When comparing the averages between the three factors of the SFS-16 (FAS-6, FRS-6, and LS-4) and the religious support scale (RSS-4) by Friedman’s test, there was a significant difference, χ2 (3, n = 633) = 1065, p < 0.001, with a very large effect size (Kendall’s W = 0.837). Dunn–Bonferroni post hoc pairwise comparison tests revealed that the averages could be ordered in the following sequence of significant differences: LS-4 < FRS-6 < RSS-4 < FAS-6 (Table 5).

3.6. Concurrent Validity

Table 6 shows the correlations of the three factors for the socio-family support scale (FAS-6, FRS-6, and LS-4) and the religious support scale (RSS-4) with the family functioning, quality of life, and resilience scales. These correlations correspond to the concurrent validity analysis.
The SFS-16 scale and its three factors had significant correlations with the convergent validity criteria, except for LS-4 with the structure factor of the resilience scale. The family support factor (FAS-6) had the highest correlations, and the friend support factor (FRS-6) had the lowest. Consistent with expectations, the correlations for the total score (SFS-16) and the family support factor (FAS-6) and friend support factor (FRS-6) had signs opposite to that of the lack of support factor (LS-4).
The religious support scale (RSS-4) showed a different correlation pattern. Almost all were not significant, except for positive correlations with social support and social competence factors in the resilience scale and negative correlations with environment and physical health factors in the quality of life scale. The first three correlations had a small strength of association, and the last had a negligible strength of association.
Table 6 shows the internal consistency reliability values of these scales and their factors in the total sample of 633 participants (SSNS and for RESI-M, FFS, and WHOQOL-BREF).

4. Discussion

The first objective of this research was to determine the factorial structure of the 45-item SSNS. The expectation was five correlated factors [32]. The model of five correlated factors was not validated. The neighbor support factor had reliability problems and was not distinguishable from the religious support factor. When exploring the factorial structure, the neighbor support factor was not defined. Therefore, neighbor support does not seem to be a relevant factor for family caregivers of children with cancer. In urban settings, especially in large cities, such as Mexico City, disconnection between neighbors is frequent [56], hence this factor is probably problematic in other populations residing in large cities. Nevertheless, the fact that neighbor support cannot be distinguished from religious support, when the latter is independent of family and friend support in the final version of the scale and presents a different correlation pattern, may indicate that the support of neighbors is not supportive in the instrumental and emotional sense but, rather, the ideological, spiritual, or imaginary sense. The family caregiver could receive intangible support by talking with their neighbors (remembering the strength they gain from their faith in God, “the tests the Lord puts in our path”, “his/her mysteries”, the value of the sacrifice) and participating with them in prayer activities in the church, Las Posadas over Christmas, preparations for the Guadalupe procession, or setting up altars for the dead.
The exploration of the factorial structure showed the expected factors: family support, friend support, lack of support, and religious support. The CFA showed that religious support is independent of the other three factors. Consequently, the SSNS was separated into two independent scales. The first scale evaluates support from family and friends (from work and outside of work), that is, support from bonding relationships that probably provide instrumental and emotional resources to cope with the child’s chronic illness. This scale was called the socio-family support scale or trusted relationships scale. One of its three factors is composed of direct family support items, another of direct friend support items, and a third of inverse items pertaining to a lack of support from family and friends. The second scale evaluates religious support. Its content includes support in faith in God, in religious beliefs, in attending church, and in prayer groups. There is no specific reference to neighbors, but they must be involved in prayer groups; therefore, it is no surprise that one of the items of the neighbor support factor refers to a religious holiday in December (Las Posadas).
One might wonder if the same results from independence of religious support would be expected in areas where one religion is not dominant and even religion does not represent significant support for many people. In the study by Medellín-Fonte et al. [33] on family functioning and its relationship with social support networks, carried out in Morelia, Mexico, religious support was associated with the support from friends, but not with family support, which is consistent with the present results. Religious people tend to seek friends with their beliefs and values [57,58], hence it is likely that this result is also seen in cities with a lot of religious diversity and a high percentage of atheists or people without religion. Given that this is not the case of Mexican cities [59], this hypothesis could be tested in cities, such as London, Potsdam, or Turin, where this diversity occurs [60,61].
The SSNS was reduced from five to four factors, three intercorrelated and one independent, that is, two scales. In addition, it was reduced from 45–20 items: 16 items for the socio-family support or trusted relationships scale and 4 items for the religious support scale. This reduction in items was done essentially to achieve a good fit, improve the convergent validity of the indicators, and preserve good or acceptable reliability. However, a concern that may arise as a result of this reduction in items is content validity.
The analysis strategy followed to eliminate the items with correlated residuals that, as a result, defined a second factor when exploring the one-dimensionality of the factor in which they were loaded, together with an improvement in convergent validity (certainty that the items measure the same construct), ensures eliminating redundant items and poor indicators [47,51]. The factors with the largest item reduction were family support and friend support (from 15–6), which went from excellent reliability (λ2 > 0.90) to good (λ2 > 0.85). Importantly, all reliability coefficients inflate as the number of indicators increases, even if they are poor [54]. This slight loss in reliability was precisely due to the loss in the number of items, not to the loss of quality in the items because their measurement weights, the AVE, and even some of the correlations of concurrent validity increased. A detailed examination of the content of the remaining items showed they were reduced to their most essential or core part, yielding a shorter and more efficient scale. In the lack of support factor, which was reduced from seven to four items, the eliminated items corresponded to nonspecific content (for example, item 29 “In my family, we don’t talk about our shortcomings and mistakes”), and in the religious support factor, no items were eliminated. The four items of the neighbor support factor (which were not in the original version reported by Nava-Quiroz [31]), of which only three made explicit reference to neighbors, were all eliminated because they were absorbed in the religious support factor.
A strong finding supporting a four-factor reproducible structure with the 20 items retained from the SSNS is the convergence of the most reliable empirical criteria to determine the number of factors in both samples A (exploration) and B (of validation) as in the total sample. These criteria include Horn’s [49] parallel analysis, Velicer’s [62] minimum average squared partial correlation test, Ruscio and Roche’s [63] comparison data approach, the optimal coordinates method of Raiche et al. [64], and the new empirical Kaiser criterion from Braeken and van Assen [48].
Another concern that might arise regards the repeatability of the results in a random sample drawn from the same population. Although the sampling procedure was not probabilistic, when probabilistic sampling would be the most suitable [65], the sample collection setting (parent who regularly cares for the child in the hospital) and the large sample size guarantee that the population of the parent who is the main caregiver of a child with cancer admitted for treatment in the hospital is well represented. Consequently, the inferences derived from the present study should be limited to this population, and they would only be hypotheses in related populations. It should be noted that a probability sampling would also have this restriction [66].
The second objective of the study was to verify the reliability of the internal consistency of the scale. The expectation was excellent reliability for the 45 items of the scale, 12 items for friend support, and 12 items for family support [32,33]. This expectation was met. However, reliability downgraded to good after items were removed when redefining the socio-family support scale with 16 items and both factors (friend and family) with six items. The expectation of acceptable reliability was also met with the four items of religious support. The expectation of acceptable reliability for the seven items pertaining to lack of support was met, and the scale was kept for further reduction. The expectation of acceptable reliability for neighbor support was not met. Reliability for this factor was questionable, especially because the Cronbach’s alpha coefficient (Guttman’s lambda-3) was less than 0.60. It should be noted that in this study, Guttman lambda-2 and McDonald’s ω coefficients were used when the assumption of tau-equivalence was not met, that is, items having homogeneous true covariances [54].
The third objective was to describe the distribution of the scale and its factors. Consistent with expectations [32,34], no distribution followed a normal probability model; all presented negative asymmetry. This is because the majority of caregivers reported receiving a lot of or enough support (short interval from 5–3.4 in which many scores were concentrated), and very few indicated a lack of support (long interval from 3.99–1 in which few scores were dispersed). The strongest support came from the family (“a lot” as the central value) and religion (“quite a bit” as the central value). The central value of the friend support factor was the ambiguous response (neither agree nor disagree), and its distribution was the least negatively skewed; however, 40.1% of the participants said they received a lot of or quite a bit of support from friends, compared to 21.4% with very little or little support. Among these family caregivers, the importance of support can be ordered according to frequency as family, religious, and friend support, as there is a significant difference in the central tendency between these factors. This order of importance reflects cultural traits. In Mexico, the family occupies a central role; in turn, religion, with a clear predominance of Catholicism, is also very important in the socioemotional and spiritual lives of Mexicans. In contrast, friends occupy third place, especially in situations of crisis or need [67].
The fourth objective was to provide evidence of concurrent validity in relation to family functioning [17,33], resilience [35], and quality of life [36]. Significant correlations were expected with these three variables. These expectations were met. The SFS-16 and its factors of family support and lack of support correlated strongly with the total score for family functioning and moderately with the total scores for resilience and quality of life. The correlation of the friend support factor was medium with family functioning and low with the other two scales. Religious support was independent of these three total scores. The religious support scale only had positive correlations with support factors and social competence factors of the resilience scale and negative correlations with the environment and physical health factors of the quality of life scale. The worse the quality of life in environmental and physical aspects and the greater the social support and social competence, the more religious support is sought. Importantly, the correlations of RSS-4 with quality of life are opposite in sign of SFS-16 and its three factors (inverted scores in lack of support). As Bustamante et al. [68] point out, socio-family support seems to improve quality of life; that is, a better quality of life is a consequence of greater socio-family support. In the case of religious support, poor quality of life behaves as a cause or antecedent, as revealed by studies with people with chronic diseases [36,69] and older adults [70].
As previously noted, a limitation of the study is the use of nonprobabilistic sampling; therefore, inferences should be made with due caution among the main family caregiver of children with cancer treated in a public children’s hospital in Mexico City. This population is mainly made up of women, young people, and married women, with secondary school education, housewives, who have no income of their own, and individuals who identify as Catholics. Another limitation is that the study design was ex post facto; consequently, no causal inferences can be made. Finally, the associations and structure of social support found are limited to the measurement instrument used, the SSNS by López-Peñaloza [32].

5. Conclusions

Among these family caregivers of children with cancer, the SSNS has four factors. Three factors are intercorrelated and define a scale of socio-family support. The fourth factor is independent and defines a scale of religious support. The neighbor support factor of the original scale is not defined, as it is absorbed within the religious support scale. The consistency of the socio-family support scale with 16 items and its family support and friend support factors with 6 items each is good, and the consistency of its lack of support factor and religious support scale with 4 items each is acceptable. The distributions are asymmetric, with most participants receiving support. The strongest support comes from the family, followed by religious support and then friend support. Greater socio-family support is strongly associated with better family functioning, and socio-family support is strongly associated with greater resilience and better quality of life; however, religious support is only weakly associated with worse quality of life in environmental and physical aspects, as well as resilience aspects of greater competence and social support.
This abbreviated scale, reduced to 20 items from the original 45 items, is recommended for family caregivers due to its reliability and validity properties, in addition to being more convenient when applying a wide battery of measuring instruments. Interpretative norms of the scale should be set through percentile scores, using a probabilistic sampling. For this purpose, it should be checked which sociodemographic variables have an effect size that is at least medium on the scores in the scale and its four factors to set different norms based on the categories of these variables (for example, civil status: “married” versus “separated/divorced/widowed”; gender: “woman” versus “man”). Values below the 20th percentile could be interpreted as a low level of support. The present data suggest that greater socio-family support results in a better quality of life but that religious support is sought during times of worse quality of life, which requires further research.

Author Contributions

Conceptualization, F.T.-T. and J.M.d.l.R.; methodology, F.T.-T. and J.M.d.l.R.; software, F.G.B.; validation, R.R.F.; formal analysis, J.M.d.l.R., R.R.F. and F.G.B.; investigation, M.S.G. and L.V.G.; resources, F.T.T.; data curation, F.G.B.; writing—original draft preparation, F.T.-T. and R.R.F.; writing—review and editing, F.T.-T. and J.M.d.l.R.; visualization, L.V.G.; supervision, J.M.d.l.R.; project administration, F.T.-T.; funding acquisition, L.V.G. and M.S.G. All authors have read and agreed to the published version of the manuscript.

Funding

This work is one of the results of the research project HIM/2015/017/SSA.1207 “Effects of mindfulness training on psychological distress and quality of life of the family caregiver”; main researcher: Filiberto Toledano-Toledano. The present research was funded by federal funds for health research and was approved by the Commissions of Research, Ethics and Biosafety [Comisiones de Investigación, Ética y Bioseguridad], Hospital Infantil de México Federico Gómez National Institute of Health. The funding agency did not control the study design, data collection, analysis, and interpretation, or the manuscript writing.

Acknowledgments

We wish to thank Carlos Alberto Bermudez Moron for their support.

Conflicts of Interest

The authors declare no conflict of interest.

References

  1. World Health Organization. Cáncer: Datos y Cifras [Cancer: Facts and Figures]; World Health Organization: Geneva, Switzerland, 2020. [Google Scholar]
  2. Centro Nacional para la Salud de la Infancia y Adolescencia. Cáncer Infantil en México [Childhood Cancer in Mexico]; National Center for the Health of Children and Adolescents: Mexico City, Mexico, 2020. [Google Scholar]
  3. Toledano-Toledano, F.; Moral de la Rubia, J.; McCubbin, L.D.; Cauley, B.; Luna, D. Brief version of the coping health inventory for parents (CHIP) among family caregivers of children with chronic diseases. Health Qual. Life Outcomes 2020, 18, 104. [Google Scholar] [CrossRef] [Green Version]
  4. Toledano-Toledano, F.; Domínguez-Guedea, M.T. Psychosocial factors related with caregiver burden among families of children with chronic conditions. Biopsychosoc. Med. 2019, 13, 6. [Google Scholar] [CrossRef] [PubMed]
  5. Toledano-Toledano, F.; Luna, D. The psychosocial profile of family caregivers of children with chronic diseases: A cross-sectional study. Biopsychosoc. Med. 2020, 14, 29. [Google Scholar] [CrossRef]
  6. Marques, G. A família da criança com câncer: Necessidades sócio-econômicas [The family of the child with cancer: Socioeconomic needs]. Rev. Gaucha Enferm. 2017, 38, e2016–e2078. [Google Scholar] [CrossRef] [PubMed]
  7. Fletcher, B.S.; Miaskowski, C.; Given, B.; Schumacher, K. The cancer family caregiving experience: An updated and expanded conceptual model. Eur. J. Oncol. Nurs. 2012, 16, 387–398. [Google Scholar] [CrossRef] [Green Version]
  8. Toledano-Toledano, F.; Contreras-Valdez, J.A. Validity and reliability of the Beck Depression Inventory II (BDI-II) in family caregivers of children with chronic diseases. PLoS ONE 2018, 13, e0206917. [Google Scholar] [CrossRef]
  9. Toledano-Toledano, F.; Moral de la Rubia, J. Factors associated with anxiety in family caregivers of children with chronic diseases. Biopsychosoc. Med. 2018, 12, 20. [Google Scholar] [CrossRef] [Green Version]
  10. Kasuya, R.T.; Polgar-Bailey, P.; Takeuchi, R. Caregiver burden and burnout. A guide for primary care physicians. Postgrad. Med. 2000, 108, 119–123. [Google Scholar] [CrossRef]
  11. Toledano-Toledano, F.; Moral de la Rubia, J.; Broche-Pérez, Y.; Domínguez-Guedea, M.T.; Granados-García, V. The measurement scale of resilience among family caregivers of children with cancer: A psychometric evaluation. BMC Public Health 2019, 19, 1164. [Google Scholar] [CrossRef] [Green Version]
  12. Toledano-Toledano, F.; Moral de la Rubia, J.; McCubbin, L.D.; Liebenberg, L.; Vera Jiménez, J.A.; Rivera-Rivera, L.; Hart, A.; Barajas Nava, L.A.; Salazar García, M.; Martínez Valverde, S.; et al. Validity and reliability of the Mexican resilience measurement scale in families of children with chronic conditions. Health Qual. Life Outcomes 2017, 15, 242. [Google Scholar] [CrossRef] [Green Version]
  13. Silva-Santos, W.; Silva-Santos, J.; Marques-Hilario, J.; Cardoso-Maia, M.; Andrade, R.D.; Halboth, N. Social support perceived by family and caregivers of children with cancer. Int. J. Pediatr. Child Health 2019, 7, 28–35. [Google Scholar] [CrossRef]
  14. Cohen, S. Social relationships and health. Am. Psychol. 2004, 59, 676–684. [Google Scholar] [CrossRef] [PubMed]
  15. Cobb, S. Presidential address-1976. Social support as a moderator of life stress. Psychosom. Med. 1976, 38, 300–314. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  16. Sarason, I.; Levine, H.; Basham, R.; Sarason, B. Assessing social support: The social support questionnaire. J. Personal. Soc. Psychol. 1983, 44, 127–139. [Google Scholar] [CrossRef]
  17. Choi, E.K.; Yoon, S.J.; Kim, J.H.; Park, H.J.; Kim, J.Y.; Yu, E.S. Depression and distress in caregivers of children with brain tumors undergoing treatment: Psychosocial factors as moderators. Psychooncology 2016, 25, 544–550. [Google Scholar] [CrossRef]
  18. García-Torres, M.; García-Méndez, M.; Rivera-Aragón, S. Apoyo social en adultos mexicanos: Validación de una escala [Social support in Mexican adults: Validation of a scale]. Acta Investig. Psicol. 2017, 7, 2561–2567. [Google Scholar] [CrossRef]
  19. Akbari, M.; Alavi, M.; Irajpour, A.; Maghsoudi, J.; Lopez, V.; Cleary, M. Support needs for family caregivers of clients with mental illness in Iran: A qualitative study. Issues Ment. Health Nurs. 2018, 39, 896–903. [Google Scholar] [CrossRef]
  20. Toledano-Toledano, F.; Moral de la Rubia, J.; Domínguez-Guedea, M.T.; Nabors, L.A.; Barcelata-Eguiarte, B.E.; Rocha-Pérez, E.; Luna, D.; Leyva-López, A.; Rivera-Rivera, L. Validity and Reliability of the Beck Anxiety Inventory (BAI) for Family Caregivers of Children with Cancer. Int. J. Environ. Res. Public Health 2020, 17, 7765. [Google Scholar] [CrossRef]
  21. Marques, G.; Araujo, B. OC11-Social support and intervention focus to minimize the effects of oncological disease in healthy siblings. Nurs. Child. Young People 2016, 28, 64. [Google Scholar] [CrossRef]
  22. Kuo, C.; Fitzgerald, J.; Operario, D.; Casale, M. Social support disparities for caregivers of AIDS-Orphaned children in South Africa. J. Community Psychol. 2012, 40, 631–644. [Google Scholar] [CrossRef] [Green Version]
  23. Littlewood, K.; Swanke, J.R.; Strozier, A.; Kondrat, D. Measuring social support among kinship caregivers: Validity and reliability of the family support scale. Child Welfare 2012, 91, 59–78. [Google Scholar] [PubMed]
  24. Pfeifer, L.I.; Silva, D.B.; Lopes, P.B.; Matsukura, T.S.; Santos, J.L.; Pinto, M.P. Social support provided to caregivers of children with cerebral palsy. Child Care Health Dev. 2014, 40, 363–369. [Google Scholar] [CrossRef] [PubMed]
  25. Bellón Saameño, J.A.; Delgado Sánchez, A.; Luna del Castillo, J.D.; Lardelli Claret, P. Validez y fiabilidad del cuestionario de apoyo social funcional Duke-UNC-11 [Validity and reliability of the Duke-UNC-11 questionnaire of functional social support]. Aten. Primaria 1996, 18, 153–156, 158–163. [Google Scholar] [PubMed]
  26. Costa Requena, G.; Salamero, M.; Gil, F. Validación del cuestionario MOS-SSS de apoyo social en pacientes con cáncer [Validity of the questionnaire MOS-SSS of social support in neoplastic patients]. Med. Clin. 2007, 128, 687–691. [Google Scholar] [CrossRef] [PubMed]
  27. Dambi, J.M.; Tapera, L.; Chiwaridzo, M.; Tadyanemhandu, C.; Nhunzvi, C. Psychometric evaluation of the Shona version of the Multidimensional Scale of Perceived Social Support Scale (MSPSS-Shona) in adult informal caregivers of patients with cancer in Harare, Zimbabwe. Malawi Med. J. 2017, 29, 89–96. [Google Scholar] [CrossRef] [Green Version]
  28. Martínez, B.; Sánchez, R.; Aguilar, V.; Rodríguez, P.; Riveros, R. Adaptación y validación del cuestionario MOS de apoyo social en pacientes mexicanos con VIH+ [Adaptation and validation of the MOS social support questionnaire in Mexican patients with HIV+]. Rev. Latinoam. Med. Conduct. 2014, 11, 71–85. [Google Scholar]
  29. Sherbourne, C.D.; Stewart, A.L. The MOS social support survey. Soc. Sci. Med. 1991, 32, 705–714. [Google Scholar] [CrossRef]
  30. De la Revilla, A.; Luna-del Castillo, J.; Bailón, M.; Medina, M. Validación del cuestionario MOSS de apoyo social en atención primaria [Validation of the MOSS questionnaire on social support in primary care]. Med. Fam. 2005, 6, 10–18. [Google Scholar]
  31. Nava-Quiroz, C. Redes de Interacción Social en la Familia Mexicana: La Búsqueda de un Modelo [Social Interaction Networks in the Mexican Family: The Search for a Model]. Ph.D. Thesis, Universidad Nacional Autónoma de México, México City, Mexico, 2002. [Google Scholar]
  32. López-Peñaloza, J. Resiliencia Familiar ante el Duelo [Family Resilience to Bereavement]. Ph.D. Thesis, Universidad Nacional Autónoma de México, México City, Mexico, 2008. [Google Scholar]
  33. Fontes, M.; Heredia, M.; Peñaloza, J.; Cedeño, M.; Rodríguez-Orozco, A. Funcionamiento familiar y su relación con las redes de apoyo social en una muestra de Morelia, México [Family functioning and their relationship with social support networks in a sample of families in the city of Morelia, Mexico]. Salud Mental 2012, 35, 147–154. [Google Scholar]
  34. Yu, H.; Li, L.; Liu, C.; Huang, W.; Zhou, J.; Fu, W.; Ma, Y.; Li, S.; Chang, Y.; Liu, G.; et al. Factors associated with the quality of life of family caregivers for leukemia patients in China. Health Qual. Life Outcomes 2017, 15, 55–66. [Google Scholar] [CrossRef] [Green Version]
  35. Baran, G.; Arda Sürücü, H.; Hülya Üzel, V. Resilience, life satisfaction, care burden and social support of mothers with a child with acute lymphoblastic leukaemia: A comparative study. Scand. J. Caring Sci. 2019, 34, 340–347. [Google Scholar] [CrossRef] [PubMed]
  36. Burnette, D.; Duci, V.; Dhembo, E. Psychological distress, social support, and quality of life among cancer caregivers in Albania. Psychooncology 2017, 26, 779–786. [Google Scholar] [CrossRef] [PubMed]
  37. Ferrando, P.; Anguiano-Carrasco, C. El análisis factorial como técnica de investigación en psicología [Factor analysis as a research technique in psychology]. Pap. Psicól. 2010, 31, 18–33. [Google Scholar]
  38. Kline, R. Principles and Practice of Structural Equation Modeling; Guilford Press: New York, NY, USA, 2016. [Google Scholar]
  39. Toledano-Toledano, F.; Rodríguez-Rey, R.; Moral de la Rubia, J.; Luna, D. A sociodemographic variables questionnaire (Q-SV) for research on family caregivers of children with chronic disease. BMC Psychol. 2019, 7, 85. [Google Scholar] [CrossRef]
  40. Palomar, J.; Gómez, N. Desarrollo de una escala de medición de la resiliencia con mexicanos (RESI-M) [Construction of a measurement scale of resilience in Mexicans (RESI-M)]. Interdisciplinaria 2010, 27, 7–22. [Google Scholar]
  41. García-Méndez, M.; Rivera-Aragón, S.; Reyes-Lagunes, I.; Díaz-Loving, R. Construcción de una escala de funcionamiento familiar [Construction of a family functioning scale]. Rev. Iberoam. Diagn. Eval. Psicol. 2006, 2, 91–110. [Google Scholar]
  42. WHOQOL Group. WHOQOL-BREF. Introduction, Administration, Scoring and Generic Version of the Assessment; WHO: Geneva, Switzerland, 1996. [Google Scholar]
  43. González-Celis, R.; Sánchez-Sosa, J. Efectos de un programa cognitivo-conductual para mejorar la calidad de vida en adultos mayores [Effects of a cognitive-behavioral program to improve the quality of life in older adults]. Rev. Mex. Psicol. 2003, 20, 43–58. [Google Scholar]
  44. Sociedad Mexicana de Psicología. Código Ético del Psicólogo [Ethical Code of the Psychologist], 5th ed.; Trillas: Ciudad de México, Mexico, 2010. [Google Scholar]
  45. American Psychological Association. Ethical Principles of Psychologists and Code of Conduct. With the 2016 Amendment to Standard 3.04; American Psychological Association Press: Washington, DC, USA, 2017. [Google Scholar]
  46. World Medical Association. World Medical Association Declaration of Helsinki: Ethical principles for medical research involving human subjects. JAMA 2013, 310, 2191–2194. [Google Scholar] [CrossRef] [Green Version]
  47. Brown, T. Confirmatory Factor Analysis for Applied Research; Guilford Press: New York, NY, USA, 2006. [Google Scholar]
  48. Braeken, J.; Van Assen, M. An empirical Kaiser criterion. Psychol. Methods 2017, 22, 450–466. [Google Scholar] [CrossRef] [Green Version]
  49. Horn, J.L. A rationale and test for the number of factors in factor analysis. Psychometrika 1965, 30, 179–185. [Google Scholar] [CrossRef]
  50. Byrne, B. Structural Equation Modeling with AMOS: Basic Concepts, Applications, and Programming, 3rd ed.; Routledge: New York, NY, USA, 2016. [Google Scholar]
  51. Jöreskog, K.; Olsson, U.; Wallentin, F. Multivariate Analysis with LISREL; Springer International Publishing: New York, NY, USA, 2016. [Google Scholar]
  52. Moral de la Rubia, J. Revisión de los criterios para validez convergente estimada a través de la varianza media extraída [Review of the criteria for estimated convergent validity through the extracted mean variance]. Psychologia 2019, 13, 25–41. [Google Scholar] [CrossRef]
  53. Fornell, C.; Larcker, D. Evaluating structural equation models with unobservable variables and measurement error. J. Mark. Res. 1981, 18, 39–50. [Google Scholar] [CrossRef]
  54. Oosterwijk, P.; Van der Ark, L.; Sijtsma, K. Numerical differences between Guttmann’s reliability coefficients and the GLB. In Quantitative Psychology Research. Springer Proceedings in Mathematics and Statistics; Van der Ark, L., Bolt, D., Wang, W., Douglas, J., Wiberg, M., Eds.; Springer: Cham, Switzerland, 2016; Volume 167, pp. 155–172. [Google Scholar]
  55. Badii, M.; Guillen, A.; Serrato, O.; Aguilar, J. Correlación no-paramétrica y su aplicación en la investigaciones científica [Non-parametric correlation and its application in scientific research]. Int. J. Good Consci. 2014, 9, 31–40. [Google Scholar]
  56. Prestby, T.; App, J.; Kang, Y.; Gao, S. Understanding neighborhood isolation through spatial interaction network analysis using location big data. Environ. Plan. A Econ. Space 2020, 52, 1027–1031. [Google Scholar] [CrossRef]
  57. Golman, R.; Loewenstein, G.; Moene, K.O.; Zarri, L. The preference for belief consonance. J. Econ. Perspect. 2016, 30, 165–188. [Google Scholar] [CrossRef] [Green Version]
  58. Bahns, A.J.; Lee, J.; Crandall, C.S. Culture and mobility determine the importance of similarity in friendship. J. Cross Cult. Psychol. 2019, 50, 731–750. [Google Scholar] [CrossRef]
  59. Lopez, A.C. A Catholicism I know. In Y tú ¿qué Hora Traes? Unpacking the Privileges of Dominant Groups in Mexico; Lopez, A.C., Ed.; Brill Sense: Boston, MA, USA, 2019; pp. 19–41. [Google Scholar] [CrossRef]
  60. Wolffe, J. Towards the post-secular city? London since the 1960s. J. Relig. Hist. 2017, 41, 532–549. [Google Scholar] [CrossRef] [Green Version]
  61. Becci, I.; Burchardt, M.; Giorda, M. Religious super-diversity and spatial strategies in two European cities. Curr. Sociol. 2017, 65, 73–91. [Google Scholar] [CrossRef]
  62. Velicer, W. Determining the number of components from the matrix of partial correlations. Psychometrika 1976, 41, 321–327. [Google Scholar] [CrossRef]
  63. Ruscio, J.; Roche, B. Determining the number of factors to retain in an exploratory factor analysis using comparison data of known factorial structure. Psychol. Assess. 2012, 24, 282–292. [Google Scholar] [CrossRef] [Green Version]
  64. Raîche, G.; Walls, T.A.; Magis, D.; Riopel, M.; Blais, J.G. Non-Graphical solutions for Cattell’s scree test. Methodology 2013, 9, 23–29. [Google Scholar] [CrossRef] [Green Version]
  65. Fei, D. The repeatability in psychology: A topic model approach. Wirel. Pers. Commun. 2018, 103, 983–989. [Google Scholar] [CrossRef]
  66. Theofanidis, D.; Fountouki, A. Limitations and delimitations in the research process. Perioper. Nurs. 2019, 7, 155–162. [Google Scholar] [CrossRef]
  67. Fuller-Iglesias, H.R.; Antonucci, T.C. Familism, social network characteristics, and well-being among older adults in Mexico. J. Cross Cult. Gerontol. 2016, 31, 1–17. [Google Scholar] [CrossRef]
  68. Bustamante, A.V.; Vilar-Compte, M.; Ochoa Lagunas, A. Social support and chronic disease management among older adults of Mexican heritage: A U.S.-Mexico perspective. Soc. Sci. Med. 2018, 216, 107–113. [Google Scholar] [CrossRef]
  69. Taheri-Kharameh, Z.; Zamanian, H.; Montazeri, A.; Asgarian, A.; Esbiri, R. Negative religious coping, positive religious coping, and quality of life among hemodialysis patients. Nephrourol. Mon. 2016, 8, e38009. [Google Scholar] [CrossRef] [Green Version]
  70. Mehri Nejad, S.A.; Ramezan-Saatchi, L.; Paydar, S. Death anxiety and its relationship with social support and adherence to religion in the elderly. Salmand Iran. J. Ageing 2017, 11, 494–503. [Google Scholar] [CrossRef] [Green Version]
Figure 1. Adjusted 3-factor model estimated using subsample B (n = 317).
Figure 1. Adjusted 3-factor model estimated using subsample B (n = 317).
Ijerph 17 07820 g001
Figure 2. Single-factor model for religious support estimated using subsample B (n = 317).
Figure 2. Single-factor model for religious support estimated using subsample B (n = 317).
Ijerph 17 07820 g002
Figure 3. Determining the number of factors to retain through 5 empirical methods for the validation of the 2 new models: 16-item socio-family support scale (SFS-16) and 4-item religious support scale (RSS-4) (subsample B, n = 317).
Figure 3. Determining the number of factors to retain through 5 empirical methods for the validation of the 2 new models: 16-item socio-family support scale (SFS-16) and 4-item religious support scale (RSS-4) (subsample B, n = 317).
Ijerph 17 07820 g003
Table 1. Summary statistics of sociodemographic and clinical variables.
Table 1. Summary statistics of sociodemographic and clinical variables.
Sociodemographic and Clinical VariablesValue LabelsCaregiverChild
n%MSDn%MSD
SexWoman5158130248
Man1181933152
Age (years)1–231.707.5822635.75.754.92
3–1027944.1
11–1712820.2
18–2926642
30–3926041.1
40–5210716.9
Civil statusMarried25740.6
Cohabitant24438.5
Divorced/separated7111.2
Single558.7
Widowed61
Number of children117227.22.321.17
220231.9
318328.9
4–107612
ScholarshipUnschooled182.8
Primary12419.6
Lower secondary28244.5
Upper secondary16325.8
Higher education467.3
OccupationHousewife41365.2
Office employee8713.7
Merchant589.2
Unemployed447
Manual worker264.1
Student50.8
IncomeFrom 0–139061.6
(minimum wages) *From 1–214022.1
From 2–38513.4
From 3–491.4
From 4–660.9
From 6–820.3
Hospitalization time≤1 week39261.9
≤1 month14823.4
<6 months507.9
<2 years203.2
≥2 years233.6
DiagnosisLeukemia28044.2
Lymphoma15724.8
CNS tumors8313.1
Rhabdomyosarcoma446.9
Neuroblastoma335.2
Wilms’ tumor203.2
Retinoblastoma132.1
Bone tumors30.5
Note. n = absolute frequency, % = percentage, M = arithmetic mean, and SD = standard deviation. * Minimum wage = MXN 185.56 per day ≈ USD 7.72.
Table 2. Fit indices for 4 models for the Social Support Networks Scale (SSNS).
Table 2. Fit indices for 4 models for the Social Support Networks Scale (SSNS).
χ2χ2dfpχ2/dfRMSEA (90% CI)TLICFIGFIAGFISRMR
1. Five factors [32]. Subsample A (n = 316)
2537.851935<0.0012.7140.074 (0.070, 0.077)0.7420.7560.6930.6520.073
2. Four factors with 33 items (EFA). Subsample A (n = 316)
1441.219489<0.0012.9470.079 (0.074, 0.083)0.8060.8200.7730.7390.063
3a. Three factors for socio-family support with 16 items (CFA). Subsample A (n = 316)
213.231101<0.0012.1110.059 (0.048, 0.071)0.9310.9420.9260.9010.055
3b. Single-factor model with 4 indicators for religious support. Subsample A (n = 316)
4.79220.0912.3960.067 (0, 0.146)0.9750.9920.9930.9640.022
4a. Three factors for socio-family support with 16 items. Subsample B (n = 317)
275.856101<0.0012.7310.074 (0.064, 0.085)0.9090.9230.9030.8700.056
4b. Single-factor model with 4 indicators for religious support. Subsample B (n = 317)
0.37820.8280.1890 (0, 0.066)110.9990.9970.006
Note. Indices: χ2 = likelihood ratio chi-square statistic; df = degrees of freedom; p = probability value under the null hypothesis (H0): χ2 = 0; χ2/df = relative chi-square; RMSEA = point estimation for root mean square error of approximation and 90% confidence interval (CI) estimation, NFI = normed fit index, CFI = comparative fit index; GFI = goodness-of-fit index, AGFI = adjusted goodness-of-fit index; and SRMR = standardized root mean square residual.
Table 3. Reliability, convergent validity, and discriminant validity.
Table 3. Reliability, convergent validity, and discriminant validity.
ModelsNISample
Total (n = 633)B (n = 317)
λ2ωAVEr2Fi,F2r2Fi,F3r2Fi,F4r2Fi,F5
López-Peñaloza [32]450.926
F1: Friend support150.9130.9120.4140.2360.0990.0170.908
F2: Family support150.9210.9240.4560.3460.0080.238
F3: Lack of support70.7030.6980.272<0.0010.079
F4: Religious support40.7750.7790.4740.027
F5: Neighbor support40.6060.5760.269
EFA—4 factors330.908
Adjusted model: 4 factors200.846
SFS-16: 3 factors160.875
F1: Family support60.8810.8880.5710.1300.293
F2: Friend support60.8650.8810.5540.053
F3: Lack of support40.7170.7140.387
Single-factor model for religious support40.7750.7810.476
Note. Statistics: NI = number of items, λ2 = Guttman’s lambda-2 coefficient, ω = McDonald’s omega coefficient, AVE = average variance extracted, and r2 = shared variance between the factors Fi (per row) and F2, F3, F4, or F5. Models: López-Peñaloza [32]: friend support (items 2, 5, 6, 10, 13, 17, 18, 27, 30, 31, 34, 38, 39, 42, and 45), family support (items 1, 3, 7, 8, 9, 12, 14, 20, 21, 33, 37, 40, 41, 43, and 44), lack of support (items 15, 22, 23, 25, 26, 29, and 35), religious support (items 4, 16, 28, and 36), and neighbor support (items 11, 19, 24, and 32). EFA: 4 factors (items 1, 3, 4, 7, 8, 9, 12, 14, 15, 16, 18, 21, 22, 23, 24, 25, 26, 27, 28, 30, 31, 33, 34, 35, 36, 37, 38, 39, 41, 42, 43, 44, and 45). Adjusted model—4 factors (items 1, 4, 8, 9, 14, 15, 16, 18, 21, 23, 25, 26, 28, 30, 31, 34, 36, 38, 39, and 43). Model 3 factors: family support (items 1, 8, 9, 14, 21, and 43), friend support (items 18, 30, 31, 34, 38, and 39), and lack of support (items 15, 23, 25, and 26). Single-factor model for religious support (items 4, 16, 28, and 36).
Table 4. Descriptive statistics and structure loadings of items.
Table 4. Descriptive statistics and structure loadings of items.
ItemsStatisticsStructure Loadings
MSDFASFRSRSLS
1. My family cares for me as I care for them.4.2630.9750.712
3. In my family, we are all equally important.4.5250.8060.695
4. My faith in God helps me overcome any difficulty.4.3510.8550.630
7. My family trusts me, and I trust them.4.4110.8020.671
8. I feel supported by my parents.4.2181.0950.668
9. Our family is very close.4.1741.0200.788
12. I have my partner’s support.4.0951.2610.470
14. We talk about problems in our family.4.1960.9850.727
15. When a problem comes up in my family, we solve it violently.1.5950.9260.466
16. When I have problems, I go to church.3.2661.1120.510
18. My friends and I get together to talk, go to parties, go to the movies, etc.2.9871.1710.629
21. In my family, we encourage each other to improve ourselves.4.2220.9160.752
22. My friends and I know very little about each other.2.7221.0950.523
23. My family and I cannot spend much time together because we fight.2.1011.1610.632
24. My friends support me when I need them.3.5921.0360.555
25. There is not enough trust in my family.2.1961.2320.455
26. It is difficult to work with friends because we end up fighting.2.2031.0910.596
27. At work, most of my colleagues and I form a team to stay ahead.3.6231.0050.500
28. Prayer groups help me overcome things.3.6491.1770.715
30. My friends and I know each other very well.3.2940.9720.680
31. Outside of work, my colleagues and I get together to go out.3.0981.0510.683
33. When someone gets sick in my family, everyone worries.4.3730.9630.708
34. My friends and I visit each other’s houses.3.3071.0910.694
35. At work, we hide information that is useful for everyone.2.3731.0170.457
36. My religious beliefs help me overcome any problem.3.8991.0790.850
37. Living within my family is excellent.3.9490.9140.742
38. My friends seek each other to discuss our problems.3.4461.0110.740
39. At work, we lend each other books, articles, etc.3.0891.0410.651
41. In my family, we are very communicative with each other.3.9531.0270.727
42. My friends and I think friendship comes first.3.5890.9440.689
43. It is normal to speak openly in our family.4.0280.9340.756
44. Despite our activities, we make time to spend as a family.4.1420.9270.740
45. My friends and I think that our friendship is valuable.3.7880.9540.813
Number of items131046
Guttman’s λ20.9240.8870.7740.721
Note. Extraction method: maximum likelihood. Rotation method: Promax. Subsample A (n = 316). FAS = family support, FRS = friend support, RS = religious support, and LA = lack of support. Statistics: M = arithmetic mean, and SD = standard deviation.
Table 5. Descriptive statistics and normality tests.
Table 5. Descriptive statistics and normality tests.
StatisticsSFS-16FAS-6FRS-6LS-4RSS-4
[1, 1.8)0.5%3%6.6%47.4%2.7%
[1.8, 2.6)4%1.4%14.8%31.3%6.3%
[2.6, 3.4)21.8%7%38.4%14.7%19.1%
[3.4, 4.2)51%34.4%32.4%5.7%36.3%
[4.2, 5]22.7%54.2%7.7%0.935.6%
M3.7564.1763.1661.9893.773
SD0.6150.7900.8440.8100.822
Mdn3.8134.3333.16724
SIQR0.4060.5000.5000.6250.500
ZSk−6.938−17.711−3.5577.361−7.907
ZK34.37619.6440.5461.1652.582
K267.290 ***699.591 ***12.949 ***55.539 ***69.193 ***
Max(│D│)0.060 ***0.148 ***0.098 ***0.123 ***0.132 ***
P103.0003.333212.750
P203.2503.6672.513
P253.3753.8332.6671.2503.250
P303.50042.8331.5003.500
P403.6254.16731.7503.750
P503.8134.3333.16724
P604.0004.5003.50024
P704.1254.6673.6672.2504.250
P754.1884.8333.6672.5004.250
P804.2504.8333.8332.7504.500
P904.43854.16734.750
Note. Total sample (n = 633); M = arithmetic mean; SD = sample standard deviation; Mdn = median; SIQR = semi-interquartile range; ZSk = standardized value of sample skewness based on the third central moment; ZK3 = standardized value of sample kurtosis excess based on the fourth central moment; K2 = D’Agostino–Pearson normality test statistic; Max(∣D∣) = Kolmogorov–Smirnov normality test statistic with Lilliefors correction for probability value. *** = probability value under null hypothesis of normal distribution < 0.001. P10 to P90 = percentile scores (deciles and quartiles). SFS-16 = socio-family support scale = (I1 + I8 + I9 + I14 + (6 − I15) + I18 + I21 + (6 − I23) + (6 − I25) + (6 − I26) + I30 + I31 + I34 + I38 + I39 + I43)/16. FAS-6 = family support factor = (I1 + I8 + I9 + I14 + I21 + I43)/6. FRS-6 = friend support factor = (I18 + I30 + I31 + I34 + I38 + I39)/6. LS-4 = lack of support factor = (I15 + I23 + I25 + I26)/4. RSS-4 = religious support scale = (I4 + I16 + I28 + I36)/4.
Table 6. Convergent validity.
Table 6. Convergent validity.
ScaleFactorλ2SFS-16FAS-6FRS-6LS-4RSS-4
Family functioning scale0.8880.642 **0.616 **0.309 **−0.600 **0.020 ns
Positive family environment0.7890.584 **0.579 **0.304 **−0.467 **0.037 ns
Cohesion0.8040.536 **0.597 **0.251 **−0.439 **0.073 ns
Hostility0.679−0.394 **−0.422 **−0.156 **0.409 **−0.037 ns
Troubles with rules/feelings0.695−0.481 **−0.368 **−0.243 **0.533 **0.043 ns
Quality of life scaleQuality of life scale0.8660.411 **0.385 **0.233 **−0.316 **−0.078 ns
Physical health0.5290.182 **0.159 **0.102 *−0.123 **−0.086 *
Psychological health0.5680.330 **0.341 **0.174 **−0.263 **0.020 ns
Social relationships0.6800.418 **0.357 **0.258 **−0.344 **−0.039 ns
Environment0.7630.397 **0.354 **0.245 **−0.310 **−0.102 **
General quality of life-0.385 **0.407 **0.174 **−0.267 **−0.006 ns
Health condition-0.346 **0.330 **0.182 **−0.271 **−0.014 ns
Resilience scale0.9520.438 **0.493 **0.234 **−0.329 **0.070 ns
Strength and confidence0.9380.263 **0.329 **0.100 *−0.231 **−0.024 ns
Social competence0.8630.343 **0.288 **0.291 **−0.226 **0.103 **
Family support0.8930.493 **0.590 **0.181 **−0.391 **0.061 ns
Social support0.9170.526 **0.546 **0.297 **−0.377 **0.125 **
Structure0.7600.172 **0.216 **0.152 **−0.033ns0.070 ns
Note. Total sample (n = 633). λ2 = Guttman’s lambda-2 coefficient. Probability value under null hypothesis of rS = 0 in a 2-tailed test: ns p > 0.05, * p < 0.05, ** p < 0.001. SFS-16 = 16-item socio-family support scale; FAS-6 = family support factor; FRS-6 = friend support factor; LA-4 = lack of support factor; RSS-4 = religious support scale.
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Share and Cite

MDPI and ACS Style

Toledano-Toledano, F.; Moral de la Rubia, J.; Reyes Frometa, R.; González Betanzos, F.; Villavicencio Guzmán, L.; Salazar García, M. The Social Support Networks Scale (SSNS) for Family Caregivers of Children with Cancer: A Psychometric Evaluation. Int. J. Environ. Res. Public Health 2020, 17, 7820. https://doi.org/10.3390/ijerph17217820

AMA Style

Toledano-Toledano F, Moral de la Rubia J, Reyes Frometa R, González Betanzos F, Villavicencio Guzmán L, Salazar García M. The Social Support Networks Scale (SSNS) for Family Caregivers of Children with Cancer: A Psychometric Evaluation. International Journal of Environmental Research and Public Health. 2020; 17(21):7820. https://doi.org/10.3390/ijerph17217820

Chicago/Turabian Style

Toledano-Toledano, Filiberto, José Moral de la Rubia, René Reyes Frometa, Fabiola González Betanzos, Laura Villavicencio Guzmán, and Marcela Salazar García. 2020. "The Social Support Networks Scale (SSNS) for Family Caregivers of Children with Cancer: A Psychometric Evaluation" International Journal of Environmental Research and Public Health 17, no. 21: 7820. https://doi.org/10.3390/ijerph17217820

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop