The most prominent goal when conducting a meta-analysis is to estimate the true effect size across a set of studies. This approach is problematic whenever the analyzed studies have qualitatively different results; that is, some studies show an effect in the predicted direction while others show no effect and still others show an effect in the opposite direction. In case of such qualitative differences

Van Oest (2019) developed a framework to assess interrater agreement for nominal categories and complete data. We generalize this framework to all four situations of nominal or ordinal categories and complete or incomplete data. The mathematical solution yields a chance-corrected agreement coefficient that accommodates any weighting scheme for penalizing rater disagreements and any number of raters

Over the past decade, there has been a surge of empirical research investigating mental disorders as complex systems. In this article, we investigate how to best make use of this growing body of empirical research and move the field toward its fundamental aims of explaining, predicting, and controlling psychopathology. We first review the contemporary philosophy of science literature on scientific

This study presents a Bayesian inference approach to evaluate the relative importance of predictors in regression models. Depending on the interpretation of importance, a number of indices are introduced, such as the standardized regression coefficient, the average squared semipartial correlation, and the dominance analysis measure. Researchers' theories about relative importance are represented by

Some important research questions require the ability to find evidence for two conditions being practically equivalent. This is impossible to accomplish within the traditional frequentist null hypothesis significance testing framework; hence, other methodologies must be utilized. We explain and illustrate three approaches for finding evidence for equivalence: The frequentist two one-sided tests procedure

Model fit assessment is a central component of evaluating confirmatory factor analysis models and the validity of psychological assessments. Fit indices remain popular and researchers often judge fit with fixed cutoffs derived by Hu and Bentler (1999). Despite their overwhelming popularity, methodological studies have cautioned against fixed cutoffs, noting that the meaning of fit indices varies based

Individuals routinely differ in how they present with psychiatric illnesses and in how they respond to treatment. This heterogeneity, when overlooked in data analysis, can lead to misspecified models and distorted inferences. While several methods exist to handle various forms of heterogeneity in latent variable models, their implementation in applied research requires additional layers of model crafting

The analysis of psychological networks has become common in multiple subfields including clinical, social, and personality psychology, where the focus is often on identifying highly central nodes that represent symptoms, beliefs, or traits. However, the boundaries of these networks are often ambiguous and relevant nodes are often missing from the network. In this article, we use a series of simulations

Structural equation modeling (SEM) is a widespread approach to test substantive hypotheses in psychology and other social sciences. However, most studies involving structural equation models neither report statistical power analysis as a criterion for sample size planning nor evaluate the achieved power of the performed tests. In this tutorial, we provide a step-by-step illustration of how a priori

The replicability crisis has drawn attention to numerous weaknesses in psychology and social science research practice. In this work we focus on three issues that cannot be addressed with replication alone, and which deserve more attention: Functional misspecification, structural misspecification, and unreliable interpretation of results. We demonstrate a number of possible consequences via simulation

As a powerful tool for synthesizing information from multiple studies, meta-analysis has gained high popularity in many disciplines. Conclusions stemming from meta-analyses are often used to direct theory development, calibrate sample size planning, and guide critical decision-making and policymaking. However, meta-analyses can be conflicted, misleading, and irreproducible. One of the reasons for meta-analyses

Causal inference in psychological research is typically hampered by unobserved confounding. A copula-based method can be used to statistically control for this problem without the need for instruments or covariates, given relatively lenient distributional assumptions on independent variables and error terms. The current study aims to: (a) provide a user-friendly introduction to the copula method for

Theories can be represented as statistical models for empirical testing. There is a vast literature on model selection and multimodel inference that focuses on how to assess which statistical model, and therefore which theory, best fits the available data. For example, given some data, one can compare models on various information criterion or other fit statistics. However, what these indices fail

Researchers increasingly use Bayes factor for hypotheses evaluation. There are two main applications: null hypothesis Bayesian testing (NHBT) and informative hypothesis Bayesian testing (IHBT). As will be shown in this article, NHBT is sensitive to the specification of the scale parameter of the prior distribution, while IHBT is not. As will also be shown in this article, for NHBT using four different

Individual differences in the timing of developmental processes are often of interest in longitudinal studies, yet common statistical approaches to modeling change cannot directly estimate the timing of when change occurs. The time-to-criterion framework was recently developed to incorporate the timing of a prespecified criterion value; however, this framework has difficulty accommodating contexts

Zero responses and their equivalents-for example, never, none, not at all-are commonly observed on measures of psychopathology inquiring about symptom frequencies, particularly when these measures are administered to community samples. Measurement researchers typically accommodate multivariate zero inflation by including a nonpathological class of respondents who endorse zero for all symptoms. While

Curbstoning, the willful fabrication of survey responses by outside data collectors, threatens the integrity of the inferences drawn from data. Researchers who outsource data collection to survey collection panels, field interviewers, or research assistants should validate whether each collection agent actually collected the data. Our review of the survey auditing literature demonstrates a consistent

This study presents the situational optimization function analysis (SOFA) and has three aims. First, to develop a Bayesian implementation of SOFA. Second, to compare this implementation with three other maximum likelihood-based models in their accuracy to estimate true scores. The third aim is to show how joint modeling can be used for validity research. A simulation study was used to examine the second

To prevent mistakes in psychological assessment, the precision of test norms is important. This can be achieved by drawing a large normative sample and using regression-based norming. Based on that norming method, a procedure for sample size planning to make inference on Z-scores and percentile rank scores is proposed. Sampling variance formulas for these norm statistics are derived and used to obtain

We recently developed a simple and general sequential sampling method for testing null hypotheses, the independent segments procedure (ISP; Miller & Ulrich, 2021). In this reply, we discuss the comments of Erdfelder and Schnuerch (2021) and Lakens (2021), who consider alternative methods such as the sequential probability ratio test (SPRT) and the group sequential design (GSD), respectively. We evaluate

In this comment, we report a simulation study that assesses error rates and average sample sizes required to reach a statistical decision for two sequential procedures, the sequential probability ratio test (SPRT) originally proposed by Wald (1947) and the independent segments procedure (ISP) recently suggested by Miller and Ulrich (2020). Following Miller and Ulrich (2020), we use sequential one-tailed

Psychological science would become more efficient if researchers implemented sequential designs where feasible. Miller and Ulrich (2020) propose an independent segments procedure where data can be analyzed at a prespecified number of equally spaced looks while controlling the Type I error rate. Such procedures already exist in the sequential analysis literature, and in this commentary, I reflect on

Technology now makes it possible to understand efficiently and at large scale how people use language to reveal their everyday thoughts, behaviors, and emotions. Written text has been analyzed through both theory-based, closed-vocabulary methods from the social sciences as well as data-driven, open-vocabulary methods from computer science, but these approaches have not been comprehensively compared

Integrative data analysis (IDA) jointly models participant-level data from multiple studies. In psychology, IDA has been conducted using different fixed-effects and multilevel modeling (MLM) approaches. However, evaluations regarding the performance of these models in an IDA context are limited. The goal of this article is to evaluate three fixed-effects models (aggregated vs. disaggregated vs. study-specific

Structural equation models (SEMs) are widely used to handle multiequation systems that involve latent variables, multiple indicators, and measurement error. Maximum likelihood (ML) and diagonally weighted least squares (DWLS) dominate the estimation of SEMs with continuous or categorical endogenous variables, respectively. When a model is correctly specified, ML and DWLS function well. But, in the

hen multiple mediators exist on the causal pathway from treatment to outcome, path analysis prevails for disentangling indirect effects along paths linking possibly several mediators. However, separately evaluating each indirect effect along different posited paths demands stringent assumptions, such as correctly specifying the mediators' causal structure, and no unobserved confounding among the mediators

Estimating causal relations between two or more variables is an important topic in psychology. Establishing a causal relation between two variables can help us in answering that question of why something happens. However, using solely observational data are insufficient to get the complete causal picture. The combination of observational and experimental data may give adequate information to properly

Social scientists have become increasingly interested in using intensive longitudinal methods to study social phenomena that change over time. Many of these phenomena are expected to exhibit cycling fluctuations (e.g., sleep, mood, sexual desire). However, researchers typically employ analytical methods which are unable to model such patterns. We present spectral and cross-spectral analysis as means

Exploratory factor analysis (EFA) is a popular method for elucidating the latent structure of data. Unfortunately, EFA models can sometimes produce improper solutions with nonsensical results. For example, improper EFA solutions can include one or more Heywood cases, where common factors account for 100% or more of an observed variable's variance. To better understand these senseless estimates, we

issing values that are missing not at random (MNAR) can result from a variety of missingness processes. However, two fundamental subtypes of MNAR values can be obtained from the definition of the MNAR mechanism itself. The distinction between them deserves consideration because they have characteristic differences in how they distort relationships in the data. This has implications for the validity

Autoregressive and vector autoregressive models are a driving force in current psychological research. In affect research they are, for instance, frequently used to formalize affective processes and estimate affective dynamics. Discrete-time model variants are most commonly used, but continuous-time formulations are gaining popularity, because they can handle data from longitudinal studies in which

n real data analysis with structural equation modeling, data are unlikely to be exactly normally distributed. If we ignore the non-normality reality, the parameter estimates, standard error estimates, and model fit statistics from normal theory based methods such as maximum likelihood (ML) and normal theory based generalized least squares estimation (GLS) are unreliable. On the other hand, the asymptotically

Traditionally, researchers have used time series and multilevel models to analyze intensive longitudinal data. However, these models do not directly address traits and states which conceptualize the stability and variability implicit in longitudinal research, and they do not explicitly take into account measurement error. An alternative to overcome these drawbacks is to consider structural equation

Monte Carlo simulations are widely used in the social sciences to explore the viability of analytic methods in the face of assumption violations. Simulation results, however, may not be applicable to substantive research applications because they often are conducted under idealized rather than realistic conditions. Shortcomings of simulation design are discussed using linear equations as a case study

Monte Carlo simulations are widely used in the social sciences to explore the viability of analytic methods in the face of assumption violations. Simulation results, however, may not be applicable to substantive research applications because they often are conducted under idealized rather than realistic conditions. Shortcomings of simulation design are discussed using linear equations as a case study

Mixed-worded scales require a more careful reading and answering process than scales with only one type of wording. The present study is about the unintended consequences of using such scales, because typically, not all respondents answer positively and negatively worded items consistently. This population heterogeneity-meaning that there are distinct groups of consistently and inconsistently answering

Mixed-worded scales require a more careful reading and answering process than scales with only one type of wording. The present study is about the unintended consequences of using such scales, because typically, not all respondents answer positively and negatively worded items consistently. This population heterogeneity-meaning that there are distinct groups of consistently and inconsistently answering

Current interrater reliability (IRR) coefficients ignore the nested structure of multilevel observational data, resulting in biased estimates of both subject- and cluster-level IRR. We used generalizability theory to provide a conceptualization and estimation method for IRR of continuous multilevel observational data. We explain how generalizability theory decomposes the variance of multilevel observational

Current interrater reliability (IRR) coefficients ignore the nested structure of multilevel observational data, resulting in biased estimates of both subject- and cluster-level IRR. We used generalizability theory to provide a conceptualization and estimation method for IRR of continuous multilevel observational data. We explain how generalizability theory decomposes the variance of multilevel observational

Structural equation modeling (SEM) of ordinal data is often performed using normal theory maximum likelihood estimation based on the Pearson correlation (cont-ML) or using least squares principles based on the polychoric correlation matrix (cat-LS). While cont-ML ignores the categorical nature of the data, cat-LS assumes underlying multivariate normality. Theoretical results are provided on the validity

Psychology research often focuses on interactions, and this has deep implications for inference from nonrepresentative samples. For the goal of estimating average treatment effects, we propose to fit a model allowing treatment to interact with background variables and then average over the distribution of these variables in the population. This can be seen as an extension of multilevel regression and

Deep learning has revolutionized predictive modeling in topics such as computer vision and natural language processing but is not commonly applied to psychological data. In an effort to bring the benefits of deep learning to psychologists, we provide an overview of deep learning for researchers who have a working knowledge of linear regression. We first discuss several benefits of the deep learning

In a world of big data and computational resources, there has been a growing interest in further validating computational models of decision making by subjecting them to more rigorous constraints. One prominent area of study is model-based cognitive neuroscience, where measures of neural activity are explained and interpreted through the lens of a cognitive model. Although some early work has developed

Latent growth curve models with spline functions are flexible and accessible statistical tools for investigating nonlinear change patterns that exhibit distinct phases of development in manifested variables. Among such models, the bilinear spline growth model (BLSGM) is the most straightforward and intuitive but useful. An existing study has demonstrated that the BLSGM allows the knot (or change-point)

Latent growth curve models with spline functions are flexible and accessible statistical tools for investigating nonlinear change patterns that exhibit distinct phases of development in manifested variables. Among such models, the bilinear spline growth model (BLSGM) is the most straightforward and intuitive but useful. An existing study has demonstrated that the BLSGM allows the knot (or change-point)

Lim and Jahng (2019) recently reported simulations supporting the conclusion that traditional parallel analysis (PA) performs more reliably than do more recent PA versions, particularly in the presence of minor factors acting as population error. With noise factors, correct identification of the number of main factors may, however, mean retaining a noise dimension at the expense of missing a signal

Lim and Jahng (2019) recently reported simulations supporting the conclusion that traditional parallel analysis (PA) performs more reliably than do more recent PA versions, particularly in the presence of minor factors acting as population error. With noise factors, correct identification of the number of main factors may, however, mean retaining a noise dimension at the expense of missing a signal

Temporal complexity refers to qualities of a time series that are emergent, erratic, or not easily described by linear processes. Quantifying temporal complexity within a system is key to understanding the time based dynamics of said system. However, many current methods of complexity quantification are not widely used in psychological research because of their technical difficulty, computational intensity

Regularization methods such as the least absolute shrinkage and selection operator (LASSO) are commonly used in high dimensional data to achieve sparser solutions. Recently, methods such as regularized structural equation modeling (SEM) and penalized likelihood SEM have been proposed, trying to transfer the benefits of regularization to models commonly used in social and behavioral research. These

This article compares two missing data procedures, full information maximum likelihood (FIML) and multiple imputation (MI), to investigate their relative performances in relation to the results from analyses of the original complete data or the hypothetical data available before missingness occurred. By expressing the FIML estimator as a special MI estimator, we predicted the expected patterns of discrepancy

Newly available data streams from experience sampling studies and social media are providing new opportunities to study individuals' dyadic relations. The "one-with-many" (OWM) model (Kenny et al., 2006; Kenny & Winquist, 2001) was specifically constructed for and is used to examine features of multiple dyadic relationships that one set of focal persons (e.g., therapists, physicians) has with others

Psychological research often builds on between-group comparisons of (measurements of) latent variables; for instance, to evaluate cross-cultural differences in neuroticism or mindfulness. A critical assumption in such comparative research is that the same latent variable(s) are measured in exactly the same way across all groups (i.e., measurement invariance). Otherwise, one would be comparing apples

Measures of explained variance, ΔR2 and f,2 are routinely used to evaluate the size of moderation effects. However, they suffer from several drawbacks: (a) Not all the variance components of the outcome variable Y are related to the effect of moderation, and so an effect size with the total variance of Y as the denominator cannot accurately characterize the moderation effect; (b) moderation and interaction

We investigated the reproducibility of the major statistical conclusions drawn in 46 articles published in 2012 in three APA journals. After having identified 232 key statistical claims, we tried to reproduce, for each claim, the test statistic, its degrees of freedom, and the corresponding p value, starting from the raw data that were provided by the authors and closely following the Method section

One of the most important methods that psychological scientists use to understand behavior and cognition is theorizing. Increasingly, theorizing is used to support not only additive hypotheses, but also multiplicative ones. And yet, authors often struggle to provide adequate theoretical justifications for multiplicative hypotheses. In the late 1960s and early 1970s, W. Mischel averred that, in "strong"

We develop a Bayesian group-based trajectory model (GBTM) and extend it to incorporate dual trajectories and Bayesian model averaging for model selection. Our framework lends itself to many of the standard distributions used in GBTMs, including normal, censored normal, binary, and ordered outcomes. On the model selection front, GBTMs require the researcher to specify a functional relationship between

Despite increased attention to the role of statistical power in psychological studies, navigating the process of sample size planning for linear regression designs can be challenging. In particular, it can be difficult to decide upon an appropriate value for the effect size, owing to a variety of factors, including the influence of the correlations among the predictors and between the other predictors

An important task of meta-analysis is to observe, quantify, and explain the heterogeneity across the reported effect sizes of primary studies. A primary issue that challenges this task is the myriad of subtle factors that could have contributed to the observed heterogeneity. We leveraged the recent advances in theoretical machine learning to develop a novel latent mixture-based method for disentangling

A huge portion of what we know about how humans develop, learn, behave, and interact is based on survey data. Researchers use longitudinal growth modeling to understand the development of students on psychological and social-emotional learning constructs across elementary and middle school. In these designs, students are typically administered a consistent set of self-report survey items across multiple

The randomized pretest posttest design is common in psychology, as is the corresponding missing data concern. Although missing data handling has seen advances over the past several decades, effective and practical solutions for handling missing data in randomized pretest posttest designs are lacking, particularly when assumptions of commonly used statistical models are violated. Although analysis of