Journal of Personality ( IF 5.429 ) Pub Date : 2021-02-22 , DOI: 10.1111/jopy.12626 Aidan Wright , Jonathan Adler , Colin DeYoung , Catherine Emily Durbin , Robin Edelstein , Christian Jordan , Kenneth Locke , Shanhong Luo , Donald Lynam , Sophie Stumm , Virgil Zeigler‐Hill , Howard Tennen
Changes are afoot in the way the scientific community is approaching the practice and reporting of research. Spurred by concerns about the fundamental reliability (i.e., replicability), or rather lack thereof, of contemporary psychological science (e.g., Open Science Collaboration, 2015), as well as how we go about our business (e.g., Gelman & Loken, 2014), several recommendations have been furthered for increasing the rigor of the published research through openness and transparency. The Journal has long prized and published the type of research with features, like large sample sizes (Fraley & Vazire, 2014), that has fared well by replicability standards (Soto, 2019). The type of work traditionally published here, often relying on longitudinal samples, large public datasets (e.g., Midlife in the United States Study), or complex data collection designs (e.g., ambulatory assessment and behavioral coding) did not seem to fit neatly into the template of the emerging transparency practices. However, as thinking in the open science movement has progressed and matured, we have decided to full‐throatedly endorse these practices and join the growing chorus of voices that are encouraging and rewarding more transparent work in psychological science. We believe this can be achieved while maintaining the “big tent” spirit of personality research at the Journal with a broad scope in content, methods, and analytical tools that has made it so special and successful all of these years. Moving forward, we will be rigorously implementing a number of procedures for openness and transparency consistent with the Transparency and Open Science Promotion (TOP) Guidelines.
The TOP Guidelines are organized into eight standards, each of which can be implemented at three levels of stringency (Nosek et al., 2015). In what follows, we outline the initial TOP Standards Levels adopted by the Journal and the associated rationale. Generally, we have adopted Level 2 standards, as we believe these strike a desirable balance between compelling a high degree of openness and transparency while not being overly onerous and a deterrent for authors interested in the Journal as an outlet for their work.
1. Citation Standards—Science is a cumulative process. The intellectual thread of scholarly contributions to scientific works are traceable through citations in the reference section of manuscripts. Citations also form the basis of evaluating a journal's impact (i.e., impact factors are calculated based on the number of citations a journal receives over a period of time), an individual article's impact, and an individual's own impact as various types of committees consider the number of times a scientist is cited when evaluating an individual for awards, promotion, and tenure. Whereas these expectations have become de rigueur for journal articles, the same has not traditionally been true of other scholarly products such as data sets, thoughtfully designed and painstakingly collected, then made open access for the greater good, or even program code, routines, or pipelines that might have wide applicability in a field. These are now understood to be important contributions that facilitate the work of and enrich the productivity of many other scientists, when they are made openly available. To incentivize and reward such sharing, as well as give credit where credit is due, the Journal will adopt a Level 2 for Citations Standards, which indicates that all data sets and program code and other methods must be appropriately cited.
2 & 3. Data and Analytic Transparency—Scientists work hard to collect and analyze their data. Data collection and analysis are two of the most important steps in the scientific process. They are also frequently complex and offer even the most conscientious and well‐intentioned researcher ample opportunities for making mistakes. Furthermore, in today's landscape of quantitative psychology, statistical models are increasingly complex and require the estimation of many parameters, only some of which are of focal interest for an article and make it into the reported text. Although parsimony in the service of efficiency and clarity is understandable, many important decisions about model specification can go unreported and, therefore, misunderstood by readers. One can think of many examples in the contemporary literature where reported results are uncertain, not because of any fault of the researcher, but because full analytic details elude the readers' grasp. Contemporary personality science is often interested in complex questions that do not lend themselves to, for instance, a neat 2 × 2 factorial design and an ANOVA that can be described in 1–2 lines of text. Taken together, the potential for good faith error and model underreporting are compelling reasons for the Journal to adopt Data and Analytic Transparency at Level 2, which require that the data and analyses (i.e., the syntax or code) be clearly and precisely documented and maximally available to any researcher interested in reproducing the published results or replicating the procedure in their own laboratory. In effect, this means that in order to be published in the Journal, an article will need to (a) post the data used for the reported research in a permanent public repository (e.g., ICPSR, osf.io), or if it is a publicly available data set already, provide the code needed to download and reproduce the specific data set, and (b) always provide the statistical code used in analyses. Naturally there are exceptions to these rules, and in some instances “maximally available” will mean a description of who can be contacted to request the data for verification, if need be. Examples of justifiable reasons would be an inability to de‐identify or anonymize the data (e.g., GPS locations and full life story narrative accounts) or archival data that the participants expressly prohibited from sharing in this manner. This latter example is not uncommon of older datasets (which still offer rich opportunities for new investigations), although we expect it to be increasingly rare, as public archiving expectations are increasing both as part of our research culture and as mandated by funding agencies (e.g., NIMH and NIAAA). Finally, we are sympathetic to researchers who are concerned that open posting of data will lead to them getting scooped with the fruits of their own labor. Although we believe these concerns to be natural, but largely overblown, there are a couple of situations in which it might be especially risky, such as when someone wants to publish the first paper of large complex or longitudinal data set, or publish the first paper from their dissertation data, which was intended to be a nest egg for their early career. In either case the authors may understandably have several papers planned and would benefit from the time to work through those submissions without concern. In these instances, authors can request an embargo (of up to 12 months) and explain why the embargo would be justified. The Editor will consider the embargo request and accept it, deny it, or ask for revisions. We understand the concerns, and believe an embargo is an appropriate mechanism for serving the masters of transparency and giving researchers protected time to publish the main aims of their studies without concern for being scooped. Please note that only the data relevant/used in the reported studies need be made public; these requirements do not extend to the entire dataset, which may be much larger in scope.
4. Research Materials Transparency—Traveling hand in hand with data and analyses are the actual materials used in the research procedures. These may include questionnaires or specific stimuli presented to participants. Fully transparent reporting of research would also include open and accessible materials for all relevant aspects of the reported study. At the same time, many of the studies that are published in the Journal employ well established measures, frequently copyrighted (e.g., NEO‐PI‐3), as opposed to novel stimuli or de novo measures. This is not uniformly the case, and it is not uncommon for researchers to generate new questionnaires, especially in the case of innovative designs (e.g., intensive longitudinal designs). Therefore, the Journal is adopting a Level 2 for Research Materials Transparency, which requires all materials that can be made open and available are open and available at the time of submission. Although the culture and expectations of the Journal reviewers and readers already include full reporting of specific items when they are novel, which is consistent with this policy, this reinforces that expectation. In the case of proprietary measures, authors should request an exception to the Level 2 requirements and post an explanation for why they cannot be reproduced verbatim. We very much expect authors will strive to be maximally transparent with their research materials for papers published in the Journal.
5. Design and Analysis Transparency—Many crucial design and analytic decisions are not necessarily revealed by making available the raw data, syntax/code, and materials. For instance, decisions like initial target sample size, who was excluded, how outliers were treated, achieved reliability (e.g., level of agreement among coders or McDonald's Omega for questionnaires), and the precise level of p values achieved for any statistical test are all important for evaluating a study. At the same time, these are not uniformly reported. We suspect that it is largely out of inattention or forgetfulness, and therefore this is easily remedied by explicitly asking for important relevant details for the type of research regularly published in the journal. Therefore, we are adopting a Design and Analysis Transparency at Level 2, where we will require authors to review our reporting criteria and include those that are relevant for the research.
6 & 7. Pre‐registration—Perhaps the most important step toward fully transparent and open research practices is to engage in registration of the study design and analysis plans prior to executing them. The purest form of registration is a registered report (see below), in which a study is evaluated for its design and analytic plan prior to beginning the research, much like a federal grant or graduate student milestone project (i.e., thesis or dissertation) as done in the United States. However, as Benning and colleagues (2019) discuss, registration exists along a continuum and can be fruitfully incorporated in research plans for ongoing data collections, qualitative research designs (Haven & vanGrootel, 2019), and even datasets that are archival and public (Weston et al., 2019). Perfect should not be the enemy of good here, and it is important to remember that the goal of registration, regardless of stage, is to be transparent about design and analytic decisions—it's not a score card for how well one can predict their findings. As such, deviations from the planned approach can, and often do, happen, and that is perfectly acceptable, so long as they are clearly identified as such in the report. Moreover, registration serves to clearly delineate between hypothesis‐generating (or exploratory) and hypothesis‐testing (i.e., confirmatory) research. We wish to stress that here at the Journal we value both exploratory and confirmatory studies, and welcome submissions from both. Indeed, we view this as another instance where most research falls along a continuum, being neither purely exploratory nor confirmatory in nature (see e.g., Wright, 2017). For Registrations we are adopting a Level 2, which means that studies that have been registered conform to expected specification of registration, and the links to the registration are provided prior to submission for evaluation alongside the research report. Appropriate registration is encouraged regardless of where it falls along the continuum mentioned above, and as we have noted earlier, we continue to prize work using large public and longitudinal data sets that are not traditionally the target of registration but can be incorporated within this framework with thoughtful consideration. Finally, registration is not a requirement of publication.
8. Replication—Reliability is the bedrock of science—if you cannot get the same results under the same apparent conditions, the findings are uninterpretable because it reflects poorly understood, possibly random, variation across studies. Accordingly, the Journal encourages submissions that are direct replications of published work, especially in this Journal. It is important to note that not all studies merit direct replication effort, and the same standards of importance and contribution will be applied to replication studies as they are to novel submission. To ensure that replication studies are evaluated on their faithful adherence to the designs of the original studies and not the associated findings, we will implement a Level 3 for Replication, and will only consider direct replication studies as Registered Reports (see next section).
9. Registered Report—We will now be accepting registered reports for consideration at the Journal. These merit a separate section because their process of evaluation and implementation differs from other registered publications. In brief, studies seeking consideration in this track will receive two stages of review (see https://osf.io/8mpji/wiki/home/). In the first stage, submissions should include an introduction and methods (including planned analysis) section, detailed enough to critically evaluate all important aspects of the research. This will be reviewed by relevant experts who may suggest the study and plan be accepted as is, recommend modifications to the planned protocol, or reject the study, due to poor design or lack of importance of the question. Assuming the study is ultimately accepted as this stage, either initially or following revisions, the study can be considered “accepted in principle.” The expectation is that the author(s) would then go and faithfully implement the agreed upon procedures for data collection and analysis, and then submit the complete project for evaluation at Stage 2. At this second stage, studies will be evaluated for adherence to the initial plan, and if this is achieved will be accepted.
10. Open Science Badges—When appropriate, authors may now apply for and can be awarded open science badges to be displayed on their publication
These are an initial effort at adopting and incentivizing openness and transparency here at the Journal. Our intention is to select levels of standards that will best serve the community of scientists who rely on the Journal as an outlet for communicating important developments, cutting edge findings, and research that falls across the hypothesis‐generating and hypothesis‐testing continuum in the broad field of personality. At the same time, we may find after a few seasons of wear that despite our best efforts the mantle was mis‐tailored, and ill fits the purposes for which it was sewn. In that case, we will adjust as needed to calibrate the implementation of these procedures to best suit the needs of the community of scholars who contribute to the continued success of the Journal. Science is itself a cumulative and recursive practice and, therefore, the publication practices of science must mirror that ideal.
中文翻译:
人格杂志的透明度和开放科学
科学界处理研究实践和报告的方式正在发生变化。受到对当代心理科学的基本可靠性(即,可复制性)或更确切地说缺乏可靠性的担忧(例如,开放科学合作, 2015 年)以及我们如何开展业务(例如,Gelman & Loken,2014 年)的刺激 ,为了通过公开和透明来提高已发表研究的严谨性,已经提出了几项建议。该杂志长期以来一直珍视和发表具有特征的研究类型,例如大样本量(Fraley & Vazire, 2014 年),在可复制性标准方面表现良好(Soto, 2019 年))。传统上在这里发表的工作类型,通常依赖于纵向样本、大型公共数据集(例如,美国中年研究)或复杂的数据收集设计(例如,动态评估和行为编码)似乎并不适合新兴透明度做法的模板。然而,随着开放科学运动中思想的进步和成熟,我们决定全力支持这些做法,并加入越来越多的声音,这些声音鼓励和奖励心理科学中更透明的工作。我们相信这可以实现,同时保持期刊人格研究的“大帐篷”精神,在内容、方法和分析工具方面具有广泛的范围,这些年来使其如此特别和成功。向前进,
TOP 指南分为八个标准,每个标准都可以在三个严格级别上实施(Nosek 等人, 2015 年)。在下文中,我们概述了期刊采用的初始 TOP 标准级别及其相关理由。一般而言,我们采用了 2 级标准,因为我们相信这些标准在强调高度开放性和透明度之间取得了理想的平衡,同时又不会过于繁重,并且对有兴趣将期刊作为他们工作的出口的作者来说是一种威慑。
1.引文标准——科学是一个累积的过程。对科学作品的学术贡献的知识线索可以通过手稿参考部分的引用进行追踪。引用也是评估期刊影响的基础(即,影响因子是根据期刊在一段时间内收到的引用次数计算的)、一篇文章的影响以及个人自身的影响,因为各种类型的委员会都认为在评估一个人的奖项、晋升和任期时,科学家被引用的次数。尽管这些期望已成为期刊文章的必备条件,但其他学术产品传统上并非如此,例如数据集,经过精心设计和精心收集,然后为了更大的利益开放获取,甚至程序代码,在某个领域可能具有广泛适用性的例程或管道。这些现在被认为是重要的贡献,当它们被公开提供时,可以促进许多其他科学家的工作并提高他们的生产力。为了激励和奖励这种分享,以及在信用到期时给予信用,期刊将采用等级2为参考文献的标准,这表明所有数据集的程序代码和其它方法必须被适当地引用。
2 & 3. 数据和分析透明度——科学家们努力收集和分析他们的数据。数据收集和分析是科学过程中最重要的两个步骤。它们通常也很复杂,即使是最认真和善意的研究人员也有充足的犯错机会。此外,在当今定量心理学的格局中,统计模型越来越复杂,需要估计许多参数,只有其中一些是文章关注的焦点,并将其纳入报告文本。尽管为效率和清晰度服务的简约是可以理解的,但许多关于模型规范的重要决定可能未被报告,因此被读者误解。人们可以在当代文献中想到许多报告结果不确定的例子,不是因为研究人员的任何错误,而是因为读者无法掌握完整的分析细节。当代人格科学通常对不适合的复杂问题感兴趣,例如,整洁的 2 × 2 因子设计和可以用 1-2 行文本描述的方差分析。总之,善意错误和模型漏报的可能性是《华尔街日报》采用的令人信服的理由数据和分析的透明度在第2级,其需要将数据和分析(即,语法或代码)可以清楚地且精确地记录和最大可用任何有兴趣复制已发表结果或在他们自己的实验室复制该程序的研究人员。实际上,这意味着为了在期刊上发表,一篇文章需要 (a) 将用于报告研究的数据发布在永久公共存储库(例如 ICPSR、osf.io)中,或者如果它是已公开可用的数据集,提供下载和复制特定数据集所需的代码,并且 (b) 始终提供分析中使用的统计代码。当然,这些规则也有例外,在某些情况下,“最大可用”意味着如果需要,可以联系谁来请求验证数据。正当理由的示例包括无法对数据进行去标识化或匿名化(例如,GPS 位置和完整的生活故事叙述帐户)或参与者明确禁止以这种方式共享的档案数据。后一个例子在较旧的数据集(仍然为新调查提供丰富的机会)中并不少见,尽管我们预计它会越来越少,因为作为我们研究文化的一部分和资助机构(例如、NIMH 和 NIAAA)。最后,我们对研究人员表示同情,他们担心公开发布数据会导致他们被自己的劳动成果榨取。虽然我们认为这些担忧是自然的,但在很大程度上被夸大了,但有几种情况可能会特别危险,例如当有人想发表大型复杂或纵向数据集的第一篇论文时,或者从他们的论文数据中发表第一篇论文,这是他们早期职业生涯的一个窝蛋。在任何一种情况下,作者都可以理解地计划了几篇论文,并且可以从时间里毫无顾虑地完成这些提交中受益。在这些情况下,作者可以申请禁运(最长 12 个月)并解释禁运的理由。编辑将考虑禁运请求并接受、拒绝或要求修改。我们理解这些担忧,并相信禁运是一种适当的机制,可以为透明度大师服务,并为研究人员提供受保护的时间来发表其研究的主要目标,而不必担心被窃取。请注意,仅需要公开报告研究中相关/使用的数据;
4. 研究材料的透明度——与数据和分析并驾齐驱是研究过程中使用的实际材料。这些可能包括向参与者呈现的问卷或特定刺激。完全透明的研究报告还将包括报告研究所有相关方面的开放和可访问材料。与此同时,许多发表在期刊上的研究采用了完善的措施,通常受版权保护(例如,NEO-PI-3),而不是新颖的刺激或从头措施。情况并非一致,研究人员生成新问卷的情况并不少见,尤其是在创新设计(例如,密集纵向设计)的情况下。因此,日刊采用2级为Research Materials Transparency要求所有可以公开和可用的材料在提交时都是公开和可用的。尽管期刊审稿人和读者的文化和期望已经包括对新颖的特定项目的完整报告,这与本政策一致,但这强化了这种期望。在专有措施的情况下,作者应请求对 2 级要求的例外,并解释为什么不能逐字复制。我们非常希望作者在发表在期刊上的论文中努力使其研究材料最大程度地透明。
5. 设计和分析的透明度——许多关键的设计和分析决策不一定通过提供原始数据、语法/代码和材料来揭示。例如,诸如初始目标样本大小、排除的对象、如何处理异常值、达到的可靠性(例如,编码人员之间的一致程度或麦当劳 Omega 对问卷的一致程度)以及任何统计测试所达到的p值的精确水平等决策都是对评估研究很重要。同时,这些并没有统一报告。我们怀疑这主要是由于疏忽或健忘,因此可以通过明确询问定期发表在期刊上的研究类型的重要相关细节来轻松纠正。因此,我们采用一种设计和分析透明度在2级,在那里我们将要求作者检讨我们的报告标准,包括那些相关的研究。
6 & 7. 预注册——也许实现完全透明和开放的研究实践最重要的一步是在执行研究设计和分析计划之前进行注册。最纯粹的注册形式是注册报告(见下文),其中在开始研究之前对其设计和分析计划进行评估,很像联邦资助或研究生里程碑项目(即论文或论文)在美国完成。然而,正如 Benning 及其同事(2019 年)所讨论的,注册存在于一个连续统一体中,并且可以有效地纳入正在进行的数据收集、定性研究设计的研究计划中(Haven & vanGrootel,2019 年)),甚至是存档和公开的数据集(Weston 等人, 2019 年)。完美不应该成为这里的敌人,重要的是要记住,无论处于哪个阶段,注册的目标都是对设计和分析决策保持透明——它不是一个人可以如何预测他们的发现的记分卡。因此,偏离计划的方法可能而且经常发生,这是完全可以接受的,只要它们在报告中明确指出。此外,注册有助于明确划分假设生成(或探索性)研究和假设检验(即验证性)研究。我们要强调的是,在《华尔街日报》中,我们重视探索性和验证性研究,并欢迎两者的提交。事实上,我们认为这是大多数研究沿着连续统一体进行的另一个例子,本质上既不是纯粹的探索性研究,也不是证实性研究(例如,Wright, 2017)。对于注册,我们采用2 级,这意味着已注册的研究符合预期的注册规范,注册链接在提交评估之前与研究报告一起提供。鼓励适当的注册,无论它属于上述连续体的哪个位置,正如我们之前提到的,我们继续重视使用大型公共和纵向数据集的工作,这些数据集传统上不是注册的目标,但可以合并到这个框架中周到的考虑。最后,注册不是出版的必要条件。
8. 重复——可靠性是科学的基石——如果你不能在相同的表观条件下得到相同的结果,那么结果就无法解释,因为它反映了对研究的理解很差,可能是随机的变化。因此,期刊鼓励直接复制已发表作品的投稿,尤其是在本期刊中。需要注意的是,并非所有研究都值得直接复制,复制研究的重要性和贡献标准将与提交小说的标准相同。为了确保复制他们的忠实坚持原始研究的设计,而不是相关的研究结果进行评估,我们将实现一个3级的复制,并且只会将直接复制研究视为注册报告(见下一节)。
9. 注册报告——我们现在将接受注册报告供《华尔街日报》审议。这些值得单独列出一部分,因为它们的评估和实施过程不同于其他已注册的出版物。简而言之,在此轨道中寻求考虑的研究将接受两个阶段的审查(参见 https://osf.io/8mpji/wiki/home/)。在第一阶段,提交的内容应包括介绍和方法(包括计划分析)部分,足够详细以批判性地评估研究的所有重要方面。这将由相关专家进行审查,他们可能会建议按原样接受研究和计划,建议修改计划的方案,或由于设计不当或问题不重要而拒绝研究。假设研究最终被接受为这个阶段,无论是最初还是修订后,该研究可以被视为“原则上接受”。期望作者随后会忠实地执行商定的数据收集和分析程序,然后在第二阶段提交完整的项目进行评估。在第二阶段,将评估研究是否符合初步计划,如果实现,将被接受。
10. 开放科学徽章——在适当的时候,作者现在可以申请并获得开放科学徽章,以展示在他们的出版物上
这些是在期刊上采用和激励开放性和透明度的初步努力。我们的目的是选择最能服务于依赖《华尔街日报》作为传播重要发展、前沿发现和跨越假设生成和假设检验连续统一体的研究的科学家社区的标准水平。人格领域。同时,我们可能会发现,尽管我们尽了最大的努力,但经过几个季节的磨损后,披风还是剪裁不当,不符合缝制的目的。在这种情况下,我们将根据需要进行调整以校准这些程序的实施,以最好地满足为期刊的持续成功做出贡献的学者社区的需求。科学本身就是一种累积和递归的实践,并且,