2.1. Obtaining models and data for the computations

For each neutron PDB entry, the following information was collected. The minimum and maximum resolution limits of the data were obtained from the PDB file header or, if absent, from the publication associated with an entry. For entries derived from neutron data alone, the reflection-array names in the data file followed the standard naming conventions in most cases. However, for joint XN data sets the data file should contain at least two data arrays: one for the neutron data and one for the X-ray data. In this case, there is unfortunately no naming convention that clarifies which array refers to which data set. To distinguish the arrays, we computed R_work/R_free using X-ray and neutron scattering factors for both arrays, assuming that the wrong set of scattering factors leads to substantially higher values. Ligand restraints were obtained from the latest version of GeoStd (N. W. Moriarty & P. D. Adams, manuscript in preparation; https://github.com/phenix-project/geostd). In the case of uncommon ligand protonation, restraints were generated with eLBOW (Moriarty et al., 2009). Also, if the recomputed R factors did not reproduce (within 5%) the values supplied by the authors of an entry, we checked for inconsistencies.

2.2. Collecting model and data properties

The entries derived from neutron diffraction experiments have inconsistent properties. Firstly, the models can originate from refinement against neutron data alone or from joint XN refinement. Secondly, the models can contain hydrogen, deuterium or both. If hydrogen and deuterium are present, a common scenario is that exchanged structures are modeled with hydrogen at nonlabile sites and a superposition of hydrogen and deuterium at labile sites. However, it is also possible to selectively protonate certain residues, such as the methyl groups of leucine and valine residues, as reported for example in Fisher et al. (2014). Finally, the diffraction data quality, such as the completeness and the resolution, can vary significantly. Therefore, it is useful to examine some properties of neutron entries before using them to test algorithms. To do this, we collected model properties, such as the hydrogenation state (how the experimentalists chose to model H-atom sites), and data properties, such as completeness and the resolution limit. We also estimated the number of unique neutron entries. As some studies involve the investigation of different mutants, ligand soaks etc. of the same molecule, the total number of neutron models does not reflect how many unique structures have been determined. For each neutron model, we obtained the sequences for all chains. If at least one sequence per model has more than 90% sequence identity to the chain of another model, the two structures are considered to be homologous. This way, we could determine groups (`clusters') of homologous models.

2.3. Refinement of models against X-ray and neutron data alone

To test our assumption that models refined separately against X-ray or neutron data are different, we separately refined four joint XN entries against the X-ray and neutron data: PDB entries 6l46, 7az3, 4ny6 and 3x2o. For refinement against neutron data, the deposited model was used. An exception is PDB entry 7az3, a model that contained either deuterated or hydrogenated residues, and which produced R_work and R_free values of 39.0% and 41.9%, respectively, for the neutron data if used as is. As the accompanying paper described the model as being perdeuterated, we replaced all non-exchangeable H atoms with D atoms and all labile sites with H/D, which yielded recomputed R_work and R_free values of 21.8% and 22.6%, respectively, which are much closer to the reported values (R_work and R_free of 18.4% and 22.1%, respectively). The reason for focusing on PDB entry 7az3 is because it has the highest neutron data completeness for a deuterated model.

For all four entries, refinement against X-ray data was performed with the deposited model in which all D and H/D sites were converted to single H sites. The refinement strategy was similar for both data sets: several macrocycles of reciprocal-space refinement with phenix.refine (Afonine et al., 2012) that included refinement of individual coordinates, atomic displacement parameters (ADPs or B factors) and occupancies (as detailed in Afonine, 2015) and ordered solvent (water) update. As the resolution of the X-ray data was high enough, we refined the ADPs of non-H atoms anisotropically. After each round of refinement, the model was inspected with Coot (Emsley et al., 2010) to validate and interactively improve the model. In the final rounds of refinement, we optimized the weights between the data and restraint targets (stereochemistry and ADP restraints).

2.4. Comparison of models refined separately against X-ray and neutron data

The cctbx library was used to compare the models that were refined separately against X-ray and neutron data. To compare the sites of water molecules, we used the water-cluster algorithm employed in the Phenix Structure comparison tool (Moriarty et al., 2018). Water molecules were considered to be at the same site if they were within 0.5 Å. To compare the residues, we computed the rotameric states, calculated the coordinate r.m.s.d. after superposing the two models based on their main-chain atoms and identified whether a residue had alternative conformations. If a residue or a water molecule had a map–model correlation coefficient of less than 0.9 or 0.7, respectively, or if a water was modeled as an alternate conformation, it was disregarded. Also, if there was a significant negative difference map peak (less than −3 r.m.s.d.) along with weak average 2mF_obs − DF_model density (<1.7 r.m.s.d.) on atomic centers, the residue was ignored. In this way, we excluded uncertain parts of the model from the comparison. We also calculated histograms of isotropic ADPs to compare their distribution.

2.5. The new joint XN procedure in phenix.refine

In previous implementations of joint XN refinement, a single model was simultaneously refined against two data sets, thus inferring features that are common to both experiments. In the new joint XN approach, two models are refined against their respective data sets. The model refined against the X-ray data uses the shorter `X-ray' X–H distances and the model refined against the neutron data uses the longer nuclear X–H distances. The refinements are not independent, however. The neutron model benefits from the details of the typically more accurate X-ray model via the use of reference-model restraints (Headd et al., 2012). The `reference-model' approach adds a restraint to each torsion angle (involving heavy atoms) in the working model; the target value of this restraint is set to the equivalent torsion in the reference model. The residuals for the reference torsion restraints use a `top-out' function, which has the shape of a harmonic potential with an asymptotic threshold (`top-out'). Unlike the procedure described in Headd et al. (2012), the joint XN approach does not perform automated correction of rotamer outliers in the working model. Similarly, the X-ray model retrieves the orientation of unambiguously resolved H atoms from the neutron model. Consequently, both models use their respective experimental data to the fullest extent, while also incorporating the unique features of each data set. Since the completeness and resolution of neutron data are almost always poorer than those of X-ray data (Figs. 2 and 3), it is expected that the neutron model benefits most from joint refinement.

2.6. Testing the new joint XN procedure

To exercise and validate the new joint XN procedure, we performed test refinements of perturbed neutron models against X-ray and neutron data. Using perturbed models offers three advantages. Firstly, the number of joint XN entries deposited in the PDB is very small (136 models), potentially obscuring trends in the interpretation of the refinement results. By using several perturbations of the same model, we can increase the number of test structures. Secondly, it can be assumed that deposited structures have been extensively validated and refined by the depositors, so it is unlikely that automated standard refinement procedures will improve the models substantially. This makes it more difficult to draw conclusions about the new procedure. However, as a perturbed model has been moved away from this optimized state, it is expected that it will improve if the refinement procedure is effective. Finally, since gradient-driven refinement is a local optimization procedure in a highly multi-dimensional space, it may occur that two similar refinement runs (for example using identical settings but slightly different initial models) converge to two slightly different refined models (Terwilliger et al., 2007; Section 3.2.4 of Afonine et al., 2018). The difference may manifest itself as changes in crystallographic R factors of as little as a fraction of a percent or as large as 1–2%. For this reason, the evaluation of refinement protocols based upon a single or a few refinements may not be reliable, and it is preferable to base conclusions on a large number of refinements.

Using phenix.dynamics, we created 20 perturbed versions for each joint XN model, applying r.m.s.d.s of 0.5 and 0.9 Å from the initial structure, yielding a total of 40 perturbations per model. These perturbations exceed the typical coordinate errors in refined models (see, for example, Rupp, 2009 and references therein) but are within the convergence radius of crystallographic refinement (Agarwal, 1978). The perturbed models were then refined (i) against the neutron data alone and (ii) using the new joint XN procedure. For the new joint XN refinement, the starting model for the neutron data was the perturbed model, while the starting model for the X-ray data was the deposited model (where deuterium was replaced with hydrogen). This scenario replicates the situation faced by an experimentalist, who in most cases has access to a good high-resolution X-ray structure and wants to determine the neutron model.

To investigate whether the new joint XN procedure systematically improves the X-ray models as well, we performed ten rounds of refinement. The starting model for the neutron data was the deposited model (curated if applicable); the starting model for the X-ray data was the deposited model where deuterium was replaced with hydrogen.

We then compared the water sites in the neutron and X-ray models from the new joint XN refinement procedure (with perturbation). We used the same water-cluster algorithm from the Phenix Structure comparison tool as was used to compare the four separately refined models (Section 2.4; waters were considered to be equivalent if within 0.5 Å). For this analysis, no map-filtering was applied.

We performed test refinements of all joint XN models that had both X-ray and neutron data available, that could be processed successfully and for which the recomputed R factors were within 5% of the published R factors.

2.7. Treating H (D) atoms as riding or as individual atoms

In addition to testing our new joint XN refinement procedure, we also investigated whether it is better to refine H (D) atoms as individual or as riding atoms. We recently reported the new riding hydrogen procedure employed in cctbx (Liebschner et al., 2020). We apply a hybrid approach of this procedure for neutron refinement with phenix.refine: H atoms without a degree of freedom are treated as riding as described in Liebschner et al. (2020), while H atoms that possess a degree of freedom (such as O—H in serine or tyrosine) are allowed to refine freely in order to match the data. To test which approach is more advantageous, we refined 188 neutron models against their neutron data, once with the riding option and once with the individual option. We analyzed the results considering data completeness, data resolution and the H/D content of the sample.

3.1. Obtaining models and data for the computations

As of April 2023, 212 neutron models had been deposited in the PDB. Among these, 136 models are from joint XN refinement and 76 models are derived from neutron data only, corresponding to a ratio of approximately 2:1. A total of 210 models could be successfully processed with Phenix tools. The two failures were due to uncommon ligand-protonation states. For example, a ligand in PDB entry 6bq8 was labeled as 6FW, but it actually had a different charge state than that implied by its chemical name. As a consequence, the restraints library file of 6FW had no restraints for a particular H atom in this ligand. After we contacted the PDB, the entry was updated (version 3.0) and the ligand was renamed WNU. As this change was very recent, the entry is not included in this study.

Several entries have no or incomplete diffraction data. Five models have no data at all (PDB entries 1gkt, 1io5, 1lzn, 1ntp and 6rsa). These entries predate the time when structure-factor deposition became mandatory for the PDB in 2008 (Young et al., 2017). Seven joint XN entries have incomplete data arrays; the X-ray data are missing for PDB entries 5a93 and 3ins, while the neutron data are missing for PDB entries 5nfw, 5nfe, 6fjj, 6fji and 4cvj. Some of these entries are fairly recent (2019). This means that 5% of all neutron entries have no or incomplete data and cannot be validated or used for methods development.

We previously reported limitations to the annotation, deposition and validation of neutron models and data, which partly stem from a lack of community-wide accepted standards (Liebschner et al., 2018). This created challenges for processing the remaining entries. For data files, for example, labels for joint XN entries could not always be allocated automatically, the R_free arrays were missing or incomplete (i.e. at least one reflection did not have an R_free flag) or the data annotation was incorrect (intensities versus structure factors). The most prominent issues with model files were related to H (or D) atoms. Some entries have the wrong atom type (H instead of D), some models had only one atom type at exchanged sites and some structures systematically missed certain H (or D) atoms. These problems could sometimes be identified by large differences between reported and recomputed R factors. We then modified the models according to information provided in the primary citation. For example, if the model contained only H atoms but the paper described the structure as being perdeuterated, we replaced all H atoms with D atoms. This time-consuming approach helped in some instances, but we still could not reproduce the X-ray or neutron R_work or R_free, i.e. the recomputed values were 5% larger or smaller than those reported in the metadata, in 11 cases (PDB entries 2inq, 2mb5, 2vs2, 3kyy, 3kyx, 3qba, 4ar4, 5e5k, 5jpc, 5k1z and 5kwf).

3.2. Overview of some neutron model and data properties

Fig. 1 shows the annual cumulative number of neutron model depositions. The oldest neutron entry was deposited in the PDB in 1984. While the early numbers of deposited neutron models remained low (fewer than ten) until the early 2000s, depositions have been growing more rapidly since then. The increased rate of model depositions can be attributed to advanced neutron sources, new neutron macromolecular crystallography beamlines and improved methods and technologies, such as neutron image-plate detectors (Niimura et al., 1994), as well as the use of powerful time-of-flight techniques (Langan et al., 2004). We note that the majority of recent depositions come from joint XN refinements, although some models are derived from neutron data only.

Fig. 2(a) shows a histogram of the author-supplied high-resolution limit (d_min) of the neutron data. The best resolution is 1.05 Å for PDB entry 4ar3 (Cuypers et al., 2013) and the lowest resolution is 2.8 Å for PDB entry 8e1w (Tandrup et al., 2023). For all neutron entries the average high-resolution limit is 1.99 Å, while for joint XN and only neutron entries the average d_min amounts to 2.04 and 1.91 Å, respectively. In addition, we observe that the distribution of d_min for only neutron models is slightly shifted towards higher resolutions, suggesting a tendency to use the joint XN approach for lower neutron data resolutions.

Fig. 2(b) shows the relationship between the high-resolution limits of neutron and X-ray data for joint XN entries. In the majority of cases the X-ray data resolution is better than that of the neutron data. This is expected for several reasons (also discussed in Section 1). Firstly, X-ray diffraction experiments are usually carried out at cryo-temperatures, which decreases the intensity decay of the high-resolution Bragg reflections. Secondly, the samples used for neutron diffraction underwent perdeuteration or soaking, which often results in smaller, less well diffracting crystals.

Fig. 3(a) shows a histogram of the completeness of the neutron data. The worst completeness is 45.5% for PDB entry 5kwf (Golden et al., 2017) and the best completeness is 99.7% for entry 5ai2 (Cuypers et al., 2016). For all neutron entries, the average completeness is 81.8%; for joint XN entries and only neutron entries, the average completeness values are 80.9% and 83.3%, respectively. For the high-resolution limit, we observed a slight shift towards higher completeness in the distribution of only neutron models. Fig. 3(b) shows the distribution between the completeness of neutron and X-ray data for joint XN entries. In most cases, the completeness of the X-ray data surpasses that of the neutron data.

Fig. 4 illustrates how many similar structures have been determined using neutron diffraction. Based on the sequence, we assigned each neutron model to a cluster of similar structures. The x axis represents the number of models in a cluster; if the number is one, there is only one instance of the structure in the PDB. Larger numbers mean that several versions of a structure have been determined. The height of the bar denotes how many clusters exist. For example, there are 36 structures that have only one instance of a model in the PDB, 12 structures that have two instances etc. Some structures have been determined many times: there are 15 models of trypsin, 12 models of rubredoxin and 12 models of human carbonic anhydrase II. This means that the 212 neutron models in the PDB contain many homologous structures. While this finding does not immediately affect the computations reported in this work, it is worthwhile noting this property.

3.3. Comparison of four models refined separately against X-ray and neutron data

To test our hypothesis that one single model cannot simultaneously satisfy the X-ray and neutron diffraction data in the best manner, we chose four joint XN models (PDB entries 6l46, 7az3, 3x2o and 4ny6) for which we performed refinements separately. To be able to compare structural details, the neutron and X-ray data needed to be of good quality, so this was our main criteria in choosing these entries. Statistics of the deposited and re-refined models for these four examples are summarized in Table 1. The resolution of the neutron data was 1.5, 1.7, 1.5 and 1.85 Å for PDB entries 6l46, 7az3, 3x2o and 4ny6, respectively. We note that the paper describing PDB entry 6l46 (Fukuda et al., 2020) reports a resolution limit of 1.3 Å for the X-ray data for their analysis, because it `yielded better R factors and less noise'. As the deposited data set included reflections (with 99.9% completeness) up to 1.0 Å resolution, we used the full resolution range. Notably, we did not observe any abnormalities in the maps or in refinement. Another criterion for choosing these particular entries was the comparably high completeness of the neutron data, ranging from 88.1% (PDB entry 7az3) to 99.6% (PDB entry 6l46), which is markedly better than the average of 81.8% (Section 3.2). This way, the resulting models should be quite accurate and well suited for a detailed comparison.

If the model was refined against the X-ray data alone, R_work and R_free improved in all cases. For PDB entry 4ny6, the improvement amounts to more than 5%. For the model refined against neutron data alone, the R factors are slightly better than or similar to the values referenced in the PDB.

Comparing each residue one by one revealed that each X-ray model had more alternative conformations than the corresponding neutron model. The number of additional alternative conformations ranged from 17 (PDB entry 4ny6) to 34 (PDB entry 6l46). Examples of alternative conformations are shown in Fig. 5. The first example shows Ile52 in PDB entry 6l46. The electron density clearly (Fig. 5a) supports the two alternative conformations, while the nuclear scattering length density only shows one conformation (Fig. 5b). Another example is Met266 in PDB entry 6l46. Methionine contains an S atom, which is a strong scatterer of X-rays [f(0) = 16 e⁻] but a weak scatterer of neutrons (b = 2.8 fm). The electron density supports the two modeled conformations (Fig. 5c). The difference density peaks around the S and C atoms of Met266 may be indicative of a third conformation or of radiation damage. In contrast, there are no peaks in the nuclear scattering length density map (Fig. 5d) that would justify modeling the second conformation present in the X-ray model. A third example is Ser160 in PDB entry 3x2o, the double conformation of which is clear in the X-ray model (Fig. 5e), while only one conformation appears in the neutron model (Fig. 5f). Asn38 in PDB entry 3x2o is an example for which the nuclear scattering length density map may be more informative. As nitrogen and oxygen have a similar number of electrons at zero scattering angle, they are often difficult to distinguish by the density map alone. The orientation of Asn and Gln side chains is thus often ambiguous, so that clashes and hydrogen-bond interactions with neighboring entities have to be taken into account. However, if the H atoms of the Asn head group are replaced by D atoms, the NH2 (ND2) group is a much stronger scatterer than the O atom, so that the side-chain orientation can be determined from the nuclear scattering length density map. It is thus conceivable that the orientations of Asn and Gln residues could be determined from the neutron data instead of the X-ray data. While this is not currently implemented in our new joint XN refinement procedure, it represents a future avenue of improvement.

A remarkable structural difference involved the number of water molecules. All X-ray models had more water molecules than the neutron models. The number of water molecules in the deposited structures was generally between the two values. For example, in the case of PDB entry 7az3, the PDB model had 203 waters, the X-ray model contained 230 waters and the neutron model had 156 waters. We also note that the coordinates of a significant part of the water molecules were not conserved. We determined which water molecules were within 0.5 Å in both models, and those that did not have a corresponding water were considered to be `lone' waters. The percentage of lone waters was generally higher in the X-ray models, which is consistent with the observation that these models generally had more water molecules. This percentage varied between 27% (PDB entry 6l46) and 81% (PDB entry 4ny6). However, even the neutron models had many lone waters, i.e. between 17% (PDB entry 6l46) and 58% (PDB entry 4ny6). We note some correlation of the percentage of lone waters with the data-collection temperature. PDB entry 6l46 has the lowest percentage of lone waters and both the neutron and X-ray data were collected at 100 K. PDB entries 7az3 and 3x2o have 43% and 49% lone waters in the X-ray model, respectively, and the data were collected at room temperature in both cases.

For PDB entry 4ny6, the percentage of lone waters is the highest (81% for X-ray and 58% for neutron). While both the neutron and X-ray data sets were obtained at room temperature, they were collected from different crystals. The neutron and X-ray data stem from a selectively deuterated crystal (valine residues and methyl groups of leucine) and a perdeuterated crystal, respectively. The authors state

(Fisher et al., 2014). While H and D are equivalent with either diffraction method, our analysis shows that the water structure varies between the two data sets that were collected from different crystals.

As an example of the differing water molecules between X-ray and neutron models, Fig. 6 shows the region between two symmetry-related molecules of PDB entry 4ny6. The four DOD molecules have corresponding HOH O atoms of the X-ray model in their vicinity. However, even if the O atoms are within 0.5 Å, their coordinates differ visibly. Further, there are several water molecules in the X-ray model that do not have an equivalent in the neutron model. Notably, there are no 2mF_obs − DF_model peaks indicating the presence of the DOD molecules near the position of the X-ray waters.

We also analyzed the isotropic ADPs (B_iso) of the non-H (non-D) atoms in the models. The minimum, maximum and mean values for protein residues and water molecules are listed in Table 1. We note that the average B_iso of protein residues is lower in most neutron models, with the exception being PDB entry 7az3. This entry is the only perdeuterated model among the four examples, but it is unclear whether this is the reason for the behavior of the ADPs. We also made ADP histograms to investigate their distribution. The histograms are comparable for the X-ray and neutron models of PDB entries 6l46 and 4ny6. The histograms for PDB entries 3x2o and 7az3 are shown in Fig. 7. In both cases, the distributions are markedly different. For PDB entry 3x2o the histogram of ADPs is shifted to lower values, while for PDB entry 7az3 it is shifted to larger values. It is obvious that one single model cannot simultaneously satisfy both respective ADP distributions.

Finally, we computed crystallographic R factors from the X-ray data and the neutron models in which all D atoms were replaced with H atoms (water H/D atoms were removed). These R factors are significantly different from those obtained from refining a model against X-ray data. For example, in the case of PDB entry 4ny6, the recomputed R_work and R_free are 28.7% and 30.8%, respectively, and are much worse than those from the re-refined X-ray model (11.1% and 13.1%, respectively). This means that a model that fits the neutron data best will not do the same with the X-ray data. We note that the reverse computation, i.e. calculating R factors for the X-ray model against the neutron model, will not be very informative unless the protonation states of the neutron model are replicated. Transferring the protonation states may be possible for residues and waters present in both models, but will prove challenging for entities that can only be found in the X-ray model. If the protonation is incorrect, the difference in R factor will not reflect model differences, but issues with H/D atoms.

Overall, this comparison showed that significant differences exist between models refined separately against neutron and X-ray data, with the largest difference being in the water structure and isotropic ADPs.

3.4. Refining H (D) as riding or as individual atoms

We investigated whether it is more advantageous to refine H/D atoms against neutron diffraction data as riding or as individual atoms. Refinements were successfully completed for 188 neutron models. We analyzed the results considering the data completeness, the high-resolution limit of the neutron data and the H/D content of the sample. Fig. 8 shows R_work, R_free and R_gap (R_free − R_work) for refinements using either the riding or individual option for H/D atoms. If the distribution was dependent on one of the three properties considered (completeness, resolution or H/D content), we colored the scatter points accordingly.

Fig. 8(a) shows the crystallographic R factor R_work. Almost all of the points of the scatter plot are below the bisector, meaning that refinement leads to a lower R_work if H/D atoms are refined individually. This behavior is unsurprising because, generally, increasing the number of model parameters typically decreases R_work. The points in Fig. 8(a) are colored according to the hydrogenation state. We observe that points representing models with a majority of D atoms (perdeuterated models) are often furthest from the bisector, i.e. they have a lower R_work. We did not observe a dependence on completeness or on the resolution limit.

Fig. 8(b) shows the crystallographic R factor R_free. Most points of the scatter plot are above the bisector, meaning that refinement leads to slightly higher R_free if H/D atoms are refined individually. As the points are generally close to the bisector, we also plotted a histogram of R_free(individual) − R_free(riding) (see the inset in Fig. 8b). The distribution is shifted to the right, and thus shows that the strategy of refining H/D atoms individually results in an increased R_free in many cases. The points in Fig. 8(b) are colored according to the high-resolution limit. We observe that neutron data with a lower resolution limit d_min result in a higher R_free. We did not observe a dependence on completeness or on the hydrogenation state.

Fig. 8(c) shows the difference between the R factors, R_gap. All points except one are above the bisector, which means that R_gap is lower if the H/D atoms of the model are refined with the riding model. This is in line with the results from Figs. 8(a) and 8(b), where R_free was lower and R_work was higher when H/D atoms were refined as riding. The points in Fig. 8(c) are colored according to the high-resolution limit. The distribution of the points depends on the resolution of the data: data with better resolution (better than 1.8 Å) yield a lower R_gap than data with poorer resolution (worse than 2.2 Å). We also observed a tendency of refinements to result in a lower R_gap (not shown) if data completeness is high, but the dependence was less apparent than for the high-resolution limit. We did not observe a dependence on the hydrogenation state.

Together, the results shown in Fig. 8 suggest that refining individual parameters of H/D atoms leads to overfitting unless the high-resolution limit of the data is high. As a general recommendation, we suggest that if the resolution of the neutron data is better than 2 Å, both strategies should be tried to find out whether H/D atoms in the model should be refined as riding or individually. At low resolution and low completeness, it can be assumed that the riding model is adequate.

We performed additional refinements, this time also optimizing the weights between the data target and the stereochemistry and ADP restraints, respectively (Afonine et al., 2011). This weight-optimization procedure tries to find the weights that result in the lowest R_free factors by performing a grid search over an array of weight candidates. Fig. 9(a) shows a comparison of R_gap with and without weight optimization for the riding model. We observe two things. Firstly, the distribution is shifted towards lower values of R_gap compared with Fig. 8(c), meaning that weight optimization can decrease the difference between R_work and R_free. Secondly, the behavior of the distribution depends on the resolution of the neutron data. For high-resolution data, R_gap is similar with or without using the weight-optimization procedure. At lower resolutions, the weight optimization clearly leads to a lower R_gap. Therefore, it seems advantageous to optimize the weights if the neutron data resolution is low. Fig. 9(b) shows R_gap computed for refinements with weight optimization using either the riding or the individual option. Again, the distribution is shifted towards lower values of R_gap compared with Fig. 8(c). Furthermore, the distribution is more centered around the bisector for data with poorer resolution limits (worse than 1.8 Å) than for high-resolution data. If the weights are optimized, there is less difference between the R_gap values for riding versus individual H/D refinements, except for neutron data with high resolution. Altogether, the results in Fig. 9 suggest that weight optimization mitigates the effects of overfitting at lower neutron data resolution limits.

3.5. Testing the new joint XN procedure: neutron models

Of the 136 joint XN entries, 120 had both X-ray and neutron data available, could be processed successfully and had recomputed R factors within 5% of the published R factors. For each of these 120 models, we created 40 perturbations (20 at a threshold of 0.5 Å and 20 at a threshold of 0.9 Å). Each perturbed model was refined against neutron data alone or used as the neutron starting model for a new joint XN procedure. This gave 2400 computations per threshold (2400 = 20 × 120). 2393 and 2390 refinements ran to completion at thresholds of 0.5 and 0.9 Å, respectively. Fig. 10 shows R_work, R_free and R_gap from refinements against neutron data only against those from new joint XN refinements, for models perturbed with a threshold of 0.5 Å. The points are colored according to the high-resolution limit of the neutron data.

Fig. 10(a) shows the crystallographic R factor R_work. Most points of the scatter plot are above the bisector, meaning that R_work tends to be lower if the starting model is refined against the neutron data only. We observe that some data points in the lower R_work region, mostly those corresponding to models determined at better than 1.8 Å resolution (blue circles), are located close to or on the bisector, i.e. R_work is similar for both refinement approaches.

Fig. 10(b) shows the crystallographic R factor R_free. Many points of the scatter plot are below the bisector, i.e. R_free tends to be lower for the new joint XN refinements. We again observe a dependence of the distribution on the high-resolution limit. R factors from high-resolution data (better than 1.8 Å, blue circles) occupy the regions of lower R_free and are arranged close to the bisector, while those from lower resolution data (worse than 2.2 Å, red circles) are in the regions of higher R_free and display more scatter around the bisector.

Fig. 10(c) shows the difference between the R factors, R_gap. The majority of points are below the bisector, indicating that new joint XN refinement minimizes R_gap. As large differences between R_work and R_free are typically associated with overfitting, it can be concluded that the new joint XN approach reduces the overfitting of models compared with refinement against neutron data alone. The effect is less pronounced for high-resolution data (blue circles), i.e. the new joint refinement or refinement against neutron data leads to similar R_gap.

Fig. 11 shows R_work, R_free and R_gap after refining models perturbed with a threshold of 0.9 Å. The shape of the distribution and the dependence on the high-resolution limit of the neutron data is very similar to that of the R factors obtained for the 0.5 Å perturbations. The conclusions to be drawn are therefore the same: the joint XN approach reduces overfitting, especially when the neutron diffraction data have medium-to-low resolution.

We also investigated whether the new joint XN procedure systematically improves the X-ray models. These computations were carried out with the deposited models (curated if applicable) where D was replaced with H for the X-ray starting model. Our assumption was that the deposited models most likely under-refined the X-ray data, so the new joint XN approach should naturally lead to improvements. Figs. 12(a) and 12(b) show the crystallographic R factors R_work and R_free recomputed from the deposited models and after the new joint XN refinement, respectively. Most points of the scatter plot are below the bisector, meaning that the R factors are lower if the model is refined against the X-ray data only.

As analysis of the individually refined models showed that the most significant difference concerned the sites of water molecules (Section 3.3), we compared the coordinates of waters in the neutron and X-ray models from the new joint XN refinement. Fig. 13 shows a histogram of the percentage of lone waters (from the refinement of perturbed neutron models), i.e. water molecules in one model that do not have an equivalent in the other model, for the two perturbation levels (0.9 Å in Fig. 13a and 0.5 Å in Fig. 13b). At the 0.9 Å perturbation level the histogram is centered on large percentage values, i.e. X-ray models have 70–100% lone waters, while neutron models have 70–90% lone waters. At smaller perturbations the percentage is centered between 60% and 80% for either model. This means that a significant number of water molecules do not occupy the same sites in the two models. Our new joint XN refinement approach is therefore justified as it allows the two models to have different water structures.