3.1. Metrological Performance under Steady Flow Conditions
The steady state flow tests were used for two main purposes. On the one hand, it was necessary to verify that the meters under analysis met the metrological requirements for new meters defined under the ISO 4064-1:2014 [
11]. On the other hand, it was necessary to obtain a reference error curve that could be used for comparison purposes with other tests performed.
Concerning to the first objective,
Figure 6 and
Figure 7 show the detailed results of the error tests per meter through boxplots. These graphs present the error distribution of the various repetitions conducted at each flow rate under steady flow conditions.
Section 5 in the Supplementary Material S–B offers numerical details about the average error and the standard deviation of the tests. Like any traditional mechanical water meter, the error of the solid-state water meters under examination should be within the maximum permissible error of ±2% for flow rates greater than Q
2 and an accuracy class 2.
Almost all meters tested met the ISO 4064-1:2014 [
11] metrological requirements for the flow rates considered. M5 and M7 type meters were exceptions to the previous statement. As explained before, the only M5 unit available did not measure any flow, although the display of the meter and the meter itself seemed to be in proper working order. M7 meters showed an average error of approximately −8.5% at the highest flow rate tested of 5000 L/h, which for these meters corresponds to the overload flow rate (Q
4). This malfunctioning at high flows has also been detected by the authors in other brands of ultrasonic meters and actual figures are detailed in the technical specifications of some brands. However, this behavior only appears under the presence of flow rates larger than 1.25·Q
4 and disappears once the flow rate decreases below that threshold. Therefore, this malfunctioning is caused by limitations of the algorithms used to calculate the flow and not by a defective component of the meter. Nevertheless, these results at high flows confirm the importance of testing the meters over a wide range of flow rates before they are put into operation.
With regards to the single-jet mechanical water meters, M6 type has a slight tendency to over register water consumption as the flow increases (
Figure 6). Meters units M0011 and M0012 exceeded the maximum permissible errors for flows greater than 1000 L/h. In addition, M0011 showed poor performance at a flow rate of 50 L/h, with an average error (five tests conducted) of −12.3%. This loss of accuracy is common in mechanical meters that have been in operation for several years (
Table 2). In contrast,
Figure 7 shows that the overall repeatability in a steady state test of a brand-new single jet mechanical water meter (M9 type) is satisfactory.
The steady state tests were employed not only to verify that the actual errors of the meters were within the maximum permissible errors allowed by the ISO 4064-1:2014 [
11] standard but also to measure their repeatability. In this regard, the ISO standard establishes that the standard deviation of the errors at a given flow should not exceed one-third of the maximum permissible error, which for flows larger than Q
2 is 2%. This means that the standard deviation of the errors should be smaller than 0.66%. However, provided that the reading resolution of the solid-state meters is limited to 1 L, the overall uncertainty of the tests is close to 1%. This means that even under steady flow conditions, a standard deviation well below the overall uncertainty cannot be expected from the tests. Considering this limitation and the results presented, it cannot be stated that the meters under analysis do not meet the repeatability requirements established in the ISO standard.
Figure 8 consolidates the repeatability of the measuring errors obtained by technology, nominal diameter, and flow rate. Each box-whisker plot is built with the standard deviations that the associated meters have shown in each test. For example, the box-whisker plot corresponding to DN20 meters with EMF technology and a flow rate of 2000 L/h is composed of five data or, in other words, five standard deviations corresponding to meters M0021, M0032, M0033, M0034, and M0035 that were tested 7, 3, 3, 3, and 3 times (
Table S1 in Section 2 in Supplementary Material S–A), respectively, under steady flow conditions at 2000 L/h. The lower standard deviation achieved by mechanical meters can be explained by the better scale resolution of these meters.
For this reason, due to the poor volume resolution that is readable in the display of solid-state meters without interacting with them, it seems evident that the ISO standard needs a significant upgrade. Improving the scale resolution of the meters, available without any interaction with them, is the only option to conduct proper accuracy tests in a laboratory that ensure that meters will operate during the tests exactly as they will in the field. Currently, the ISO standard and the OIML recommendation specify that all meters should incorporate a verification device that “provides means for visual, nonambiguous verification testing and calibration”. The problem is that this requirement on the verification scale interval of the display of a water meter is only met when the test mode is activated. Consequently, this constraint related to the allowable resolution has not been properly interpreted by manufacturers, and currently, published standards do not require that the verification scale interval is permanently readable. This way, according to the ISO 4064-1:2014 [
11] or the OIML R49-1:2013 [
24], the required resolution of the verification scale of a meter having a Q
3 of 2500 L/h, and a metrological class R160, is 0.0586 L. If the metrological class changes to R250 or R400, the required resolution decreases to 0.0375 and 0.0234 L, respectively. In all static meters tested, these resolutions can only be achieved by activating the test mode; otherwise, the scale resolution is 1 L. Unfortunately, once the test mode is activated there is no means of guaranteeing that the meter will have the exact same performance as with the test mode deactivated as this modes changes the sampling frequency and other operating conditions of the meter.
As expected, the error curve of solid-state water meters is relatively more uniform (flat) compared to the error curve of a single-jet mechanical water meter, which suffers from more oscillations throughout the measuring range.
Figure 9 shows by means of a box-whisker plot, the error distribution at different flow rates of all units tested.
Figure 9 does not include the results from tests at 50 L/h of the unit M0011 (M6 meter type) and at 5000 L/h of the units M0016 and M0017 (M7 meter type), with an associated error that exceeded ±2.5% of the reference volume, in order not to bias the results. An assessment of the variability of the error through the measuring range can be easily analyzed by the interquartile range amplitude of the box-whisker diagram.
Figure 9 also shows the ability of the manufacturers to produce meters with the same performance. Surprisingly, solid-state meters do not show any significant improvement in this respect to brand-new mechanical meters [
26], especially when compared to oscillating piston meters. The tests conducted showed that meters of the same manufacturer can have an average measuring error throughout the tested range of more than 1.5%. Higher variabilities in performance indicate lower control over the production processes. This implies that water utilities need to implement stricter quality control procedures on the meters received from that manufacturer to guarantee that there are no defective units in the inspected lot. This is the case of M1 and M3 type meters.
Figure 9 also shows that all solid-state meters, except M7 type at Q
4, can easily maintain the errors within the maximum permissible error of 2% under steady state conditions.
The single DN15 electromagnetic meter unit under analysis, M4 type, showed a decline in repeatability at high flows (
Figure 10). This behavior was not observed in the DN20 units by the same manufacturer, M10 meter type, which presented a more stable performance throughout the flow rate range. In any case, the repeatability of this technology during the steady state tests was better than ultrasonic meters.
The single-jet meters removed from the field, M6 type, presented the expected variability in the performance of meters that have been in operation during some time (
Table S4 in
Section 5 in Supplementary Material S–B). Contrarily, the M9 meter type, which corresponds to a brand-new water meter, exhibits an extremely low variability of the error at each flow (thanks to a volume reading resolution of 0.1 L), with some oscillations throughout the flow rate range.
Figure 11 presents the consolidated variability of the errors obtained by technology and the flow rate. As expected, ultrasonic meters showed a uniform behavior throughout the measuring range, and the average error at each flow rate slightly oscillates around the average value. The error distribution of the electromagnetic meters, M4, and M10 types show a difference in behavior between the two, which essentially are the same meter of different diameters.
3.2. Metrological Performance under Intermittent Flow Conditions
Domestic water demand is exceptionally heterogeneous, and flow rate and duration of water consumption events are extremely scattered [
14,
15,
16,
17,
18]. From a duration perspective, a shower event is not comparable with shorter uses, like filling a glass of water. Additionally, the flow rate of a leak is much lower than the consumption flow occurring when several water appliances are used simultaneously. Each water end-use has its own independent characteristics, which make the modelling of residential water demands a complex topic. However, some authors have proposed a simplification of all this casuistic by modelling water consumption events as a series of pulses of a given duration and a flow rate that are distributed through time, both of these parameters (duration and flow rate) being described by probabilistic functions that can be specific to a water end-use and individual user [
15,
27,
28,
29,
30,
31,
32]. Therefore, from a standardization point of view, employing complex consumption profiles to conduct the tests is not an option, as the purpose of this experiment is to design and conduct a test programme that provides repeatable results and can be reproduced by an independent third party. The test programme used intended to limit the intermittency and variability of flow. In the case of flow intermittency, the cyclic periodicity of the consumption pulses was set to 2, 5, 10, and 20 s. In the case of flow rate, and provided that the typical consumption flow of a domestic appliance is between 200 and 2000 L/h, the test flow rates were primarily chosen in this interval.
As it has been already mentioned, sampling of the flow rate signal is a common technique to all solid-state meters to extend their battery life. The purpose of conducting metrological tests under intermittent flow conditions is to establish if the signal sampling has any effect in the measuring error of the meters. Some meters feature a fixed sampling rate, which typically is in the order of 5–7 s. Other meters are designed with a variable sampling frequency depending on the presence and magnitude of the flow rate. The algorithms that change the sampling frequency are confidential, and no details have been provided by the manufacturers.
Consequently, to obtain more realistic results, it was not acceptable to notify the meter that it was subject to test by interacting with it or activating the test mode. Therefore, as it has already been said, all meters were tested in the same conditions as they would have been in the field.
The analysis of the results of the experiments described in this section focuses on three main issues: (1) differences in metrological performance of the meters when subject to steady and intermittent flow conditions; (2) impact of cyclic periodicity and magnitude of intermittent flows in the measuring errors; (3) potential biases caused by intermittent flows that could favor one of the parties.
Figure 12 and
Figure 13 describe through boxplots the error distribution of the tests performed under intermittent flow conditions at different flows by meter type. The error distribution obtained per meter can be found in
Section 6 in Supplementary Material S–B. It is important to highlight that these charts compile the raw results of all the test types under intermittent flow conditions defined in
Section 2.3 or, in other words, the error obtained in each repetition of a test conducted, not the average error.
The results show that the error magnitude significantly increased when compared to steady state conditions. It was not unusual to obtain a measuring error of ±20% (the percentage of results in which the error of the meters tested was greater than ±5% for the various test types considered is detailed in
Section 7 of the Supplementary Material S–B). This statement applies to a greater or lesser extent to all meter types. However, ultrasonic meters are more affected than the electromagnetic meters under examination. This is mainly due to the fact that signal sampling frequency is higher for electromagnetic meters (1 Hz or more) than for ultrasonic meters (0.2 Hz or less). Consequently, electromagnetic meters are more prepared for accurately measuring short duration consumptions events like the ones found in households.
Therefore, all meter types presented a significant difference in performance between steady and intermittent flow conditions. As will be shown later in the analysis, this difference is affected by the cyclic periodicity of the flow, the duration of a consumption event being more significant than the test flow. Although the results obtained must be contextualized, since the proposed tests magnify the potential biases associated with short consumptions events, it is a fact that the operating conditions in the field are continuously changing depending on the consumption profile. In the field, the error measuring a consumption event may be positive, and the following consumption event may be measured with a negative error. Therefore, the concern of water service managers is whether the errors of various signs that happen over time compensate each other in the long run. A more detailed analysis of the T6 and T7 tests (variable and intermittent flow conditions) was carried out in
Section 8 of the Supplementary Material S–B to address this concern.
In the case of solid-state water meters, an increase in error dispersion was also observed. As explained in
Section 2.3, the test volume of the tests under intermittent flow conditions is relatively large, and the expanded uncertainty of the test does not exceed 1%. Moreover, the errors obtained under steady flow conditions show a dispersion that meets the requirements of the ISO standard (1/3 of the maximum permissible error). Thus, the large dispersion of the errors found is strictly related to the internal algorithms and signal sampling periodicity. In addition, differences in performance between manufacturers and technologies can be easily identified, although a larger sample would have to be analyzed in the future to draw well founded conclusions. For example, the M7 and M2 meter types, both ultrasonic meters of different diameter (DN20 and DN15, respectively) from manufacturer B5, show more significant errors when the cyclic periodicity of the intermittent flow is set to 2 s (test type T2). Contrarily, the M1, M3, and M8 meters, which are also ultrasonic meters from three different manufacturers, have a more uniform response to different durations of intermittent consumption events. They even seem to achieve a worse metrological performance when the cyclic periodicity is set to 5 s. On the other hand, the electromagnetic meters tested show a better performance than the ultrasonic meters under all types of intermittency. Their average metrological response to the different intermittent flow tests is very similar to the one found under steady flow conditions. As explained above, this is mainly due to the fact that the signal sampling frequency is higher than 1 Hz.
For comparison, the single-jet mechanical water meters in the sample are still remarkably repetitive, but there is a strong tendency to over register as the duration of the consumption event decreases. Nevertheless, this behavior tends to diminish as the flow rate increases. The over registration of the meter is caused by the rotational inertia of the impeller, which keeps turning for a period of time after the consumption has finished.
Despite the fact that the reference volume in each test was sufficient to reduce the uncertainty to less than 1%, a number of repetitions showed remarkably large errors.
Table 4 describes the tests in which the errors obtained were above 50%. In such cases and considering that for steady state conditions, the errors of the meters were within the maximum permissible errors of the standard, it is considered that the meter presented an abnormal performance. Therefore, these results were excluded from further statistical analysis in order not to distort the boxplots and the conclusions obtained. It should be noted that the volume passed during the intermittent flow test is, as described in
Section 2.3, equal to or larger than the volumes used in steady state tests. Hence, these results can only be interpreted as anomalous performances of the meters that need further investigation and more detailed analysis. In addition, it should be clarified that not all meters described in the sample could be tested under intermittent flow conditions. The tests and number of repetitions conducted on each meter are specified in
Section 2 of the Supplementary Material S–A.
Figure 14 allows for a comparison between technologies in terms of repeatability and error bias, where the standard desviation and coefficient of skewness corresponding to error distribution under intermittent flow conditions are represented through histograms. It can be observed that errors of ultrasonic meters are more dispersed than for the electromagnetic units tested, especially meters belonging to M2 type. In turn, mechanical meters are by far the ones presenting the best repeatability, with the standard deviation that, in most cases, is less than 1%. Regarding the coefficient of skewness, it is symmetrically distributed around the zero value. Thus, the error distribution associated with a meter tested according to a type of test and a given flow or flow profile could show a positive skewness (i.e., the mean is greater than the median). However, the error distribution associated with another meter of the same type or even the same meter subject to another type of test could show a negative skewness. Therefore, this evidence that the internal algorithms of the static meters tested do not intentionally exploit the errors in any direction or cause a clear bias in the errors distribution.
To conclude this section, a detailed comparison of the results obtained under steady and intermittent flow conditions was conducted. To facilitate the analysis, tests results were grouped into two flow rate ranges: (1) the lower range comprises average flow rates between 200 and 500 L/h; (2) the upper range includes the errors obtained between 700 and 2000 L/h.
Table 5 and
Table 6 describe per meter unit the mean error and standard deviation associated with the tests performed under steady state (S) and intermittent (I) flow conditions for the two flow rate ranges considered, respectively. These tables also show the difference in mean error between the two flow regimes. Additionally, the table provides the average mean error and the corresponding standard deviation per meter type.
In the case of single-jet mechanical water meters there is a tendency towards over registration, significantly more pronounced at medium flows (200–500 L/h). This can also be observed in
Figure 12 and
Figure 13. However, in the same average flow rate range, the difference between steady and intermittent flow conditions for solid-state water meters reaches a maximum of +2.2% for M7 type meters. Compared to mechanical meters, this difference increases to +4.3% and +10.0% for M6 and M9 type meters, respectively. When conducting the same analysis in the upper flow rate range, from 700 to 2000 L/h, these differences are significantly reduced: for solid-state meters this parameter is always less than ±2.2% and for mechanical meters the maximum error difference is found for M9 meter type, reaching a value of +3.1%.
In summary, the error values obtained in the intermittent flow tests for solid-state water meters are significantly higher than those of single-jet mechanical water meters. However, on average, the behavior of the single-jet mechanical water meters is more deficient due to the appearance of positive bias, despite being more repetitive. In addition, solid-state water meters have frequently shown a null performance (errors reaching ± 100%) in the test under extreme conditions (cyclic period of 2 s) and the hypothesis that the errors may cancel out in the medium-long term cannot be rejected from the analysis conducted.
Figure 15 presents a series of boxplots charts that summarize the results obtained per meter type. In line with what has been previously stated, solid-state water meters show more dispersed behavior than single-jet mechanical water meters. Apart from the magnitude of this dispersion, it is important to note that the error distribution of the tests performed under intermittent flow includes the 0% error. Therefore, the exploitation of errors or any potential bias cannot be statistically confirmed with the amount of testing performed. Consequently, due to the lack of repeatability of the water meters, it would be necessary to test a larger sample and design a more detailed testing programme that includes specific assays to verify the accuracy of the meters under intermittent working conditions. In order to protect both the users and the water utilities, these test types will need to be included in the ISO 4064 and OIML R49 meter approval test programme, which should also keep records of the firmware version used by the meter.