ProCal Uncertainty Rounding Rules

Modified on Fri, 17 Mar 2023 at 02:28 PM

There are multiple interpretations of the GUM with regard to rounding, with the guidance in the GUM (https://www.bipm.org/en/committees/jc/jcgm/publications) stating :

7.2.6 The numerical values of the estimate y and its standard uncertainty uc(y) or expanded uncertainty U should not be given with an excessive number of digits. It usually suffices to quote uc(y) and U [as well as the standard uncertainties u(xi) of the input estimates xi] to at most two significant digits, although in some cases it may be necessary to retain additional digits to avoid round‑off errors in subsequent calculations.

In reporting final results, it may sometimes be appropriate to round uncertainties up rather than to the nearest digit. For example, uc(y) = 10,47 mΩ might be rounded up to 11 mΩ. However, common sense should prevail and a value such as u(xi) = 28,05 kHz should be rounded down to 28 kHz. Output and input estimates should be rounded to be consistent with their uncertainties; for example, if y = 10,057 62 Ω with uc(y) = 27 mΩ, y should be rounded to 10,058 Ω. Correlation coefficients should be given with three‑digit accuracy if their absolute values are near unity.

ProCal as of V6.8.91 has been updated so that the below rules are always followed.

Intermediary calculations will be calculated with no rounding to a high precision.
Expanded Uncertainties will always be rounded UP to the least significant digit of the resolution. I.e. for a resolution of 0.001V, and an uncertainty of 0.00101V, the reported uncertainty will be rounded up to 0.002V (2uV)
Uncertainties will be reported and rounded up to 2 Significant Places, based on the resolution and the calculated uncertainty. I.e., an uncertainty of 587.2 kOhms will be rounded UP to 590 kOhms

This is taking the most risk-averse approach to reporting and calculating the uncertainty. Rounding UP is always used in any event where the reporting resolution causes the expanded uncertainty to be rounded for display.