From 1e2ec34a57e65a4a34e719509b4ea0ddd9ee3aca Mon Sep 17 00:00:00 2001 From: Raymond Hettinger Date: Sun, 19 Feb 2023 10:51:03 -0600 Subject: [PATCH 01/10] Example is more intelligible without the leading zeros --- Doc/tutorial/floatingpoint.rst | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/Doc/tutorial/floatingpoint.rst b/Doc/tutorial/floatingpoint.rst index cedade6e336608..31d7327dea5242 100644 --- a/Doc/tutorial/floatingpoint.rst +++ b/Doc/tutorial/floatingpoint.rst @@ -12,9 +12,9 @@ Floating Point Arithmetic: Issues and Limitations Floating-point numbers are represented in computer hardware as base 2 (binary) -fractions. For example, the **decimal** fraction ``0.125`` -has value 1/10 + 2/100 + 5/1000, and in the same way the **binary** fraction ``0.001`` -has value 0/2 + 0/4 + 1/8. These two fractions have identical values, the only +fractions. For example, the **decimal** fraction ``0.625`` +has value 6/10 + 2/100 + 5/1000, and in the same way the **binary** fraction ``0.101`` +has value 1/2 + 0/4 + 1/8. These two fractions have identical values, the only real difference being that the first is written in base 10 fractional notation, and the second in base 2. From 082d7a84335cd635bb6a8ac218f9f3c413399dfa Mon Sep 17 00:00:00 2001 From: Raymond Hettinger Date: Sun, 19 Feb 2023 10:55:26 -0600 Subject: [PATCH 02/10] Leading zeros are easier to read and match the surrounding text. --- Doc/tutorial/floatingpoint.rst | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/Doc/tutorial/floatingpoint.rst b/Doc/tutorial/floatingpoint.rst index 31d7327dea5242..c447b58ee998f6 100644 --- a/Doc/tutorial/floatingpoint.rst +++ b/Doc/tutorial/floatingpoint.rst @@ -108,21 +108,21 @@ simply rounding the *display* of the true machine value. One illusion may beget another. For example, since 0.1 is not exactly 1/10, summing three values of 0.1 may not yield exactly 0.3, either:: - >>> .1 + .1 + .1 == .3 + >>> 0.1 + 0.1 + 0.1 == 0.3 False Also, since the 0.1 cannot get any closer to the exact value of 1/10 and 0.3 cannot get any closer to the exact value of 3/10, then pre-rounding with :func:`round` function cannot help:: - >>> round(.1, 1) + round(.1, 1) + round(.1, 1) == round(.3, 1) + >>> round(0.1, 1) + round(0.1, 1) + round(0.1, 1) == round(0.3, 1) False Though the numbers cannot be made closer to their intended exact values, the :func:`round` function can be useful for post-rounding so that results with inexact values become comparable to one another:: - >>> round(.1 + .1 + .1, 10) == round(.3, 10) + >>> round(0.1 + 0.1 + 0.1, 10) == round(0.3, 10) True Binary floating-point arithmetic holds many surprises like this. The problem From 2ff520d94823f5bb690785e641dfb4fe68ab3132 Mon Sep 17 00:00:00 2001 From: Raymond Hettinger Date: Sun, 19 Feb 2023 11:34:40 -0600 Subject: [PATCH 03/10] Compare sum() to straight addition and to math.fsum() --- Doc/tutorial/floatingpoint.rst | 29 +++++++++++++++++++++++------ 1 file changed, 23 insertions(+), 6 deletions(-) diff --git a/Doc/tutorial/floatingpoint.rst b/Doc/tutorial/floatingpoint.rst index c447b58ee998f6..e1e53133ab7e3e 100644 --- a/Doc/tutorial/floatingpoint.rst +++ b/Doc/tutorial/floatingpoint.rst @@ -186,17 +186,34 @@ Since the representation is exact, it is useful for reliably porting values across different versions of Python (platform independence) and exchanging data with other languages that support the same format (such as Java and C99). -Another helpful tool is the :func:`math.fsum` function which helps mitigate -loss-of-precision during summation. It tracks "lost digits" as values are -added onto a running total. That can make a difference in overall accuracy -so that the errors do not accumulate to the point where they affect the -final total: +Another helpful tool is the :func:`sum` function which helps mitigate +loss-of-precision during summation. It uses extended precision for +intermediate rounding steps as values are added onto a running total. +That can make a difference in overall accuracy so that the errors do not +accumulate to the point where they affect the final total:: >>> 0.1 + 0.1 + 0.1 + 0.1 + 0.1 + 0.1 + 0.1 + 0.1 + 0.1 + 0.1 == 1.0 False - >>> math.fsum([0.1] * 10) == 1.0 + >>> sum([0.1] * 10) == 1.0 True +The :func:`math.fsum()` goes further and tracks all of the "lost digits" +as values are added onto a running total so that the result has only a +single rounding. This is slower than :func:`sum` but will be more +accurate in uncommon cases where large magnitude inputs mostly cancel +each other out leaving a final sum near zero:: + + >>> arr = [-125546.2071591587, 0.04336622547760194, 18741774.60535662, + ... 57.207123035704896, 454267277684011.1, -0.04815896904694777, + ... -18741774.600527752, -454267277558522.1] + >>> float(sum(map(Fraction, arr))) # Exact summation with single rounding + 5.777552047891987e-12 + >>> math.fsum(arr) # Single rounding + 5.777552047891987e-12 + >>> sum(arr) # Multiple roundings in extended precision + 5.777545108998083e-12 + + .. _tut-fp-error: Representation Error From 22e1a96a6a63809905b462fdbd28a7865a9f1fb8 Mon Sep 17 00:00:00 2001 From: Raymond Hettinger Date: Sun, 19 Feb 2023 11:42:23 -0600 Subject: [PATCH 04/10] Show math.isclose() for comparing inexact values --- Doc/tutorial/floatingpoint.rst | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-) diff --git a/Doc/tutorial/floatingpoint.rst b/Doc/tutorial/floatingpoint.rst index e1e53133ab7e3e..ab3f13c2dc0cf7 100644 --- a/Doc/tutorial/floatingpoint.rst +++ b/Doc/tutorial/floatingpoint.rst @@ -119,11 +119,10 @@ Also, since the 0.1 cannot get any closer to the exact value of 1/10 and False Though the numbers cannot be made closer to their intended exact values, -the :func:`round` function can be useful for post-rounding so that results -with inexact values become comparable to one another:: +the :func:`math.isclose` function can be useful for comparing inexact values:: - >>> round(0.1 + 0.1 + 0.1, 10) == round(0.3, 10) - True + >>> math.isclose(0.1 + 0.1 + 0.1, 0.3) + True Binary floating-point arithmetic holds many surprises like this. The problem with "0.1" is explained in precise detail below, in the "Representation Error" From cb1f78ab21b144266e2f1cf2d181775518f8a47b Mon Sep 17 00:00:00 2001 From: Raymond Hettinger Date: Sun, 19 Feb 2023 12:05:47 -0600 Subject: [PATCH 05/10] Add doctests throughout --- Doc/tutorial/floatingpoint.rst | 86 +++++++++++++++++++++++++--------- 1 file changed, 65 insertions(+), 21 deletions(-) diff --git a/Doc/tutorial/floatingpoint.rst b/Doc/tutorial/floatingpoint.rst index ab3f13c2dc0cf7..068c85711f4a9e 100644 --- a/Doc/tutorial/floatingpoint.rst +++ b/Doc/tutorial/floatingpoint.rst @@ -57,13 +57,17 @@ Many users are not aware of the approximation because of the way values are displayed. Python only prints a decimal approximation to the true decimal value of the binary approximation stored by the machine. On most machines, if Python were to print the true decimal value of the binary approximation stored -for 0.1, it would have to display :: +for 0.1, it would have to display: + +.. doctest:: >>> 0.1 0.1000000000000000055511151231257827021181583404541015625 That is more digits than most people find useful, so Python keeps the number -of digits manageable by displaying a rounded value instead :: +of digits manageable by displaying a rounded value instead: + +.. doctest:: >>> 1 / 10 0.1 @@ -90,7 +94,10 @@ thing in all languages that support your hardware's floating-point arithmetic (although some languages may not *display* the difference by default, or in all output modes). -For more pleasant output, you may wish to use string formatting to produce a limited number of significant digits:: +For more pleasant output, you may wish to use string formatting to produce a +limited number of significant digits: + +.. doctest:: >>> format(math.pi, '.12g') # give 12 significant digits '3.14159265359' @@ -101,32 +108,41 @@ For more pleasant output, you may wish to use string formatting to produce a lim >>> repr(math.pi) '3.141592653589793' - It's important to realize that this is, in a real sense, an illusion: you're simply rounding the *display* of the true machine value. One illusion may beget another. For example, since 0.1 is not exactly 1/10, -summing three values of 0.1 may not yield exactly 0.3, either:: +summing three values of 0.1 may not yield exactly 0.3, either: + +.. doctest:: >>> 0.1 + 0.1 + 0.1 == 0.3 False Also, since the 0.1 cannot get any closer to the exact value of 1/10 and 0.3 cannot get any closer to the exact value of 3/10, then pre-rounding with -:func:`round` function cannot help:: +:func:`round` function cannot help: + +.. doctest:: >>> round(0.1, 1) + round(0.1, 1) + round(0.1, 1) == round(0.3, 1) False Though the numbers cannot be made closer to their intended exact values, -the :func:`math.isclose` function can be useful for comparing inexact values:: +the :func:`math.isclose` function can be useful for comparing inexact values: + +.. doctest:: >>> math.isclose(0.1 + 0.1 + 0.1, 0.3) True Binary floating-point arithmetic holds many surprises like this. The problem with "0.1" is explained in precise detail below, in the "Representation Error" -section. See `The Perils of Floating Point `_ +section. See `Examples of Floating Point Problems +`_ for +a pleasant summary of how binary floating point works and the kinds of +problems commonly encountered in practice. Also see +`The Perils of Floating Point `_ for a more complete account of other common surprises. As that says near the end, "there are no easy answers." Still, don't be unduly @@ -157,26 +173,34 @@ statistical operations supplied by the SciPy project. See . Python provides tools that may help on those rare occasions when you really *do* want to know the exact value of a float. The :meth:`float.as_integer_ratio` method expresses the value of a float as a -fraction:: +fraction: + +.. doctest:: >>> x = 3.14159 >>> x.as_integer_ratio() (3537115888337719, 1125899906842624) Since the ratio is exact, it can be used to losslessly recreate the -original value:: +original value: + +.. doctest:: >>> x == 3537115888337719 / 1125899906842624 True The :meth:`float.hex` method expresses a float in hexadecimal (base -16), again giving the exact value stored by your computer:: +16), again giving the exact value stored by your computer: + +.. doctest:: >>> x.hex() '0x1.921f9f01b866ep+1' This precise hexadecimal representation can be used to reconstruct -the float value exactly:: +the float value exactly: + +.. doctest:: >>> x == float.fromhex('0x1.921f9f01b866ep+1') True @@ -189,7 +213,9 @@ Another helpful tool is the :func:`sum` function which helps mitigate loss-of-precision during summation. It uses extended precision for intermediate rounding steps as values are added onto a running total. That can make a difference in overall accuracy so that the errors do not -accumulate to the point where they affect the final total:: +accumulate to the point where they affect the final total: + +.. doctest:: >>> 0.1 + 0.1 + 0.1 + 0.1 + 0.1 + 0.1 + 0.1 + 0.1 + 0.1 + 0.1 == 1.0 False @@ -200,7 +226,9 @@ The :func:`math.fsum()` goes further and tracks all of the "lost digits" as values are added onto a running total so that the result has only a single rounding. This is slower than :func:`sum` but will be more accurate in uncommon cases where large magnitude inputs mostly cancel -each other out leaving a final sum near zero:: +each other out leaving a final sum near zero: + +.. doctest:: >>> arr = [-125546.2071591587, 0.04336622547760194, 18741774.60535662, ... 57.207123035704896, 454267277684011.1, -0.04815896904694777, @@ -241,20 +269,28 @@ as :: J ~= 2**N / 10 and recalling that *J* has exactly 53 bits (is ``>= 2**52`` but ``< 2**53``), -the best value for *N* is 56:: +the best value for *N* is 56: + +.. doctest:: >>> 2**52 <= 2**56 // 10 < 2**53 True That is, 56 is the only value for *N* that leaves *J* with exactly 53 bits. The -best possible value for *J* is then that quotient rounded:: +best possible value for *J* is then that quotient rounded: + +.. doctest:: >>> q, r = divmod(2**56, 10) >>> r 6 Since the remainder is more than half of 10, the best approximation is obtained -by rounding up:: +by rounding up: + +.. doctest:: + + >>> q+1 7205759403792794 @@ -272,13 +308,17 @@ if we had not rounded up, the quotient would have been a little bit smaller than 1/10. But in no case can it be *exactly* 1/10! So the computer never "sees" 1/10: what it sees is the exact fraction given -above, the best 754 double approximation it can get:: +above, the best 754 double approximation it can get: + +.. doctest:: >>> 0.1 * 2 ** 55 3602879701896397.0 If we multiply that fraction by 10\*\*55, we can see the value out to -55 decimal digits:: +55 decimal digits: + +.. doctest:: >>> 3602879701896397 * 10 ** 55 // 2 ** 55 1000000000000000055511151231257827021181583404541015625 @@ -286,13 +326,17 @@ If we multiply that fraction by 10\*\*55, we can see the value out to meaning that the exact number stored in the computer is equal to the decimal value 0.1000000000000000055511151231257827021181583404541015625. Instead of displaying the full decimal value, many languages (including -older versions of Python), round the result to 17 significant digits:: +older versions of Python), round the result to 17 significant digits: + +.. doctest:: >>> format(0.1, '.17f') '0.10000000000000001' The :mod:`fractions` and :mod:`decimal` modules make these calculations -easy:: +easy: + +.. doctest:: >>> from decimal import Decimal >>> from fractions import Fraction From e9e959a08a86e5a7ad41a813bf0da76c32980141 Mon Sep 17 00:00:00 2001 From: Raymond Hettinger Date: Sun, 19 Feb 2023 12:11:12 -0600 Subject: [PATCH 06/10] Add a more useful example of round() for comparisons --- Doc/tutorial/floatingpoint.rst | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/Doc/tutorial/floatingpoint.rst b/Doc/tutorial/floatingpoint.rst index 068c85711f4a9e..47e603dbbbb1b7 100644 --- a/Doc/tutorial/floatingpoint.rst +++ b/Doc/tutorial/floatingpoint.rst @@ -136,6 +136,14 @@ the :func:`math.isclose` function can be useful for comparing inexact values: >>> math.isclose(0.1 + 0.1 + 0.1, 0.3) True +Alternatively, the :func:`round` function can be used to compare rough +approximations:: + +.. doctest:: + + >>> round(math.pi, ndigits=2) == round(22 / 7, ndigits=2) + True + Binary floating-point arithmetic holds many surprises like this. The problem with "0.1" is explained in precise detail below, in the "Representation Error" section. See `Examples of Floating Point Problems From 8f02dc507c46b4fa5759ea1811394823aaf14e62 Mon Sep 17 00:00:00 2001 From: Raymond Hettinger Date: Sun, 19 Feb 2023 12:15:49 -0600 Subject: [PATCH 07/10] Shorter example --- Doc/tutorial/floatingpoint.rst | 17 ++++++++--------- 1 file changed, 8 insertions(+), 9 deletions(-) diff --git a/Doc/tutorial/floatingpoint.rst b/Doc/tutorial/floatingpoint.rst index 47e603dbbbb1b7..6f8512289db983 100644 --- a/Doc/tutorial/floatingpoint.rst +++ b/Doc/tutorial/floatingpoint.rst @@ -238,15 +238,14 @@ each other out leaving a final sum near zero: .. doctest:: - >>> arr = [-125546.2071591587, 0.04336622547760194, 18741774.60535662, - ... 57.207123035704896, 454267277684011.1, -0.04815896904694777, - ... -18741774.600527752, -454267277558522.1] - >>> float(sum(map(Fraction, arr))) # Exact summation with single rounding - 5.777552047891987e-12 - >>> math.fsum(arr) # Single rounding - 5.777552047891987e-12 - >>> sum(arr) # Multiple roundings in extended precision - 5.777545108998083e-12 + >>> arr = [-0.10430216751806065, -266310978.67179024, 143401161448607.16, + ... -143401161400469.7, 266262841.31058735, -0.003244936839808227] + >>> float(sum(map(Fraction, arr))) # Exact summation with single rounding + 8.042173697819788e-13 + >>> math.fsum(arr) # Single rounding + 8.042173697819788e-13 + >>> sum(arr) # Multiple roundings in extended precision + 8.042178034628478e-13 .. _tut-fp-error: From d8bcc4cd3efe3f5a8d1ff890e46ad29cac0a40e3 Mon Sep 17 00:00:00 2001 From: Raymond Hettinger Date: Sun, 19 Feb 2023 12:27:45 -0600 Subject: [PATCH 08/10] Fix doctests: Missing import and counterfactual example. --- Doc/tutorial/floatingpoint.rst | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/Doc/tutorial/floatingpoint.rst b/Doc/tutorial/floatingpoint.rst index 6f8512289db983..f538b1d69c6dc3 100644 --- a/Doc/tutorial/floatingpoint.rst +++ b/Doc/tutorial/floatingpoint.rst @@ -1,6 +1,7 @@ .. testsetup:: import math + from fractions import Fraction .. _tut-fp-issues: @@ -57,9 +58,7 @@ Many users are not aware of the approximation because of the way values are displayed. Python only prints a decimal approximation to the true decimal value of the binary approximation stored by the machine. On most machines, if Python were to print the true decimal value of the binary approximation stored -for 0.1, it would have to display: - -.. doctest:: +for 0.1, it would have to display:: >>> 0.1 0.1000000000000000055511151231257827021181583404541015625 From cd8abc516370508654387097cafeb1032e3eed38 Mon Sep 17 00:00:00 2001 From: Raymond Hettinger Date: Sun, 19 Feb 2023 13:05:45 -0600 Subject: [PATCH 09/10] Compare sum() and fsum() with straight addition --- Doc/tutorial/floatingpoint.rst | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/Doc/tutorial/floatingpoint.rst b/Doc/tutorial/floatingpoint.rst index f538b1d69c6dc3..c51a66b0d3e010 100644 --- a/Doc/tutorial/floatingpoint.rst +++ b/Doc/tutorial/floatingpoint.rst @@ -245,6 +245,12 @@ each other out leaving a final sum near zero: 8.042173697819788e-13 >>> sum(arr) # Multiple roundings in extended precision 8.042178034628478e-13 + >>> total = 0.0 + >>> for x in arr: + ... total += x # Multiple roundings in standard precision + ... + >>> total # Straight addition has no correct digits! + -0.0051575902860057365 .. _tut-fp-error: From d6a6b6b9ea4a1f9c192176063213ca7e1d705c81 Mon Sep 17 00:00:00 2001 From: Raymond Hettinger Date: Sun, 19 Feb 2023 13:08:19 -0600 Subject: [PATCH 10/10] Add section author --- Doc/tutorial/floatingpoint.rst | 1 + 1 file changed, 1 insertion(+) diff --git a/Doc/tutorial/floatingpoint.rst b/Doc/tutorial/floatingpoint.rst index c51a66b0d3e010..306b1eba3c45b8 100644 --- a/Doc/tutorial/floatingpoint.rst +++ b/Doc/tutorial/floatingpoint.rst @@ -10,6 +10,7 @@ Floating Point Arithmetic: Issues and Limitations ************************************************** .. sectionauthor:: Tim Peters +.. sectionauthor:: Raymond Hettinger Floating-point numbers are represented in computer hardware as base 2 (binary)