Talk:Machine epsilon

Definition

This page defines the machine epsilon as "the smallest floating point number such that$1\oplus \epsilon >1$". What I have seen more commonly is the definition to be the largest floating point number such that $1\oplus \epsilon =1$. In fact the equation provided $\epsilon ={\beta \over 2}\cdot \beta ^{-p}$ gives the latter definition. Granted the two definitions lead to numbers adjacent on the floating point number line, but I would like to see this article either switch to the other definition or else discuss the presence of two definitions in use. Any thoughts? --Jlenthe 01:01, 9 October 2006 (UTC)

Hi, with a quick Google check I found as expected some confirmations for the "smallest thingy making a difference" definition, not "biggest making no difference". IIRC that's also what I learned two and a half decades ago. 212.82.251.209 20:48, 3 December 2006 (UTC)

Yes; if it didn't make a difference, it wouldn't be an "epsilon". --Quuxplusone 01:05, 19 December 2006 (UTC)

Two numbers that are adjacent on the floating point number line are, of course, infinitely far apart on the real number line. I'm just saying... 129.42.208.184 (talk) 19:05, 6 July 2014 (UTC)

That doesn't make sense. By what definition of distance? 184.23.18.144 (talk) 06:21, 4 March 2015 (UTC)

The definition now appears to have changed away from either of those in the first comment in this note, to "an upper bound on the relative error due to rounding". That definition, assuming rounding to nearest, gives us half the value that is generally used, so for example 2^(-24) for IEEE binary32, instead of 2^(-23). The larger value appears as FLT_EPSILON in float.h for C/C++. Then in the table below, the standard definition is used in the last three columns for binary32 and binary64, in contradiction with the header. (As if to make the result match FLT_EPSILSON and DBL_EPSILON??) 86.24.142.189 (talk) 17:21, 5 January 2013 (UTC)

I'd be hugely in favor changing to the industry-standard definition (which is widely used in academia), and doing away altogether with the definition that seems to be used by one academic, but isn't used widely in the industry. (i.e. go with the larger values). That was certainly the definition I was taught in my Numeric Analysis classes. rerdavies (talk)

I'm probably thinking it wrong, but shouldn't the definition be "lower bound on relative error"? Makes more sense for me in a mathematical sense, as $\epsilon <E_{r}$, where $E_{r}$ is the relative error, i.e., epsilon would be a lower bound on the error. Am I getting it wrong? Sabian2008 (talk) 21:50, 27 February 2018 (UTC)

No, "lower bound" is wrong. That would say that all operations produce AT LEAST that much relative error. Consider 1.0 + 1.0 = 2.0 has zero error. We want an upper bound, to say all operations produce AT MOST that much relative error, $E_{r}<\epsilon$.

The variant definition section said, "smallest number that, when added to one, yields a result different from one", that is, $1+\epsilon >1$. But that $\epsilon$, in double, is $2^{-53}+2^{-105}$, i.e., the next floating point number greater than $2^{-53}$. To get the value that was intended, $2^{-52}$, the C & C++ standards, Matlab, etc. refer to distance between 1 and the next larger floating point number.

Code or definition wrong

If you use the code with

float machEps = 1.00001f;

you get smaller numbers with 1+eps>1, Instead the relativ difference between two floating point numers is computed. --Mathemaduenn 10:20, 10 October 2006 (UTC)

I don't see any code in the current article with

float machEps = 1.0001f;

Therefore, I guess this has been resolved, and I'm removing the {{contradict}} tag now. --Quuxplusone 01:05, 19 December 2006 (UTC)

No, the point Mathemaduenn was trying to make was that if you start with machEps = 1.00001f rather than machEps = 1.0f then you end up with a smaller machine epsilon.

However, that we know (from calculating 2^(-23)) the correct answer should be 1.19209E-07 suggests that the algorithm in "Approximation using C" is wrong. As far as I can tell, Mathemaduenn is claiming that the algorithm in "Approximation using C" produces the relativ difference rather than the machine epsilon. This confusion of definitions is in fact covered in this article under "Other definitions". -- Tom Fitzhenry 13:43, 7 October 2009 (UTC) —Preceding unsigned comment added by 130.88.199.107 (talk)

Calculation wrong

This is wrong: the difference between these numbers is 0.00000000000000000000001, or 2⁻²³. 0.00000000000000000000001 is 10^-23, but that is probably not the right number. —Preceding unsigned comment added by 193.78.112.2 (talk) 06:43, 19 October 2007 (UTC)

It's a binary fraction, not a decimal fraction. I agree it's confusing, but I can't think of a better way to explain it. -- BenRG 19:52, 19 October 2007 (UTC)

References

The page lists 'David Goldberg (March 1991). "What Every Computer Scientist Should Know About Floating-Point Arithmetic".' as one of it's references. That paper defines machine epsilon as the largest possible relative error when a real number is approximated by a floating point number closest to it. ("..the largest of the bounds in (2) above.."). The formula give is as mentioned by Jlenthe above. So either that paper should be removed from the list of references, or the definition provided in it should also be mentioned. —Preceding unsigned comment added by Gautamsewani (talk • contribs) 16:11, 18 August 2008 (UTC)

Actually, that would be smaller of the two values -- typically HALF the smallest value that you can add to one and make a difference -- since the value Goldberg describes would be +/- the closest machine respresentation. rerdavies (talk) —Preceding undated comment added 03:49, 31 January 2014 (UTC)

C++ style cast

"we can simply take the difference of 1.0 and double(long long (1.0) + 1)."

It's problematic because double(long long(1.0) + 1) evaluates to 2. It should be reinterpret_cast, but I don't think that it will improve the readability. bungalo (talk) 09:32, 5 October 2009 (UTC)

About to Make Major Changes

As of October 20, 2009, the information on this page is wrong. Unless anyone has some strong thoughts to the contrary, in a few days I will make some major changes to put it right. Jfgrcar (talk) 03:54, 21 October 2009 (UTC)

I corrected this section today, October 25, 2009. I did not alter the examples, which are still wrong. —Preceding unsigned comment added by Jfgrcar (talk • contribs) 00:15, 26 October 2009 (UTC)

"How to determine machine epsilon"

The section "How to determine machine epsilon" contains strange implementations that try to approximate machine epsilon.

In standard C we can nowadays simply use the DBL_EPSILON constant from float.h. And more generally, you can use the nextafter family of functions from math.h; for example, "nextafter(1.0, 2.0) - 1.0" should evaluate to DBL_EPSILON if I'm not mistaken. (By the way, the C standard even gives an example showing what this constant should be if you use IEEE floating point numbers: DBL_EPSILON = 2.2204460492503131E-16 = 0X1P-52.)

In Java, we have methods like java.lang.Math.nextAfter and java.lang.Math.ulp. Again, no need to use approximations and iterations. — Miym (talk) 07:22, 26 October 2009 (UTC)

I think in fortran, you can call epsilon(one) (I know this because this line caused an error when I was trying to convert C to fortran with f2c). 78.240.11.120 (talk) 13:49, 25 February 2012 (UTC)

Other languages also define constants and/or functions that can be used to obtain the value. I agree that the article spends too many column inches on all of those different approximation subroutines. 129.42.208.184 (talk) 19:11, 6 July 2014 (UTC)

"...do not provide methods to change the rounding mode..."

Section "Values for standard hardware floating point arithmetics" claims that "while the standard allows several methods of rounding, programming languages and systems vendors do not provide methods to change the rounding mode from the default: round-to-nearest with the tie-breaking scheme round-to-even."

This claim seems to be incorrect. First, "programming languages" do provide such methods: the C standard provides the functions fesetround and fegetround and macros such as FE_DOWNWARD and FE_UPWARD in fenv.h. Second, "system vendors" do implement these: I just tried in a standard Gnu/Linux environment and these seem to be working (mostly) as expected. — Miym (talk) 07:37, 26 October 2009 (UTC)

Machine epsilon

Why is the table at the article's beginning listing the machine epsilon as pow(2, -53), when the calculation (correctly) arrives at the conclusion that it is indeed pow(2, -52) (for double precision, i.e. p=52? —Preceding unsigned comment added by 109.90.227.146 (talk) 21:14, 28 September 2010 (UTC)

Conversion of "Approximation using Java" to Matlab code

I don't know if this is worth putting on the page, but I've just converted the Java estimation code for double type in Matlab. If it's worth adding, here it is to save other people converting it.

function calculateMachineEpsilonDouble()
    machEps = double(1.0);
    done = true;
    while done
        machEps = machEps/2.0;
        done = (double(1.0 + (machEps/2.0)) ~= 1.0);
    end
    fprintf('Machine Epsilon = %s\n', num2str(machEps));
end

137.108.145.10 (talk) 15:29, 3 February 2011 (UTC)

In MATLAB there exists an eps() function that will give you back the machine eps Gicmo (talk) 22:41, 25 January 2012 (UTC)

Approximation using C#

I ported the C version into a C# version. I don't know if this would be valuable or worth adding to the article, so I'm adding it here and letting someone else make the judgement call.

static void Main(string[] args)
{
    float machEps = 1.0f;
    do
    {
        Console.WriteLine(machEps.ToString("f10") + "\t" + (1.0f + machEps).ToString("f15"));
        machEps /= 2.0f;
    }while((float)(1.0f + (machEps / 2.0f)) != 1.0f);
    Console.WriteLine("Calculated Machine Epsilon: " + machEps.ToString("f15"));
}

Approximation using Prolog

This Prolog code approximates the machine epsilon.

epsilon(X):-
	Y is (1.0 + X),
	Y = 1.0,
	write(X).
epsilon(X):-
	Y is X/2,
	epsilon(Y).

An example execution in SWI-Prolog:

1 ?- epsilon(1.0).
1.1102230246251565e-16
true .

--201.209.40.226 (talk) 04:23, 30 July 2011 (UTC)

In practice

"The following are encountered in practice?" Whose practice? If you ask for the machine epsilon of a double in any programming language I can think of that has a specific function for it (eps functions in Matlab and Octave, finfo in numpy, std::numeric_limits<double>::eps() in C++), then you get $2^{-52}$, not $2^{-53}$. Yes, I realise that there are other definitions you can use, but I find it very weird to say that "in practice" you find these numbers, whereas actual people who have to deal with epsilon deal with the smallest step you can take in the mantissa for a given exponent. JordiGH (talk) 01:08, 1 September 2011 (UTC)

inconsistency

The article states for double "64-bit doubles give 2.220446e-16, which is 2-52 as expected." but the table at the top lists 2-53. Brianbjparker (talk) 17:23, 8 March 2012 (UTC)

confusing definition

The definition of machine epsilon given use a definition of precision p that excludes the implicit bit so e.g. for double uses a p of 52 rather than the usual definition of p=53. This is very confusing-- the definition and table should be changed to use the standard definition of p including the implicit bit. Brianbjparker (talk) 17:39, 8 March 2012 (UTC)

I changed the definition of p throughout to refer to IEEE 754 precision p-- i.e. including implicit bit. Brianbjparker (talk) 23:21, 8 March 2012 (UTC)

Simpler python

The python example uses numpy which is not always installed. Wouldn't the following example, using the standard sys module be more relevant, even if restricted to floats ?

    In [1]: import sys
    In [2]: sys.float_info.epsilon
    Out[2]: 2.220446049250313e-16

Frédéric Grosshans (talk) 19:49, 15 March 2012 (UTC)

C++ sample

Those programs given here all uses an aproximation, you have to note that when calculating 'double' you are actually dealing with numbers of 80 bits (not 64 bits). 64 bits is only a storage, what the procesor does is calculating with 80 bit (or more) precission. The simplest way to calculate machine epsilon can be as such:

for double:

#include <iostream>
#include <stdint.h>
#include <iomanip>

int main()
{
        union
        {
                double f;
                uint64_t i;
        } u1, u2, u3;
			   
        u1.i = 0x3ff0000000000000ul;
	// one (exponent set to 1023 and mantissa to zero
	// one bit from mantissa is implicit

        u2.i = 0x3ff0000000000001ul;
	// one and a little more

        u3.f = u2.f - u1.f;

        std::cout << "machine epsilon: " << std::setprecision(18) << u3.f << std::endl;
}

above program gives:

/home/tomek/roboczy/test$ g++ -O2 -o test test.cpp && ./test
machine epsilon: 2.22044604925031308e-16

and a sample for 32 bit (float)

#include <iostream>
#include <stdint.h>
#include <iomanip>

int main()
{
        union
        {
                float f;
                uint32_t i;
        } u1, u2, u3;
			   
        u1.i = 0x3F800000;
        u2.i = 0x3F800001;
        u3.f = u2.f - u1.f;

        std::cout << "machine epsilon: " << std::setprecision(18) << u3.f << std::endl;
}

and it gives:

/home/tomek/roboczy/test$ g++ -O2 -o test test.cpp && ./test
machine epsilon: 1.1920928955078125e-07

of course above *decimal* values are only an aproximation so I don't see the sense to print them directly. Also C++ users can use std::numeric_limits to get those constants:

       std::cout << std::numeric_limits<float>::epsilon() << std::endl;
       std::cout << std::numeric_limits<double>::epsilon() << std::endl;

---

Fails for portability on two counts, I'm afraid. Works only if you're using a C++ compiler that (1) conforms to IEEE floating point (many ARM compilers don't by default, fwiw); and (2) you're running on a big-endian machine. rerdavies (talk)

Mathematical proof

And a next sample how to calculate it in a more 'mathematical' manner. Let's talk about double:

                S EEEEEEEEEEE FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF
                0 1        11 12                                                63
		The value V represented by the word may be determined as follows:

		* If E=2047 and F is nonzero, then V=NaN ("Not a number")
		* If E=2047 and F is zero and S is 1, then V=-Infinity
		* If E=2047 and F is zero and S is 0, then V=Infinity
		* If 0<E<2047 then V=(-1)**S * 2 ** (E-1023) * (1.F) where "1.F" is intended
		  to represent the binary number created by prefixing F with an implicit
		  leading 1 and a binary point.
		* If E=0 and F is nonzero, then V=(-1)**S * 2 ** (-1022) * (0.F) These are
		  "unnormalized" values.
		* If E=0 and F is zero and S is 1, then V=-0
		* If E=0 and F is zero and S is 0, then V=0

so if you want to set a double to 1.0 you have to set exponent to 1023 and mantissa to zero (one bit is implicit), e.g.

0 01111111111 0000000000000000000000000000000000000000000000000000

if you want a 'little' more than one you have to change the last bit to one:

0 01111111111 0000000000000000000000000000000000000000000000000001

the last bit above has value 2^-52 (not 2^-53) and the machine epsilon can be caltulated in this way: $[2^{(exp-1023)}]\cdot (1+2^{-52})-[2^{(exp-1023)}]\cdot (1)=[2^{(exp-1023)}]\cdot (1+2^{-52}-1)=2^{-52}$

I have changed values in the first table (binary32, binary64), the rest I don't have a time to test.

--Tsowa (talk) 13:14, 3 November 2012 (UTC)

Well there's your problem. You are using the definition of machine epsilon which is compatible with C FLT_EPSILON and DBL_EPSILON: the largest relative difference between neighbouring values. The definition in this article is the maximum relative rounding error, which is half that amount. It may be better to change the definition - I don't really mind either way - but we must certainly keep the article internally consistent. I have changed those two values and related formulae back. 86.24.142.189 (talk) 17:25, 9 January 2013 (UTC)

External links modified

Hello fellow Wikipedians,

I have just modified 2 external links on Machine epsilon. Please take a moment to review my edit. If you have any questions, or need the bot to ignore the links, or the page altogether, please visit this simple FaQ for additional information. I made the following changes:

• Added archive https://web.archive.org/web/20130807071540/http://www.mathworks.de/de/help/matlab/ref/eps.html to http://www.mathworks.de/de/help/matlab/ref/eps.html
• Added archive https://web.archive.org/web/20060904045658/http://orion.math.iastate.edu/burkardt/c_src/machar/machar.html to http://orion.math.iastate.edu/burkardt/c_src/machar/machar.html

When you have finished reviewing my changes, you may follow the instructions on the template below to fix any issues with the URLs.

This message was posted before February 2018. After February 2018, "External links modified" talk page sections are no longer generated or monitored by InternetArchiveBot. No special action is required regarding these talk page notices, other than regular verification using the archive tool instructions below. Editors have permission to delete these "External links modified" talk page sections if they want to de-clutter talk pages, but see the RfC before doing mass systematic removals. This message is updated dynamically through the template {{source check}} (last update: 18 January 2022).

• If you have discovered URLs which were erroneously considered dead by the bot, you can report them with this tool.
• If you found an error with any archives or the URLs themselves, you can fix them with this tool.

Cheers.—InternetArchiveBot (Report bug) 12:50, 11 January 2018 (UTC)

Confusion between machine epsilon and unit roundoff

This article seems to make a confusion between the machine epsilon ε and the unit roundoff u. While the definition of the unit roundoff u seems standard (= $b^{1-p}/2$ when considering round-to-nearest), the definition of ε is not, as explained in the article. The lead says

The quantity is also called macheps or unit roundoff, and it has the symbols Greek epsilon $\varepsilon$ or bold Roman u, respectively.

as if there were no differences between them. — Vincent Lefèvre (talk) 22:17, 22 December 2021 (UTC)

Suggestion about improved terminology: "Rounding Machine Epsilon" and "Interval Machine Epsilon"

The wiki page on "Machine epsilon" acknowledges that there are two camps: the "formal definition" and "variant definitions". I notice that the software industry overwhelmingly uses the "variant definition", e.g for IEEE 754 'binary64', we have the C programming langiage DBL_EPSILON=$2^{-52}$ constant from float.h, Python sys.float_info.epsilon=$2^{-52}$, Fortran 90 EPSILON(1.0_real64)=$2^{-52}$, MATLAB eps=$2^{-52}$, Pascal epsreal=$2^{-52}$, etc. Therefore I feel that these applications deserve a clearer terminology than just "variants". What do you think about following suggestion?:

• Rounding Machine Epsilon, $b^{-(p-1)}/2$: This term can be used to refer to the formal academic definition of machine epsilon as per Prof. Demmel, LAPACK & SciLab. It represents the largest relative rounding error in round-to-nearest mode. The rationale is that the rounding error is half the interval upwards to the next representable number in finite-precision. Thus, the relative rounding error for number $x$ is $[interval/2]/x$. In this context, the largest relative error occurs when $x=1.0$, and is equal to $[ULP(1.0)/2]/1.0$, because real numbers in the lower half of the interval 1.0 ~ 1.0+ULP(1) are rounded down to 1.0, and numbers in the upper half of the interval are rounded up to 1.0+ULP(1). Here we use the definition of ULP(1) [Unit in Last Place] as the positive difference between 1.0 (which can be represented exactly in finite-precision) and the next greater number representable in finite-precision. IEEE 754 'binary64' double-precision has ULP(1) = $2^{-52}$, so "rounding machine epsilon" is ULP(1)/2 = $2^{-53}$ ≈ 1.11E-16. This means that 1.0+1.10E-16 gets rounded down to 1.0, while 1.0+1.12E-16 gets rounded up to 1.0+2.22E-16.

• Interval Machine Epsilon, $b^{-(p-1)}$: This term can be used for the "widespread variant definition" of machine epsilon as per Prof. Higham, and applied in language constants in C, C++, Python, Fortran, MATLAB, Pascal, Ada, Rust, and textsbooks like «Numerical Recipes» by Press et al. It represents the largest relative interval between two nearest numbers in finite-precision, or the largest rounding error in round-by-chop mode. The rationale is that the relative interval for number $x$ is $[interval]/x$ where $interval$ is the distance to upwards the next representable number in finite-precision. In this context, the largest relative interval occurs when $x=1.0$, and is the interval between 1.0 (which can be represented exactly in finite-precision) and the next larger representable floating-point number. This interval is equal to ULP(1). IEEE 754 'binary64' double-precision has ULP(1) = $2^{-52}$, so "interval machine epsilon" is $2^{-52}$ ≈ 2.22E-16.

In summary: "rounding machine epsilon" is simply half the "interval machine epsilon", because the rounding error is equal to half the interval in round-to-nearest mode. I hope that this can help to clarify the distinction between the two definitions of machine epsilon. What do you think? Peter.schild (talk) 21:40, 19 November 2023 (UTC)