Remove lower bound on size of epsilon #131
Open
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR basically removes the lower bound on the size of
epsilon
. It fixes the current problem for smallx
with some functions, like, e.g.,f(x) = log(x)
. MWE of the issue:It is not only a problem of throwing an error, it is also a problem for accuracy, as illustrated in @andreasnoack's slack example, reproduced here:
Note this PR is not the ideal solution. In particular, the test for the gradient of
f(x) = sin(x[1]) + cos(x[2])
at[0, 0]
, which currently passes thanks to the lower bound, does not pass with this PR. (I thus changed the test to be at[1, 1]
rather than[0, 0]
.)My understanding is that even for simpler cases, e.g., for
f(x) = x + a
, wherea
is a "big", thenepsilon
must be big too in order to not be collapsed to0
when added toa
. (That means ifepsilon
just scales withx
, the finite-difference will only work forx
of a similar magnitude toa
).However, this seems like a rarer and more pathological case than the
f(x) = log(x)
case to me. So my preference would go to removing themax(1, abs(x))
, hence this PR.As noted by Jeffrey Sarnoff, an ideal solution should implement some lower bound. Maybe one of you knows of a better way to implement said lower bound so that it is not a problem in the cases like
f(x) = log(x)
?Another thing noted by @andreasnoack is that maybe
cbrt
is not ideal in:central
difference and may be replaced bysqrt
(as in the example above). I am not sure about that either, but I was told to explicitly raise the issue here as well 😃.[Note on why I would like this to be solved:] I want to be able to use Julia's state-of-the-art numerical differentiation tools (AD, dual and hyperdual numbers, etc.) and would like to be able to quantify how these tools perform vs state-of-the-art finite differences in some specific scientific applications. But this
f(x) = log(x)
issue is preventing me from doing this...Last note, I edited the tests quite a bit (apart from the
[0,0]
-to-[1,1]
change), but it was mostly to enclose each group of tests in a@testset
of its own for me to be able to find where the tests failed more easily.