Lazy rectilinear interpolator

[fnattino]

🚀 Pull Request

Description

Different take to enable the rectilinear interpolator to run lazily #6002 .

Trying to address the same issue as #6006, but there I have made the underlying _RegularGridInterpolator from _scipy_interpolate to run on lazy data, which required switching from scipy.sparse to sparse (not ideal since it would add numba as a dependency).

Here I have tried instead to implement a similar approach as used for regridding, which works on lazy data as well. The downside is that the chunks in the dimensions we are interpolating over need to be merged, but at least we could run interpolation in parallel over the chunks in the other dimensions (and we do not need to add extra dependencies to iris).

[bouweandela]

Nice to see this progressing @fnattino! Did you notice that CI is failing on this pull request?

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 89.82%. Comparing base (df4c52b) to head (3b210e4).

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #6084   +/-   ##
=======================================
  Coverage   89.81%   89.82%           
=======================================
  Files          88       88           
  Lines       23347    23348    +1     
  Branches     4344     4341    -3     
=======================================
+ Hits        20970    20972    +2     
  Misses       1649     1649           
+ Partials      728      727    -1     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[trexfeathers]

@fnattino just want to reassure you that I have been looking at this, but since I have never worked with our interpolators before it is slow progress. Having another go this afternoon with help from some coffee ☕

[fnattino]

No worries @trexfeathers, but thanks for the heads-up! :)

[trexfeathers]

Hi @fnattino, thank you for your hard work on this.

Here is a partial review. I have left myself a couple of TODO comments. But the suggestions I have already might take some time, and may change the code significantly - mainly #6084 (comment) - so it seemed important to get these suggestions to you as soon as possible.

Also thank you to @HarriKeenan for helping me with this review last week 🤜🤛

[fnattino]

@trexfeathers thanks a lot for the review and apologies for the scattered response.

As I have tried to explain in reply to your comments, I am a bit hesitant to implement the solution that would copy the current instance of the RectilinearInterpolation - but I am very curious to hear your thoughts on this!

[trexfeathers]

OK I got my head back in the space and I understand your logic better now. Nearly there.

If you're busy with other things please let me know and I can try actioning the remaining stuff myself.

[trexfeathers]

Clicked the wrong button with my previous review!

[fnattino]

Thanks a lot for the effort of getting back to this @trexfeathers ! I should have time to get back to this next week, but I will keep you posted!

[fnattino]

Thanks again for the review @trexfeathers and sorry with the delay in replying. In addition to replying to your comments, I have updated a couple of docstrings (where I left comments). I have also realized that the user guide should probably also be updated, especially the section on caching the interpolator? I can take care of this tomorrow. Any other docs that should be updated?

[fnattino]

I have made some small updates to the userguide to add that the linear and nearest-neighbour interpolators can now work with lazy data as well.

Just wanted to double check, but I think we should also remove the subsections on "caching an interpolator" and "caching a regridder": as discussed in the thread below this comment, almost all the work is done when calling the interpolator (within RectilinearInterpolator._interpolate) so there seem to be no actual benefit in constructing and reusing the interpolator objects? And I think this is also true for the regridder, for which all the work seems to done within RectilinearRegridder._regrid?

PS: the linkchecker action seems to fail for something unrelated - the website https://fesom.de (linked here) seems to be unreachable.

[trexfeathers]

Also the UnstructuredNearest regridder does not support lazy data (here is a good summary of regridders: Cube Interpolation and Regridding — Iris 3.11.0 documentation

[trexfeathers]

Just wanted to double check, but I think we should also remove the subsections on "caching an interpolator" and "caching a regridder": as discussed in the thread below #6084 (comment), almost all the work is done when calling the interpolator (within RectilinearInterpolator._interpolate) so there seem to be no actual benefit in constructing and reusing the interpolator objects? And I think this is also true for the regridder, for which all the work seems to done within RectilinearRegridder._regrid?

Re-using regridders

Please do NOT remove the section about re-using a regridder. This receives much more attention from the team and we know that it provides important performance benefits - the preparation step is more expensive for regridding.

Re-using interpolators

Thanks for the heads-up: I was not aware that interpolator re-use was an advertised feature (we talk a lot more about regridder re-use).

Since it's an advertised feature, I raised this with @SciTools/peloton to make sure we are happy with losing this functionality:

• We confirmed that re-use code does not break, which is essential. The only change for the user is that re-use no longer provides a benefit.
• It is believable that the performance benefits of re-use are not as great as with regridding - we are not caching weights here, but a SciPy object.
• The ideal situation would be parallelisation AND re-use, but if there is a choice then parallelisation wins.
• There are future plans (perhaps @stephenworsley can provide cross-references here) to use modern matrix-multiplication for all regridding and interpolation, which makes us more comfortable with the possibility that performance still isn't the best for all cases - it will get even better in the long-term!

So we're still 👍 on removing the re-use functionality. Outstanding work for you:

• Remove this documentation section as you suggest.
• Share in the What's New entry that this is no longer a performance feature (although the suggested code will still work).