Skip to content

Speed up landice.mesh.preprocess_ais_data extrapolation#949

Draft
trhille wants to merge 1 commit intoMPAS-Dev:mainfrom
trhille:landice/speed_up_gridded_extrapolation
Draft

Speed up landice.mesh.preprocess_ais_data extrapolation#949
trhille wants to merge 1 commit intoMPAS-Dev:mainfrom
trhille:landice/speed_up_gridded_extrapolation

Conversation

@trhille
Copy link
Copy Markdown
Collaborator

@trhille trhille commented Mar 26, 2026

Speed up extrapolation step in landice.mesh.preprocess_ais_data by replacing scipy.interpolate.NearestNDInterpolator with smarter algorithm using scipy.ndimage.distance_transform_edt.

Checklist

  • User's Guide has been updated
  • Developer's Guide has been updated
  • API documentation in the Developer's Guide (api.rst) has any new or modified class, method and/or functions listed
  • Documentation has been built locally and changes look as expected
  • The E3SM-Project submodule has been updated with relevant E3SM changes
  • The MALI-Dev submodule has been updated with relevant MALI changes
  • Document (in a comment titled Testing in this PR) any testing that was used to verify the changes
  • New tests have been added to a test suite

Speed up extrapolation step in landice.mesh.preprocess_ais_data
by replacing scipy.interpolate.NearestNDInterpolator with
smarter algorithm using scipy.ndimage.distance_transform_edt.
@trhille
Copy link
Copy Markdown
Collaborator Author

trhille commented Mar 26, 2026

Testing

Using current implementation in main with antarctica_1km_2024_01_29.nc input file, we get this:

calling preprocess_ais_data
Beginning building interpolator for thk
Finished building interpolator in 9.447678919066675 seconds
Beginning interpolation for thk
Interpolation completed in 529.3699218650581 seconds
Beginning building interpolator for bheatflx
Finished building interpolator in 9.67620110407006 seconds
Beginning interpolation for bheatflx
Interpolation completed in 501.6405820630025 seconds
Beginning building interpolator for vx
Finished building interpolator in 9.535582176060416 seconds
Beginning interpolation for vx
Interpolation completed in 540.0124549659668 seconds
Beginning building interpolator for vy
Finished building interpolator in 9.551368530956097 seconds
Beginning interpolation for vy
Interpolation completed in 539.4295184250223 seconds
Beginning building interpolator for ex
Finished building interpolator in 9.488464000984095 seconds
Beginning interpolation for ex
Interpolation completed in 561.9257037510397 seconds
Beginning building interpolator for ey
Finished building interpolator in 9.515914903022349 seconds
Beginning interpolation for ey
Interpolation completed in 560.7867423400749 seconds
Beginning building interpolator for thkerr
Finished building interpolator in 9.296070468961261 seconds
Beginning interpolation for thkerr
Interpolation completed in 526.2319564099889 seconds
Beginning building interpolator for dhdt
Finished building interpolator in 8.225148725905456 seconds
Beginning interpolation for dhdt
Interpolation completed in 328.5181841789745 seconds
All interpolations completed in 4162.666724437033 seconds.

Using these changes, we get this:

calling preprocess_ais_data
Beginning nearest-fill preprocessing for thk
thk: 13301973 valid cells, 31146916 cells to fill
Nearest-fill preprocessing for thk completed in 3.007 seconds
Beginning nearest-fill preprocessing for bheatflx
bheatflx: 13313571 valid cells, 31135318 cells to fill
Nearest-fill preprocessing for bheatflx completed in 2.935 seconds
Beginning nearest-fill preprocessing for vx
vx: 13278353 valid cells, 31170536 cells to fill
Nearest-fill preprocessing for vx completed in 2.995 seconds
Beginning nearest-fill preprocessing for vy
vy: 13278353 valid cells, 31170536 cells to fill
Nearest-fill preprocessing for vy completed in 3.006 seconds
Beginning nearest-fill preprocessing for ex
ex: 13201865 valid cells, 31247024 cells to fill
Nearest-fill preprocessing for ex completed in 2.999 seconds
Beginning nearest-fill preprocessing for ey
ey: 13201861 valid cells, 31247028 cells to fill
Nearest-fill preprocessing for ey completed in 3.004 seconds
Beginning nearest-fill preprocessing for thkerr
thkerr: 13301973 valid cells, 31146916 cells to fill
Nearest-fill preprocessing for thkerr completed in 2.986 seconds
Beginning nearest-fill preprocessing for dhdt
dhdt: 11333700 valid cells, 33115189 cells to fill
Nearest-fill preprocessing for dhdt completed in 2.728 seconds
All nearest-fill preprocessing completed in 23.674 seconds.

So, a 200x speedup.
I still need to verify that these changes are producing the expected behavior.

@matthewhoffman matthewhoffman added land ice in progress This PR is not ready for review or merging labels Mar 29, 2026
@matthewhoffman matthewhoffman marked this pull request as draft March 29, 2026 16:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

in progress This PR is not ready for review or merging land ice

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants