Docker containers need to update netcdf

[fmaussion]

Unidata/netcdf-c#2674 has been hitting us. I will try a few things and get back here - this is more to remember about it

[fmaussion]

This is confirmed. All containers with netcdf-c 4.9.2 are unusable on cluster. They clutter the logs with millions of:

DF5-DIAG: Error detected in HDF5 (1.10.10) thread 1:
  #000: ../../../src/H5A.c line 484 in H5Aopen_by_name(): can't open attribute
    major: Attribute
    minor: Can't open object
  #001: ../../../src/H5Aint.c line 542 in H5A__open_by_name(): unable to load attribute info from object header
    major: Attribute
    minor: Unable to initialize object
  #002: ../../../src/H5Oattribute.c line 496 in H5O__attr_open_by_name(): can't locate attribute: '_QuantizeBitGroomNumberOfSignificantDigits'
    major: Attribute
    minor: Object not found

and end up stalling the run. netcdf-c 4.9.3 has a fix.

On cluster I managed to get rid of the cluttered logs with pip install --quiet --force-reinstall --no-deps netCDF4

@TimoRoth bringing this to your attention for when you are back.

[TimoRoth]

Annoyingly, netcdf in the docker images comes straight from Ubuntu, so I cannot update or downgrade it.
Someone from NetCDF will have to bug Ubuntu about backporting the fix.

[TimoRoth]

Actually, checking on https://packages.ubuntu.com/resolute/libnetcdf-dev says it already is on v4.9.3.
And the changelog at https://changelogs.ubuntu.com/changelogs/pool/universe/n/netcdf/netcdf_4.9.3-1build2/changelog says it's been on that version since February already, long before I switched the Docker images to 26.04. So this issue should never have been present in our docker images, since they jumped from whatever older version was in 25.10 straight to the fixed version.

[TimoRoth]

Ah, the problem is that the tests haven't been passing in the latest base images since ages.
So the oggm/oggm:latest image keeps using a really old base image, the last one the tests passed in months ago.
And that one happens to have v4.9.2 on Ubuntu 24.04.

I can forcibly update the base image to the latest one from Ubuntu 26.04. Since I think the tests are broken on the old image in the same way anyway.
Or we need to fix those tests, most of which likely just require new baseline images.

[fmaussion]

Thanks for investigating. you can let @gampnico or me know if you need help with the tests.

[TimoRoth]

Nicolas said he's already working on fixing the tests a week or two ago.

[gampnico]

I updated the baseline images last week here: OGGM/oggm-sample-data#28

I can regenerate some new images with the exact Docker containers used, but I think it would be more useful in the future to look at ways of ignoring minor formatting changes when comparing images.

[TimoRoth]

There already should be quite large tolerances in those tests, but the various libraries keep changing how stuff looks.
Maybe we should change the tests to simply verify the data we pass to matplotlib is what we expect, and don't actually generate any images?
We're mostly testing matplotlib, and not so much OGGM there.

[fmaussion]

We're mostly testing matplotlib, and not so much OGGM there.

True - the tests are purely for OGGM's plot* functions, the underlying data is tested elsewhere. But getting rid of the tests entirely is a bit risky - we can chat with Nico

[gampnico]

The fastest would be to use the remove_text argument in the decorator, or strip all text entirely. We could then pass expected axis labels etc. into separate tests, which is what I've done in the past. But I think this would be a nightmare to maintain given the number of test images.

There is the pytest-regressions that might be more useful for this?

[TimoRoth]

Seems like the images test fine again with the new baselines.
Just a random other breakage that coincided perfectly with us fixing our tests prevented new ones from publishing.
That is worked around for now, until cloudpickle releases a fixed version.
So the next set of oggm images should work fine again.

[TimoRoth]

Latest images built and tested fine, and I also swapped them to Python 3.14.
It completes the full test suite in ~35 instead of ~45 minutes on 3.13, so it seems a decent improvement in terms of performance for OGGM.
So the netCDF issue should also be gone alongside that.