Repository navigation
memory usage in histogram2d(). #197
Description
The loading is occurring fine like "load("TNG-Cluster")" and I was able to plot galaxy stellar mass vs star formation.
While reading the group catalogue fields, it takes around 6GB of space.
grp_cat = snap.data["Group"]
print("available group catalog fields:\n", grp_cat.keys())Now the kernel is dying while "Request a histogram2d() calculation, summing the gas masses across the entire simulaton box":
weights = weights.rechunk(x.chunks)
grid, _, _ = da.histogram2d(x, y, weights=weights, bins=nPixels, range=[minMax,minMax])
grid = grid.compute() # do actual calculation; might take some time
Activity
dnelson commentedon Mar 27, 2025
As a general question: is it possible to limit the memory usage (in particular, of dask operations) to a user-specified limit?
cbyrohl commentedon Mar 28, 2025
@Saqib53GB Could you activate the distributed backend with a LocalCluster, limiting the memory usage to 4GB? See https://scida.io/largedatasets/ and https://distributed.dask.org/en/stable/quickstart.html . Maybe something like
prior to any dask operations. Also consider https://docs.dask.org/en/latest/dashboard.html for further diagnostics.
I assume you are doing this on the TNGLab. We need to find reasonable defaults for the LocalCluster as well as the chunksize for TNGLab @dnelson86 . Could you check the dask distributed version installed on the lab and whether the queueing (default in recent versions) is default? I think this might reduce the memory requirement for reduction operations such as histograms quite a bit. Queuing policy: https://distributed.dask.org/en/stable/scheduling-policies.html#queuing
dnelson commentedon Mar 28, 2025
(1) Could we add to the documentation an example of how to limit memory usage to a specific amount?
(2) Is there a way via e.g. user configuration files to set a default behavior (on the TNG Lab) to respect such a limit?