Skip to content

add_inactive_sensors=True yields -inf charge for IceCube detectors (log10(0)) #903

Description

@sevmag

Describe the bug

When add_inactive_sensors=True is used with the IceCube Detector classes that standardize charge with a bare log10(x) (IceCube86, IceCubeKaggle, IceCubeUpgrade), the resulting node features contain -inf in the charge column.

Inactive sensors are padded from the geometry table in DataRepresentation._attach_inactive_sensors, and the IceCube geometry tables store charge = 0 for every sensor. Those zero charges then go through _charge:

def _charge(self, x: torch.tensor) -> torch.tensor:
    return torch.log10(x)   # log10(0) = -inf

so every inactive sensor becomes -inf. The forward pass does not raise, but it hands the model non-finite inputs, which NaN-out the loss/gradients as soon as you train on it.

This is independent of #898: it triggers with add_inactive_sensors alone, without any perturbation — but it shares the same root cause (log10 of a non-positive charge).

To Reproduce

Minimal and self-contained — no dataset needed, the single active pulse is built straight from the geometry table:

import torch
from graphnet.models.graphs import GraphDefinition
from graphnet.models.detector import IceCube86

detector = IceCube86()
features = list(detector.feature_map().keys())
charge_idx = features.index("charge")

# A single real pulse: one sensor from the geometry table, given a positive charge.
x = detector.geometry_table[features].to_numpy(dtype=float)[:1]
x[:, charge_idx] = 1.0

graph = GraphDefinition(detector=detector, add_inactive_sensors=True)(x, features)

charge = graph.x[:, charge_idx]
print("total nodes        :", graph.x.shape[0])
print("non-finite charges :", int((~torch.isfinite(charge)).sum()))
print("all charges finite :", bool(torch.isfinite(charge).all()))

Output on main (ced21ab):

total nodes        : 5407
non-finite charges : 5406
all charges finite : False

Expected behavior

Padded inactive sensors (charge 0) should map to a finite value. Flooring the charge before log10 — as proposed in #899, torch.log10(torch.clamp(x, min=1e-2)) — keeps the transform identical for all valid charges and maps inactive sensors to the bottom of the charge scale (log10(1e-2) = -2) instead of -inf.

Affected detectors

  • IceCube86, IceCubeKaggle, IceCubeUpgrade — all use bare log10(x) and produce -inf.
  • IceCubeDeepCore — uses the identity for charge, unaffected.
  • NuBenchDetector — its geometry tables have no charge column, so add_inactive_sensors raises KeyError: "['charge'] not in index" instead; that is a separate problem.

Related: #898, #899

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions