perf: clone extracted video frames to prevent DataLoader shared memory bloat

[deep9539]

The Problem: When using PyTorch's DataLoader with multiple workers (num_workers > 0), tensors yielded by the dataset are passed to the main process via shared memory (/dev/shm).

Currently, extracting frames via video.get_frames_at(_indices(n)).data creates a memory view of the original decoded video with the selected indices given by _indices(n). PyTorch's shared memory serialization mechanism does not slice the underlying storage; it copies the entire underlying memory block. Because decoded videos can easily be hundreds of megabytes, passing views of them causes the entire video to be duplicated in /dev/shm.

This leads to a few critical issues:

• Shared Memory Exhaustion: /dev/shm fills up almost immediately, often leading to RuntimeError: shared memory is full.
• Bottlenecked Concurrency: To avoid crashes, we are forced to artificially lower batch_size and num_workers.
• GPU Underutilization: Because data loading is bottlenecked by the restricted worker count, the GPU is starved for data and severely underutilized.

The Solution: This PR appends .clone() to the extracted frame tensor.

By cloning the tensor, PyTorch allocates a new, tightly packed memory block containing only the specific frames requested by the indices.

Impact / Trade-offs: While .clone() does introduce a small computational overhead to copy the memory on the CPU, it is a highly favorable trade-off particularly when max_frames count is very low compared to total_frames.
Drastic Memory Reduction: We only store the actual batch data in shared memory instead of the entire source videos.
Higher Throughput: This memory optimization allows us to safely increase num_workers and batch_size, eliminating the CPU data-loading bottleneck and keeping the GPU fully fed.
Testing:

Possible further optimization: We can consider doing this only when max_frames << total_frames for the video or something similar.

Verification done on docker environment with few other unrelated changes:

• Verified that RuntimeError regarding shared memory is no longer thrown at higher worker counts with restricted /dev/shm memory.
• Monitored /dev/shm usage during training to confirm the footprint remains stable.

Actual verfication pending

[KennethEnevoldsen]

@deep9539 can you provide some sort of validation on this in practice - what is the tradeoff that we are looking at here?

runtime on slurm, runtime not on slurm, memory etc.

[Samoed]

Can you add a small comment why we do clone?

[KennethEnevoldsen]

@deep9539 just a friendly ping on this PR