On small netowrk (ResNet 18) data augmentation on CPU will be the bottleneck and the GPU will not be used fully leading to funky solutions like: https://www.sagivtech.com/2017/09/19/optimizing-pytorch-training-code/ (which uses multiprocessing to bypass Python GIL and resort to killing spawned thread)
Image loading need to be fast. Benchmarks like https://t0rakka.silvrback.com/jpeg-decoding-benchmark shows that libjpeg-turbo can be a bottleneck. See repo https://github.com/t0rakka/mango/tree/master/source/mango/jpeg and Nvidia nvJPEG https://developer.nvidia.com/nvjpeg and Nvidia DALI (Data Augmentation Library) https://github.com/NVIDIA/DALI.
Alternative libraries to look at:
On small netowrk (ResNet 18) data augmentation on CPU will be the bottleneck and the GPU will not be used fully leading to funky solutions like: https://www.sagivtech.com/2017/09/19/optimizing-pytorch-training-code/ (which uses multiprocessing to bypass Python GIL and resort to killing spawned thread)
Image loading need to be fast. Benchmarks like https://t0rakka.silvrback.com/jpeg-decoding-benchmark shows that libjpeg-turbo can be a bottleneck. See repo https://github.com/t0rakka/mango/tree/master/source/mango/jpeg and Nvidia nvJPEG https://developer.nvidia.com/nvjpeg and Nvidia DALI (Data Augmentation Library) https://github.com/NVIDIA/DALI.
Alternative libraries to look at: