Hi,
I am not a user of pbdff but an administrator of an HPC system where it is used.
As I understand it, when used with a scheduler, the tool creates individual jobs with identical resource requirements rather than what on a system using Slurm would be called a job array. The problem with this is that it can significantly reduce the effectiveness of backfill, even to the point of essentially disabling it. Slurm can check waiting jobs to see whether gaps in the schedule can be filled with jobs that would otherwise have to wait. However, only a finite number of jobs are checked and checks on multiple jobs with the the same resource requirements are redundant. With a job array, Slurm can just check a single element of the array and then move on to the next job or job array.
It is obviously going to be a bit of a pain to implement job arrays, since I assume that PBS has something similar but different. However, if backfill does become disabled by jobs run via pbdff, HPC admins may consider penalising such jobs in order to maintain the total throughput of their system. So I hope addressing this issue will be in your interest and that of users of pbdff too.
Hi,
I am not a user of
pbdffbut an administrator of an HPC system where it is used.As I understand it, when used with a scheduler, the tool creates individual jobs with identical resource requirements rather than what on a system using Slurm would be called a job array. The problem with this is that it can significantly reduce the effectiveness of backfill, even to the point of essentially disabling it. Slurm can check waiting jobs to see whether gaps in the schedule can be filled with jobs that would otherwise have to wait. However, only a finite number of jobs are checked and checks on multiple jobs with the the same resource requirements are redundant. With a job array, Slurm can just check a single element of the array and then move on to the next job or job array.
It is obviously going to be a bit of a pain to implement job arrays, since I assume that PBS has something similar but different. However, if backfill does become disabled by jobs run via
pbdff, HPC admins may consider penalising such jobs in order to maintain the total throughput of their system. So I hope addressing this issue will be in your interest and that of users ofpbdfftoo.