What / Why
Running FastSim currently requires seven SLURM dumps, most of which need privileged access (sacctmgr, sinfo, sreport on a live cluster). Most researchers only have a historical job trace (e.g., from a database of job records) and can usually obtain the site's slurm.conf. This feature makes those two inputs sufficient to run the simulator, while every other dump remains usable when available.
Design
Required inputs: job_dump (sacct-style pipe-delimited trace), slurm_conf, considered_partitions, sim_start, sim_end.
Optional inputs (each independently; used when provided, synthesized/empty when not):
| Input |
When absent |
assocs_dump |
Synthesize a flat fairshare tree from the trace: root → one account per unique Account → one user per unique (User, Account), all with identical Shares=1 (users generally can't see real allocation awards). Partition=NaN so each user assoc covers all partitions (matches existing _load_tree_slurm semantics). No MaxJobs/MaxSubmit → no association limits. |
qos_dump |
Synthesize QOS entries for every QOS name referenced by the trace, plus every partition QOS= name from slurm.conf, plus normal (the partition default). Equal priority, all limits None → no QOS holds. |
node_events_dump |
Empty events frame → no down/drain events. |
resv_dump_current / resv_dump_historic |
Each independently optional; empty reservation frame when neither. Queue._clean_reservations already clears job reservation args not in valid_resv, so jobs that referenced reservations run as normal jobs (marked ignore_in_eval). |
Job trace column tolerance: get_cleaned_job_df gains a required-column check with a clear error (JobID, Submit, Start, End, State, Partition, User, Account, ReqNodes, AllocNodes, Timelimit) and tolerates missing optional columns: QOS (→ normal), ConsumedEnergyRaw (→ triggers existing zero-power fallback), JobName/Reason (→ None), SubmitLine (already optional).
Key implementation points:
config.py: shrink mandatory_fields; add None defaults for the five optional dump paths, [] for impromptu_reservation_names, "default" for system, so the config namedtuple always has the attributes.
data_reader.py: empty-frame paths for events/reservations; synthesize_qos(names); synthesize_assocs(df_jobs) producing the same User|Account|ParentName|Partition|Shares frame _load_tree_slurm consumes.
fairshare.py: _load_tree_slurm accepts a path or a prepared DataFrame.
controller.py: pass QOS name universe to get_qos and assoc source (path or synthesized frame) to FairTree.
- Full-dump behavior must be byte-for-byte unchanged (verified against baseline via
scripts/compare_results.py).
Subtasks
What / Why
Running FastSim currently requires seven SLURM dumps, most of which need privileged access (
sacctmgr,sinfo,sreporton a live cluster). Most researchers only have a historical job trace (e.g., from a database of job records) and can usually obtain the site's slurm.conf. This feature makes those two inputs sufficient to run the simulator, while every other dump remains usable when available.Design
Required inputs:
job_dump(sacct-style pipe-delimited trace),slurm_conf,considered_partitions,sim_start,sim_end.Optional inputs (each independently; used when provided, synthesized/empty when not):
assocs_dumpAccount→ one user per unique(User, Account), all with identicalShares=1(users generally can't see real allocation awards).Partition=NaNso each user assoc covers all partitions (matches existing_load_tree_slurmsemantics). NoMaxJobs/MaxSubmit→ no association limits.qos_dumpQOS=name from slurm.conf, plusnormal(the partition default). Equal priority, all limitsNone→ no QOS holds.node_events_dumpresv_dump_current/resv_dump_historicQueue._clean_reservationsalready clears job reservation args not invalid_resv, so jobs that referenced reservations run as normal jobs (markedignore_in_eval).Job trace column tolerance:
get_cleaned_job_dfgains a required-column check with a clear error (JobID, Submit, Start, End, State, Partition, User, Account, ReqNodes, AllocNodes, Timelimit) and tolerates missing optional columns:QOS(→normal),ConsumedEnergyRaw(→ triggers existing zero-power fallback),JobName/Reason(→None),SubmitLine(already optional).Key implementation points:
config.py: shrinkmandatory_fields; addNonedefaults for the five optional dump paths,[]forimpromptu_reservation_names,"default"forsystem, so the config namedtuple always has the attributes.data_reader.py: empty-frame paths for events/reservations;synthesize_qos(names);synthesize_assocs(df_jobs)producing the sameUser|Account|ParentName|Partition|Sharesframe_load_tree_slurmconsumes.fairshare.py:_load_tree_slurmaccepts a path or a prepared DataFrame.controller.py: pass QOS name universe toget_qosand assoc source (path or synthesized frame) toFairTree.scripts/compare_results.py).Subtasks
sim_start/sim_endmandatoryqos_dumpabsentassocs_dumpabsent