Skip to content

feat: run FastSim from a job trace + slurm.conf only (optional dumps, synthesized associations/QOS) #4

Description

@kevinmenear

What / Why

Running FastSim currently requires seven SLURM dumps, most of which need privileged access (sacctmgr, sinfo, sreport on a live cluster). Most researchers only have a historical job trace (e.g., from a database of job records) and can usually obtain the site's slurm.conf. This feature makes those two inputs sufficient to run the simulator, while every other dump remains usable when available.

Design

Required inputs: job_dump (sacct-style pipe-delimited trace), slurm_conf, considered_partitions, sim_start, sim_end.

Optional inputs (each independently; used when provided, synthesized/empty when not):

Input When absent
assocs_dump Synthesize a flat fairshare tree from the trace: root → one account per unique Account → one user per unique (User, Account), all with identical Shares=1 (users generally can't see real allocation awards). Partition=NaN so each user assoc covers all partitions (matches existing _load_tree_slurm semantics). No MaxJobs/MaxSubmit → no association limits.
qos_dump Synthesize QOS entries for every QOS name referenced by the trace, plus every partition QOS= name from slurm.conf, plus normal (the partition default). Equal priority, all limits None → no QOS holds.
node_events_dump Empty events frame → no down/drain events.
resv_dump_current / resv_dump_historic Each independently optional; empty reservation frame when neither. Queue._clean_reservations already clears job reservation args not in valid_resv, so jobs that referenced reservations run as normal jobs (marked ignore_in_eval).

Job trace column tolerance: get_cleaned_job_df gains a required-column check with a clear error (JobID, Submit, Start, End, State, Partition, User, Account, ReqNodes, AllocNodes, Timelimit) and tolerates missing optional columns: QOS (→ normal), ConsumedEnergyRaw (→ triggers existing zero-power fallback), JobName/Reason (→ None), SubmitLine (already optional).

Key implementation points:

  • config.py: shrink mandatory_fields; add None defaults for the five optional dump paths, [] for impromptu_reservation_names, "default" for system, so the config namedtuple always has the attributes.
  • data_reader.py: empty-frame paths for events/reservations; synthesize_qos(names); synthesize_assocs(df_jobs) producing the same User|Account|ParentName|Partition|Shares frame _load_tree_slurm consumes.
  • fairshare.py: _load_tree_slurm accepts a path or a prepared DataFrame.
  • controller.py: pass QOS name universe to get_qos and assoc source (path or synthesized frame) to FairTree.
  • Full-dump behavior must be byte-for-byte unchanged (verified against baseline via scripts/compare_results.py).

Subtasks

  • Config: optional dump paths with defaults; shrunk mandatory set; sim_start/sim_end mandatory
  • data_reader: optional node events and reservation dumps (each reservation dump independently optional)
  • data_reader + controller: QOS synthesis when qos_dump absent
  • data_reader + fairshare + controller: association synthesis (flat tree, identical shares) when assocs_dump absent
  • data_reader: job-trace column tolerance (required-column validation + optional-column defaults)
  • Example minimal config + README documentation
  • Verification: trace-only run end-to-end; full-dump baseline regression (identical results); partial-data smoke run

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions