Skip to content

Update model lists - #4420

Open
bjarzemb wants to merge 12 commits into
mainfrom
barjarz/update_model_list
Open

Update model lists#4420
bjarzemb wants to merge 12 commits into
mainfrom
barjarz/update_model_list

Conversation

@bjarzemb

@bjarzemb bjarzemb commented Jul 22, 2026

Copy link
Copy Markdown
Contributor

Changes

1. Splitted priority lists per run configuration

Restructure each priority (p0, p1, p2) from a single flat model list into
per-run-configuration lists so that accuracy and performance, inference and
training can each carry their own model set:

p{0,1,2}/
  accuracy/
    inference/{torchbench,huggingface,timm}_models_list.txt
    training/{torchbench,huggingface,timm}_models_list.txt
  performance/
    inference/{torchbench,huggingface,timm}_models_list.txt
    training/{torchbench,huggingface,timm}_models_list.txt

2. Added new models to priority list

  • huggingface (P0)
    • OPTForCausalLM (ALL)
    • XGLMForCausalLM (ALL)
    • M2M100ForConditionalGeneration (acc/inf, perf/inf)
  • timm models (P2)
    • cait_m36_384 (ALL)
    • xcit_large_24_p8_224 (ALL)
  • torchbench (P0)
    • torchrec_dlrm (ALL)
    • demucs (ALL)
    • detectron2_maskrcnn_r_50_c4 (ALL)
    • llava (ALL)
    • hf_BigBird (ALL)
    • hf_Longformer (ALL)
    • detectron2_fcos_r_50_fpn (acc/inf, perf/inf)
    • llama_v2_7b_16h (acc/inf, perf/inf)
    • moondream (acc/inf, perf/inf)
    • cm3leon_generate (perf/inf)

3. Removed models from priority list (because they are explicitly skipped in benchmarks/dynamo/torchbench.yaml:

  • stable_diffusion_text_encoder
  • stable_diffusion_unet
  • doctr_det_predictor (acc/train, perf/train)
  • doctr_reco_predictor (acc/train, perf/train)
  • maml (acc/train, perf/train)
  • pyhpc_equation_of_state (acc/train, perf/train)
  • pyhpc_isoneutral_mixing (acc/train, perf/train)
  • pyhpc_turbulent_kinetic_energy (acc/train, perf/train)
  • sam_fast (acc/train, perf/train)

4. Removed duplicate HuggingFace entries from top-level list

  • mistralai/Mistral-7B-Instruct-v0.3
  • openai/gpt-oss-20b

@github-actions

Copy link
Copy Markdown

Performance outliers, please check!

  • 🟡 [80%, 90%), may be fluctuations
Category Model Target vs. Baseline [Eager] Target vs. Baseline [Inductor]
huggingface_bfloat16_training BlenderbotForCausalLM 0.869883 0.880032
huggingface_bfloat16_training DistilBertForMaskedLM 0.909026 0.883223

@bjarzemb
bjarzemb force-pushed the barjarz/update_model_list branch from 81fd745 to 110ffa5 Compare July 22, 2026 13:05
@github-actions

Copy link
Copy Markdown

Performance outliers, please check!

  • 🟡 [80%, 90%), may be fluctuations
Category Model Target vs. Baseline [Eager] Target vs. Baseline [Inductor]
huggingface_bfloat16_training DistilBertForMaskedLM 0.883406 0.894316
huggingface_bfloat16_training BlenderbotForCausalLM 0.890704 0.905839

@chuanqi129
chuanqi129 requested a review from mengfei25 July 23, 2026 02:15
@github-actions github-actions Bot added the disable_all Disable all ci test jobs for the PR, just keep basic lint check label Jul 23, 2026
@chuanqi129 chuanqi129 added disable_ut Disable all non-distributed UT test jobs for the PR disable_distributed Disable distributed UT test jobs for the PR disable_build Disable source code build for CI test, use nightly wheel disable_auto Disable auto label workflow for PR and removed disable_all Disable all ci test jobs for the PR, just keep basic lint check labels Jul 27, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

disable_auto Disable auto label workflow for PR disable_build Disable source code build for CI test, use nightly wheel disable_distributed Disable distributed UT test jobs for the PR disable_ut Disable all non-distributed UT test jobs for the PR

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants