Issues

• Scoring: model-graded scorer template is exploitable

  UKGovernmentBEIS/inspect_ai#6277 · Filter by author MattFisher opened on Nov 3, 2026

• exact() and f1() split a negative exponent, so 1e-3 misses 0.001 and -1e-3 matches 1e-3

  UKGovernmentBEIS/inspect_ai#6268 · Filter by author chrikrah opened on Nov 2, 2026

• Tool effect commits after time_limit/working_limit; scorer and log miss it

  UKGovernmentBEIS/inspect_ai#6262 · Filter by author astrogilda opened on Nov 2, 2026

• docs: add mlx-lm (Apple Silicon) recipe and AWS_PROFILE note to providers page

  UKGovernmentBEIS/inspect_ai#6246 · Filter by author tefraz opened on Nov 2, 2026

• OpenAI native compaction crashes when a user message contains an image or file

  UKGovernmentBEIS/inspect_ai#6236 · Filter by author tim-hua-01 opened on Nov 2, 2026

• match(numeric=True) does not trim full-width punctuation, so 42。 or （42） scores INCORRECT

  UKGovernmentBEIS/inspect_ai#6235 · Filter by author Socrates10101 opened on Nov 2, 2026

• Tool parameters that default to an Enum member fail every sample with TypeError: ... is not JSON serializable

  UKGovernmentBEIS/inspect_ai#6230 · Filter by author ShaanGS opened on Nov 1, 2026

• samples_df() uses drop_duplicates(inplace=True), deprecated in pandas 3.1

  UKGovernmentBEIS/inspect_ai#6226 · Filter by author kaanutkuy opened on Nov 1, 2026

• eval_async resets caller-owned Rich Live/Progress state, causing a post-eval IndexError

  UKGovernmentBEIS/inspect_ai#6225 · Filter by author ndrew1337 opened on Oct 31, 2026
  1

• match(numeric=True) reads a number grouped with typographic spaces (U+00A0, U+202F, U+2009, e.g. "2 500") as its last group (500)

  UKGovernmentBEIS/inspect_ai#6223 · Filter by author ChrystianSchutz opened on Oct 31, 2026
  1

• pass_at accepts negative k and reports negative probabilities

  UKGovernmentBEIS/inspect_ai#6222 · Filter by author deavgprime opened on Oct 31, 2026
  1

• json_dataset drops image content internal data while resolving paths

  UKGovernmentBEIS/inspect_ai#6221 · Filter by author deavgprime opened on Oct 31, 2026

• Filtering a sample with a superscript-digit string ID raises ValueError

  UKGovernmentBEIS/inspect_ai#6220 · Filter by author deavgprime opened on Oct 31, 2026

• grouped() silently overwrites groups when name_template produces duplicate labels

  UKGovernmentBEIS/inspect_ai#6219 · Filter by author deavgprime opened on Oct 31, 2026

• pattern(ignore_case=True) rejects Unicode caseless matches after extraction

  UKGovernmentBEIS/inspect_ai#6218 · Filter by author deavgprime opened on Oct 31, 2026

• exact() and f1() award full credit to distinct large integers after float normalization

  UKGovernmentBEIS/inspect_ai#6216 · Filter by author deavgprime opened on Oct 31, 2026

• eval(sample_shuffle=0) doesn't shuffle, but eval_retry() does and reruns different samples

  UKGovernmentBEIS/inspect_ai#6203 · Filter by author Mramaran opened on Oct 30, 2026
  1

• Extension task registration: editable installs lose their package namespace when the source directory precedes site-packages on sys.path

  UKGovernmentBEIS/inspect_ai#6202 · Filter by author scks1 opened on Oct 29, 2026

• Dataset media paths are resolved relative to the dataset file only for user messages

  UKGovernmentBEIS/inspect_ai#6197 · Filter by author BlueX888 opened on Oct 29, 2026
  1

• turn_count is 0 when a token limit trips, although the generation completed

  UKGovernmentBEIS/inspect_ai#6194 · Filter by author BlueX888 opened on Oct 29, 2026
  1

• Replay refuses logs where two agents reuse a tool call id

  UKGovernmentBEIS/inspect_ai#6193 · Filter by author dragonstyle opened on Oct 29, 2026

• Re-score only the samples whose scorer failed, without re-running the model

  UKGovernmentBEIS/inspect_ai#6178 · Filter by author MattFisher opened on Oct 29, 2026

• ScoreEdit(reason=None) reloads as reason='UNCHANGED' in Score.history

  UKGovernmentBEIS/inspect_ai#6177 · Filter by author MattFisher opened on Oct 29, 2026

• retry_on_error with checkpointing re-runs the agent after a scorer error, and the retry's checkpoints fail

  UKGovernmentBEIS/inspect_ai#6176 · Filter by author MattFisher opened on Oct 29, 2026

• Path and set task arguments lose values in logs and retries

  UKGovernmentBEIS/inspect_ai#6171 · Filter by author vayllon301 opened on Oct 29, 2026