Upgrading transformers version to 4.51.3 to support recent models by filyp · Pull Request #175 · locuslab/open-unlearning

filyp · 2026-01-26T15:38:52Z

What does this PR do?

It upgrades transformers to 4.51.3, to support newer models like gemma3 and qwen3
Makes UnlearnTrainer implementation more future proof. (See Upgrading transformers version to support recent models #173)
Makes other necessary changes to be compatible with new transformers version
- Adds num_items_in_batch to compute_loss signature
- Prevent exceptions from evaluating when eval_dataset=None
- Uses trainer.processing_class instead of trainer.tokenizer (it's depracated and transformers==5 removes it)

Additionally it:

Simplifies the installation of lm_eval (no need to have a special install group in setup.py)
Minor fix to the leaderboard doc
Makes .gitignore more comprehensive

Related issues: #173 and #155

Before submitting

This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
Have you gone through the contributions guide?
Are your changes documented? Read documentation guidelines here.

Tests

I manually tested the new evaluation, with removed prediction_step from UnlearninTrainer, and it works the same.

When I tested unlearning, the unlearning trajectories are exactly the same when using gradient_accumulation_steps=1. But when it's =/=1, the upgrade changes the scale of the logged training/loss (it's 4x higher.), and subtly changes the unlearning trajectory. This most likely comes from the gradient accumulation fix in transformers=4.46.

(Tested unlearning with this command:

python src/train.py --config-name=unlearn.yaml experiment=unlearn/tofu/default eval=tofu_simple question_key=paraphrased_question eval.tofu.batch_size=16 trainer.args.report_to=wandb trainer=NPO task_name=...

Where tofu_simple is a config with fewer eval metrics.)

compute_loss docstring now states:

        Subclass and override for custom behavior. If you are not using `num_items_in_batch` when computing your loss,
        make sure to overwrite `self.model_accepts_loss_kwargs` to `False`. Otherwise, the loss calculating might be slightly inaccurate when performing gradient accumulation.

When I tried setting trainer.model_accepts_loss_kwargs = False it restores the previous scale of the training loss, but it doesn't affect the unlearning trajectory at all, only that logged loss scale.

make UnlearnTrainer implementation more future proof make other necessary changes to be compatible with new transformers version

fix leaderboard docs wider gitignore

… transformers==5

filyp · 2026-02-08T12:09:00Z

I was actually planning to upgrade transformers to 5.0.0, because it has some major MoE optimizations + I'm working on an unlearning method that depends on the MoE implementation, so I'd rather already work on the better implementation.

But I'd rather wait until you merge or at least review this upgrade to 4.51.3, because I don't want to change too many things at once. What do you think?

molereddy · 2026-02-11T05:27:28Z

Looks great!! Can you clean up the linter errors?

filyp · 2026-02-11T13:02:22Z

Ah, right, fixed now.

molereddy · 2026-02-18T15:03:18Z

Seems the tests are still failing -- maybe a version mismatch? Our instructions for formatting are here. Let me know if you are still stuck!

filyp · 2026-02-18T17:23:31Z

Ah I see locally it passed for me because I had a newer ruff version. I reformatted to the one that the github action uses, so should be good now

Dornavineeth · 2026-02-22T05:32:56Z

Hey @filyp

Really appreciate all the work you put into upgrading the Transformers version. This is a pretty substantial change. I left a few comments; most are minor, except for one issue I called out here that I think we should address before merging.

Once that’s resolved, I’m happy to merge. This upgrade will make it much easier for folks to try the latest models and benefit from the newer bug fixes.

On a side note: I’d love to hear your thoughts on #145 and if it’s easy to incorporate the fix into this PR, that would be great.

Dornavineeth

Few changes. Overall, pretty impressive work! Thank you so much.

Dornavineeth · 2026-02-22T04:56:19Z

+            self.log(eval_metrics)
+            return eval_metrics
+
+        if eval_dataset is None or eval_dataset == "dummy":


Why is this hardcoded? Should we remove this?

eval_dataset == "dummy"

I'm not fully happy with that fix, but that's related to this line in train.py:

eval_dataset=data.get("eval", "dummy"), # None would trigger Trainer exception

It's just that in the new transformers version, the trainer asserts that eval is not None, if we tell it to do evaluations, which breaks our custom evaluators setup. LMK if you see a better solution.

Okay. Seems reasonable.

Can you just define a variable at the top _EVAL_PLACEHOLDER = "_EVAL_PLACEHOLDER" and use this variable at all places?

Dornavineeth · 2026-02-22T05:12:33Z

-            labels = nested_detach(tuple(inputs.get(name) for name in self.label_names))
-            if len(labels) == 1:
-                labels = labels[0]
+    def compute_loss(self, model, inputs, **kwargs):


I see why you renamed compute_loss to compute_unlearn_loss, but this could be a breaking change for anyone pulling the update into their forks; it could lead to runtime errors like compute_unlearn_loss is not defined.

I’d prefer to keep the backward compatibility. Can we keep the previous pattern (per the earlier docstrings): have prediction_step and then call super().compute_loss() for the actual loss computation.

NOTE: You need to copy the prediction_step from the transformers 4.51.3

Yeah, good point. I committed a 3rd alternative now, take a look at current base.py. It preserves backwards compatibility and also is interoperable with other transformers versions.

I regression tested on both tofu unlearning, and some prediction, and it works the same.

Dornavineeth · 2026-02-22T05:13:02Z

-            else:
-                if has_labels or loss_without_labels:
-                    with self.compute_loss_context_manager():
-                        ### Call compute_loss of super class since overridden compute_loss is not be applicable to eval_dataset.


This is the line I am referring here:

https://github.com/locuslab/open-unlearning/pull/175/changes#r2837139317

Dornavineeth · 2026-02-22T05:13:51Z

-        ignore_keys: Optional[List[str]] = None,
-    ) -> Tuple[Optional[torch.Tensor], Optional[torch.Tensor], Optional[torch.Tensor]]:
-        """
-        The only change to this function is calling the Trainer's compute_loss, as it's often overridden by unlearning methods, and we want to maintain the Trainer's evaluation setup.


Can you also include these docstrings in the updated code including prediction_step

Dornavineeth · 2026-02-22T05:14:56Z

        ...

-    def compute_loss(self, model, inputs, return_outputs=False):
+    def compute_unlearn_loss(self, model, inputs, return_outputs=False, num_items_in_batch=None):


I would like to retain it as compute_loss

see https://github.com/locuslab/open-unlearning/pull/175/changes#r2837139317

Dornavineeth · 2026-02-22T05:19:09Z

        train_dataset=data.get("train", None),
-        eval_dataset=data.get("eval", None),
-        tokenizer=tokenizer,
+        eval_dataset=data.get("eval", "dummy"),  # None would trigger Trainer exception


Remove the "dummy" and make it
eval_dataset=data.get("eval", None),

The thing is that using None triggers a Trainer exception (see #175 (comment))

FYI, I moved this hacky fix into the FinetuneTrainer to hide it more. I don't see any better way to fix it.

Dornavineeth · 2026-02-22T05:20:06Z

Looks like these additions in gitignore are specific to your usecases.
can you revert this?

Dornavineeth · 2026-02-22T05:20:52Z

 conda create -n unlearning python=3.11
 conda activate unlearning
-pip install .[lm_eval]
+pip install .


Any reason you removed this?
I would like to give this as an option instead of keeping it requirements.txt because the lm_eval harness is a more involved build.

Ah, I see why I didn't notice lm_eval heaviness, I think I was using version 0.4.11 which installs in 20s (I tested it now in a fresh venv), while the older 0.4.8 in 70s. Should I bump it then? Or revert?
(And the reason I moved it is to simplify the installation.)

You can bump the version and still keep it optional for people having
pip install .[lm_eval]

Dornavineeth · 2026-02-22T05:22:57Z

+huggingface-hub==0.36.0
+transformers==4.51.3
+hf-xet==1.2.0
+lm-eval==0.4.8


see https://github.com/locuslab/open-unlearning/pull/175/changes#r2837147182

Dornavineeth · 2026-02-22T05:23:29Z

@@ -17,13 +17,10 @@
    packages=find_packages(),
    install_requires=requirements,  # Uses requirements.txt
    extras_require={


see https://github.com/locuslab/open-unlearning/pull/175/changes#r2837147182

filyp · 2026-02-22T09:51:53Z

Thanks! I'll go through the review soon. First, about the #145 you mentioned:

It looks pretty complicated, I left a comment there with some alternative simpler fix. But I'd rather keep it separate from this PR, because in my setup I can't test #145 easily. (Unless you want to go with the simple fix, then I can add it, but I still can't test it fully.)

filyp · 2026-02-22T10:47:24Z

I also committed now runners/modal_runner.py, which is not related to PR, but people may find it useful for running with less setup required. I also plan to add more runners in the future (slurm, maybe runpod).

Besides that, I addressed all your comments.

Dornavineeth · 2026-02-23T02:28:11Z

All good! Left comments for 2 minor nits. Once fixed, I will merge it.

filyp · 2026-02-26T21:08:36Z

Hey, somehow I can't see these 2 new comments. Could you link them? (Or maybe you have a "review" started and didn't "submit" the comments, and then only you see them. I remember once having this problem because the UI is quite confusing.)

Dornavineeth · 2026-03-02T03:35:32Z

Here are the comments:

filyp · 2026-03-02T15:39:03Z

Hm, I'm still having problems accessing these comments for some reason :/
(Can you access them when you're not logged in to your github account?)
Maybe simply paste them here?

molereddy · 2026-03-02T18:45:25Z

From my side, I can see unresolved comments in the "Files Changed" viewer. But only when logged in.
Unsure if there's a permissions issue preventing you from viewing them.

filyp · 2026-03-02T19:41:01Z

@molereddy yes I see the same ones. I'm understanding that @Dornavineeth added some new ones that we both don't see?

And these unresolved ones are either outdated or I left a question in them.

molereddy · 2026-03-02T19:55:29Z

Yes, the links @Dornavineeth shard don't open up any comment for me.

Dornavineeth

Please see the comments.

Dornavineeth · 2026-02-23T02:07:11Z

Can we remove this just for the sake of simplicity?

Done, removed. (Let me know if you'd like me to include that runner someplace else, but ok if not.)

Dornavineeth · 2026-02-23T02:08:56Z



 class UnlearnTrainer(FinetuneTrainer):
+    def prediction_step(self, *args, **kwargs):


Can we just copy the orginal transformers code of prediction_step and make edits just wherever required?

Just to not accidentally break any other functionalities included in the hf prediction_step.

Done — copied prediction_step from transformers 4.51.3 with the one change of calling super().compute_loss() instead of self.compute_loss().

(Note though that this implementation could potentially break at some future bump because of these cherrypicked imports at the top. But for the current bump, I regression tested and it's fine.)

Dornavineeth · 2026-02-23T02:23:24Z

+            self.log(eval_metrics)
+            return eval_metrics
+
+        if eval_dataset is None or eval_dataset == "dummy":


Okay. Seems reasonable.

Can you just define a variable at the top _EVAL_PLACEHOLDER = "_EVAL_PLACEHOLDER" and use this variable at all places?

Dornavineeth · 2026-03-02T03:35:21Z

 conda create -n unlearning python=3.11
 conda activate unlearning
-pip install .[lm_eval]
+pip install .


You can bump the version and still keep it optional for people having
pip install .[lm_eval]

filyp · 2026-03-06T18:24:40Z

Ok, I applied all these changes. I also regression tested on tofu unlearning and on some code that uses prediction_step and it's unchanged.

So I think it's all done now.

Dornavineeth · 2026-03-07T06:36:52Z

Great Work. Thank you so much for this PR.

filyp added 4 commits January 26, 2026 16:15

bump transformers to 4.51.3

56b662e

make UnlearnTrainer implementation more future proof make other necessary changes to be compatible with new transformers version

simplify installation

31546d3

fix leaderboard docs wider gitignore

make FinetuneTrainer more readable

9987ee3

use trainer.processing_class instead of trainer.tokenizer, to support…

4b43848

… transformers==5

filyp had a problem deploying to tests February 11, 2026 05:26 — with GitHub Actions Failure

fix ruff linter errors

e180d7f

filyp had a problem deploying to tests February 18, 2026 14:59 — with GitHub Actions Failure

molereddy requested a review from Dornavineeth February 18, 2026 16:28

format for ruff==0.6.6

1a3b7c8

filyp temporarily deployed to tests February 18, 2026 17:21 — with GitHub Actions Inactive

Dornavineeth requested changes Feb 22, 2026

View reviewed changes

filyp added 2 commits February 22, 2026 11:35

preserve compute_loss backwards compatibility

367fee9

revert gitignore

d6f1f1a

hide the trainer fix that allows for evaluating when eval_dataset=None

f1d2572

Dornavineeth requested changes Mar 4, 2026

View reviewed changes

filyp added 2 commits March 6, 2026 19:07

apply changes from the review

d3dfad6

typo fix

faf942f

Dornavineeth approved these changes Mar 7, 2026

View reviewed changes

filyp had a problem deploying to tests March 7, 2026 06:36 — with GitHub Actions Failure

fix: lint

2ff5029

Dornavineeth temporarily deployed to tests March 7, 2026 07:01 — with GitHub Actions Inactive

Dornavineeth merged commit a456aa2 into locuslab:main Mar 7, 2026
1 check passed

filyp deleted the pr-transformers-4.51.3 branch March 9, 2026 14:40

filyp mentioned this pull request May 13, 2026

Upgrading transformers version to 5.5.4 #191

Open

3 tasks



		class UnlearnTrainer(FinetuneTrainer):
		def prediction_step(self, args, *kwargs):

Conversation

filyp commented Jan 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Before submitting

Tests

Uh oh!

filyp commented Feb 8, 2026

Uh oh!

molereddy commented Feb 11, 2026

Uh oh!

filyp commented Feb 11, 2026

Uh oh!

molereddy commented Feb 18, 2026

Uh oh!

filyp commented Feb 18, 2026

Uh oh!

Dornavineeth commented Feb 22, 2026

Uh oh!

Dornavineeth left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

filyp commented Feb 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

filyp commented Feb 22, 2026

Uh oh!

Dornavineeth commented Feb 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

filyp commented Feb 26, 2026

Uh oh!

Dornavineeth commented Mar 2, 2026

Uh oh!

filyp commented Mar 2, 2026

Uh oh!

molereddy commented Mar 2, 2026

Uh oh!

filyp commented Mar 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

molereddy commented Mar 2, 2026

Uh oh!

Dornavineeth left a comment

filyp commented Jan 26, 2026 •

edited

Loading

filyp commented Feb 22, 2026 •

edited

Loading

Dornavineeth commented Feb 23, 2026 •

edited

Loading

filyp commented Mar 2, 2026 •

edited

Loading