[bsprenger]

Stack from ghstack (oldest at bottom):

• -> [Feature] Export replay buffer contents to external datasets #4574
• [Feature] Add a Hugging Face adapter for replay datasets #4573
• [Feature] Define replay dataset adapter protocols #4572

Description

• Add ReplayBuffer.export to pass stored replay data to a ReplayDatasetExporter and return the external dataset produced by that exporter.
• Read entries in storage order by default, or order them by trajectory and step when both keys are supplied.
• Add synchronization around export and a share_memory option on StorageDataset so export does not need to move CPU storage into shared memory.
• Add replay buffer, storage, and dataset tests plus reference documentation.

Motivation and Context

This provides a replay buffer API for exporting stored samples to external dataset formats while allowing callers to choose physical storage order or episodic order.

• I have raised an issue to propose this change (required for new features and bug fixes)

Types of changes

• New feature (non-breaking change which adds core functionality)
• Documentation (update in the documentation)

Checklist

• I have read the CONTRIBUTION guide (required)
• My change requires a change to the documentation.
• I have updated the tests accordingly (required for a bug fix or a new feature).
• I have updated the documentation accordingly.

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/rl/4574

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

✅ You can merge normally! (1 Unrelated Failure)

As of commit 692b16b with merge base 264a3c2 ():

BROKEN TRUNK - The following job failed but were present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

• Unit-tests on Linux / tests-optdeps (3.12, 13.0) / linux-job (gh) (trunk failure)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 2

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[github-actions]

⚠️ PR Title Label Error

PR title must start with a label prefix in brackets (e.g., [BugFix]).

Current title: Add replay buffer export API

Supported Prefixes (case-sensitive)

Your PR title must start with exactly one of these prefixes:

Prefix	Label Applied	Example
[Algorithm]	new algo	[Algorithm] Add new RL objective
[BE]	BE	[BE] Improve error messages
[Benchmark] or [Benchmarks]	Benchmarks	[Benchmark] Add collector benchmark
[BugFix]	BugFix	[BugFix] Fix memory leak in collector
[Example] or [Examples]	Examples	[Example] Add training script
[Feature]	Feature	[Feature] Add new optimizer
[Doc] or [Docs]	Documentation	[Doc] Update installation guide
[Refactor]	Refactoring	[Refactor] Clean up module imports
[CI]	CI	[CI] Fix workflow permissions
[Test] or [Tests]	Tests	[Tests] Add unit tests for buffer
[Trainer] or [Trainers]	Trainers	[Trainer] Add trainer config
[Environment] or [Environments]	Environments	[Environments] Add Gymnasium support
[Data]	Data	[Data] Fix replay buffer sampling
[LLM]	llm/	[LLM] Add reward model integration
[Minor]	small change	[Minor] Fix typo in error message
[Performance] or [Perf]	Performance	[Performance] Optimize tensor ops
[BC-Breaking]	bc breaking	[BC-Breaking] Remove deprecated API
[Deprecation]	Deprecation	[Deprecation] Mark old function
[Algorithm] or [Algorithms]	new algo	[Algorithm] Add new objective
[Quality]	Quality	[Quality] Fix typos and add codespell
[Versioning]	versioning	[Versioning] Bump release version
[WIP]	WIP	[WIP] Draft implementation

Note: Common variations like singular/plural are supported (e.g., [Doc] or [Docs]).

[github-actions]

⚠️ PR Title Label Error

PR title must start with a label prefix in brackets (e.g., [BugFix]).

Current title: Add replay buffer export API

Supported Prefixes (case-sensitive)

Your PR title must start with exactly one of these prefixes:

Prefix	Label Applied	Example
[Algorithm]	new algo	[Algorithm] Add new RL objective
[BE]	BE	[BE] Improve error messages
[Benchmark] or [Benchmarks]	Benchmarks	[Benchmark] Add collector benchmark
[BugFix]	BugFix	[BugFix] Fix memory leak in collector
[Example] or [Examples]	Examples	[Example] Add training script
[Feature]	Feature	[Feature] Add new optimizer
[Doc] or [Docs]	Documentation	[Doc] Update installation guide
[Refactor]	Refactoring	[Refactor] Clean up module imports
[CI]	CI	[CI] Fix workflow permissions
[Test] or [Tests]	Tests	[Tests] Add unit tests for buffer
[Trainer] or [Trainers]	Trainers	[Trainer] Add trainer config
[Environment] or [Environments]	Environments	[Environments] Add Gymnasium support
[Data]	Data	[Data] Fix replay buffer sampling
[LLM]	llm/	[LLM] Add reward model integration
[Minor]	small change	[Minor] Fix typo in error message
[Performance] or [Perf]	Performance	[Performance] Optimize tensor ops
[BC-Breaking]	bc breaking	[BC-Breaking] Remove deprecated API
[Deprecation]	Deprecation	[Deprecation] Mark old function
[Algorithm] or [Algorithms]	new algo	[Algorithm] Add new objective
[Quality]	Quality	[Quality] Fix typos and add codespell
[Versioning]	versioning	[Versioning] Bump release version
[WIP]	WIP	[WIP] Draft implementation

Note: Common variations like singular/plural are supported (e.g., [Doc] or [Docs]).

[theap06]

Lgtm

[bsprenger]

@torchrlbot merge

[github-actions]

Merging ghstack PR via ghstack land (requested by @bsprenger).

[github-actions]

ghstack land failed.

ERROR: Fatal exception
Traceback (most recent call last):
  File "/opt/hostedtoolcache/Python/3.12.15/x64/lib/python3.12/site-packages/ghstack/logs.py", line 245, in manager
    yield
  File "/opt/hostedtoolcache/Python/3.12.15/x64/lib/python3.12/site-packages/ghstack/cli.py", line 56, in cli_context
    config = ghstack.config.read_config(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.12.15/x64/lib/python3.12/site-packages/ghstack/config.py", line 191, in read_config
    github_url = input("GitHub enterprise domain (leave blank for OSS GitHub): ")
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
EOFError: EOF when reading a line

[bsprenger]

Superseded by #4592