[BUG] tool.setuptools.license-files results in invalid metadata

setuptools version

setuptools==74.1.2

Python version

Python 3.13

OS

any

Additional environment information

No response

Description

If any of the the glob patterns specified intool.setuptools.license-files matches a file in the package, setuptools generates invalid metadata: it includes a License-File field while specifying Metadata-Version to be 2.1. This is invalid and packaging raises an exception while parsing the metadata. This likely results in the resulting distributions to not be accepted by PyPI.

Because tool.setuptools.license-files has a default value of ['LICEN[CS]E*', 'COPYING*', 'NOTICE*', 'AUTHORS*'] the problem can be encountered also in packages that do not explicitly set this field in pyproject.toml but happen to have a file matching the default glob pattern.

Expected behavior

Do not emit the License-File field or do it and specify Metadata-Version: 2.4 as per PEP 639.

How to Reproduce

Here is a short reproducer:

$ cat >pyproject.toml <<EOF
[build-system]
requires = ["setuptools"]
build-backend = "setuptools.build_meta"
[project]
name = "foo"
version = "1.2.3"
[tool.setuptools]
license-files = ["LICENSE"]
EOF
$ touch LICENSE
$ uv build .
$ tar xf dist/foo-1.2.3.tar.gz
$ python
>>> import pathlib
>>> import packaging.metadata
>>> packaging.metadata.Metadata.from_email(pathlib.Path('dist/foo-1.2.3/PKG-INFO').read_text())

Output

  + Exception Group Traceback (most recent call last):
  |   File "<python-input-3>", line 1, in <module>
  |     packaging.metadata.Metadata.from_email(pathlib.Path('dist/foo-1.2.3/PKG-INFO').read_text())
  |     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  |   File "/Users/daniele/src/twine/.venv/lib/python3.13/site-packages/packaging/metadata.py", line 781, in from_email
  |     raise ExceptionGroup(
  |         "invalid or unparsed metadata", exc_group.exceptions
  |     ) from None
  | ExceptionGroup: invalid or unparsed metadata (1 sub-exception)
  +-+---------------- 1 ----------------
    | packaging.metadata.InvalidMetadata: license-file introduced in metadata version 2.4, not 2.1
    +------------------------------------
>>>

This is a well-known case of an early implementation of a PEP 639 draft. Right now tools are equipped to accept this variation.

In time it will be fixed. But not immediately, due to effort constraints and release scheduling (we are in the process of implementing previous versions of metadata first).

Probably we can close this as a "kind of duplicate" of the request to implement PEP 639.

This is a well-known case of an early implementation of a PEP 639 draft. Right now tools are equipped to accept this variation.

What I am trying to say is that tools are not equipped to accept this variation. packaging is the PyPA library for parsing distribution metadata and, as shown above, it does not handle this variation. Which other tools dealing with packages metadata that handle this correctly do you have in mind?

@dnicolodi

@dnicolodi Is right here. The License-File metadata generated by setuptools causes an error during validation with packaging==24.2 since the metadata version isn't >=2.4. This hasn't been an issue so far as twine doesn't support version 2.4 and thus License-File isn't included in the form fields during upload.

What I'm uncertain about is the solution. Sure, we can remove the License-File field from the generated metadata, and probably do so anyway as it isn't spec compliant even with 2.4. However that wouldn't resolve the issue for all packages which pin older setuptool versions. As it was added here quite a while ago, I suspect the impact will be quite large. This might be something which needs to be handled upstream (twine / warehouse) too, e.g. only push the License-File field if the metadata version is >=2.4 and ignore it for older versions.

Accommodate for invalid metadata produced by setuptools

@cdce8p

Which other tools dealing with packages metadata that handle this correctly do you have in mind?

I was thinking that twine/PyPI/pip have been accepting License-File with metadata version 2.2 for a while.

Just to emphasize that we are going to tackle this problem in time (btw, thanks @cdce8p for the PRs). But before that we will implement metadata version 2.3 (I just got a review recently on one of the PRs necessary for 2.3, but this week I don't have the time to delve into it).

Meanwhile, my strong preference is to not change anything that has been in place for the last 2 or 3 years.

If you need the process core metadata using packaging before we proceed with the updates in setuptools from 2.2, thourgh 2.3 and all the way to 2.4, maybe you can consider deleting/ignoring the License-File field before the validation?

I was thinking that twine/PyPI/pip have been accepting License-File with metadata version 2.2 for a while.

I am not very familiar with all the consumers of package metadata. I have the impression that there are several tools using different approaches for metadata parsing and validation. Only relatively recently packaging implemented metadata validation. I hope there is going to be a convergence toward more strict parsing of metadata, either via packaging or via equivalent tools.

Indeed, I bumped into the issue with License-File because I would like to move twine from using a semi-maintained library for extracting metadata from distribution files to a solution based on packaging pypa/twine#1180 This comes with stricter metadata validation, which IMHO is a good thing. I have now implemented a "work-around" for the metadata emitted by existing setuptools versions as there is nothing we can do about those.

I was under the impression that PyPI implements validation of the distribution files metadata, but if it does, the validation is not very strict. What is validated strictly is the form data that is sent alongside the distribution files. That can be tweaked as needed to support the metadata emitted by existing setuptools releases.

pip is most likely the most permissive consumer of metadata, thus I don't expect it to do any validation.

I don't see any reason to change setuptools to fix this issue straight away. I filed this issue to make sure that you are aware of it. Because, as I wrote above, metadata parsing has not been very strict so far, it could have been that you are not aware of it.

@dnicolodi

Thank you very much @dnicolodi.
Hopefully we can have it soon and start to converge towards packaging.

One thing that would help a lot is if packaging itself could provide functionally to emit metadata.
This way we use a single and consistent library for both emitting and parsing metadata.
It is probably been already tracked by pypa/packaging#570.

meson-python and scikit-core (and possibly other build backends) use pytproject-metadata https://github.com/pypa/pyproject-metadata/ for translating pyproject.toml into an RFC822 metadata file (I am one of the maintainers of meson-python).

There is some work toward incorporating pyproject-metadata into packaging pypa/packaging#846 pypa/packaging#847

On the other hand, I like that there are more that one implementation of the standard: that makes it easier to ensure that the only implementation does not diverge from the standard and avoids such implementation, bugs included, to become the de-facto standard. We were in that situation before PEP 517 and PEP 621, and we are still cleaning up the mess... 🙂

@dnicolodi

I was thinking that twine/PyPI/pip have been accepting License-File with metadata version 2.2 for a while.

PyPI uses packaging.metadata to validate the submitted form fields. Unknown fields are just ignored. With 24.2 license_files was defined as being added in 2.4 thus it would start to fail now when submitted with older version.

Saw the commit from @dnicolodi on the packaging PR: dnicolodi/twine@ab3bf7d. All considered, that's probably the most practical solution here.

Just to emphasize that we are going to tackle this problem in time (btw, thanks @cdce8p for the PRs). But before that we will implement metadata version 2.3.

Meanwhile, my strong preference is to not change anything that has been in place for the last 2 or 3 years.

Sorry in advance if I'm a bit annoying here. I can understand your position, but wouldn't fully agree with it. We can do more even before we implement metadata version 2.3.

Store license-files in licenses subfolder #4728
We already include the License-File file in the core metadata. Why not make it spec compliant then? It would also fix an issue were license files with the same names but in different folders would just be overwritten, see [FR] Support License-File root license directory (PEP 639) #3596.
Add initial support for license expression (PEP 639) #4706
While the initial approach for it was wrong, adding the License-Expression field now would just result in the same issue we have with License-File already, there is nothing stopping us from repurposing the existing License field. In fact that's the same approach hatchling and poetry use as they are also blocked on the twine update and can't yet move to 2.4.

Why am I pushing for these changes? For Home Assistant we use a script to try to validate the licenses of all requirements and tbh it's just a mess. Some packages use the outdated classifiers, some custom license strings and others the full license in the metadata. I'm prepared to open PRs for some of these dependencies but it only really makes sense if I can use the final pyproject.toml metadata. Otherwise I'd have to do that all over again once setuptools is able to emit valid 2.4 metadata. With the changes above, it only needs a new release with an updated setuptools version and everything would be fine.

PyPI uses packaging.metadata to validate the submitted form fields. Unknown fields are just ignored. With 24.2 license_files was defined as being added in 2.4 thus it would start to fail now when submitted with older version.

AFAIU warehouse (the software powering PyPI) does not ignore unknown fields. This should be the validation code: https://github.com/pypi/warehouse/blob/d57082ee37327bc1e8a28f96470b00ed226c0f87/warehouse/forklift/metadata.py#L263-L348

However, twine submits only fields it known about, and it does not know yet about license_files (which should appear as multiple license_file in the form data, but this is a minor technical detail).

hatchling and poetry [...] are also blocked on the twine update and can't yet move to 2.4.

Can you point me to more information regarding this? I was under the impression that hatchling users use the upload mechanism provided by the hatch framework, and same for poetry. Isn't this the case? One of the PyPI maintainers on Discourse said that there are PyPI uploads conforming to metadata 2.4. I thought those were hatchling projects.

PyPI uses packaging.metadata to validate the submitted form fields. Unknown fields are just ignored. With 24.2 license_files was defined as being added in 2.4 thus it would start to fail now when submitted with older version.

AFAIU warehouse (the software powering PyPI) does not ignore unknown fields. This should be the validation code: https://github.com/pypi/warehouse/blob/d57082ee37327bc1e8a28f96470b00ed226c0f87/warehouse/forklift/metadata.py#L263-L348

You're right. I only checked the packaging call Metadata.from_raw(...) which ignores unknown fields. Didn't know warehouse implemented additional checks.

hatchling and poetry [...] are also blocked on the twine update and can't yet move to 2.4.

Can you point me to more information regarding this? I was under the impression that hatchling users use the upload mechanism provided by the hatch framework, and same for poetry. Isn't this the case?

For hatchling see pypa/hatch#1828 (comment). The same would likely apply to poetry as well. Although they haven't yet started implementing PEP 639 to begin with python-poetry/poetry#9670. The comment for poetry was more referencing the fact that they already allow project.license = MIT in pyproject.toml and just back-populate it to the License field instead of License-Expression.

One of the PyPI maintainers on Discourse said that there are PyPI uploads conforming to metadata 2.4. I thought those were hatchling projects.

AFAIK you can overwrite the Metadata-Version with hatchling and at some point (before PyPI added official support for it I believe) this was even the default. It has since been reverted though.

The last comment on discuss I saw was

No further uploads have occurred since that first attempt.

Hi guys, I understand the urgency of the topic, but I am also very conscious that we need to move very carefully to avoid breaking the ecosystem all at once. In the past we had a lot of bad experiences with botched releases, so I am trying to take the most conservative approach possible (and even with that it is possible there will be problems).

The good thing is that we have most of the pieces already in place (thanks again for the PRs). We now need to coordinate to release them, collect feedback and fix if things are broken.

My preference is to do things step by step and wait one week or so between steps to receive feedback of early adopters on edge cases (of course respecting the holiday season coming ahead and the times of all collaborations, so probably longer than that). I suggest the following:

Setuptools to finish testing Metadata 2.2 implementation (followed by release + feedback + bug fixing)
Setuptools to bump Metadata 2.3 - should be trivial - (followed by release + feedback + bug fixing)
Setuptools to implement PEP 639 and Metadata 2.4 - (followed by release + feedback + bug fixing)
Twine and other downstream consumers to support strict Metadata 2.4 ¹

I think that step 2 is going to be trivial to support, once we concede that we don't have to worry to much about the pypa/packaging#845 and treat it as a temporary bug.

I am not sure twine will be able to fully drop lenient parsing for Metadata < 2.4 (many users do pin the version of their build dependencies, so there may be some backlash form the community). But this is a decision that the twine team can decide to take. ↩

Release version 0.13.3

@abravalheri

Hey @abravalheri, what's the current status here? How can we move this forward? AFAICT Dynamic works as expected. Should we accept the packaging bug as given and bump the metadata version to 2.3?

only use one license definition in pyproject.toml

@cdce8p

Hi @cdce8p, sorry for the delay and thank you very much for the contributions. These first months of 2025 have been a bit busy for me (probably likely to continue to be busy until April).

I agree with you, we should be in a good position to continue the work towards Core Metadata version 2.3 (accepting the bug in packaging).

Could you please confirm that from your PoV, the following PRs are enough for an initial implementation of PEP 639:

Or is there anything else needed for compatibility with Core Metadata version 2.3?

Because the release process in setuptools always publishing everything that hits the main not necessarily waiting for feature-completeness, the way I would go about it is:

I can create a branch preparing for Core Metadata 2.3 (update: feature/pep639)
We can continue to review and merge the PRs into this new branch.
Once everything is merged, we can merge back into the main and release.

In my opinion, the release of PEP 639 should be atomic. What I would like to avoid for example is "changing the license directory" without "bumping the metadata version" (although it should be possible to do that without offending the standards, we cannot guarantee how the existing tools would react to piecewise changes).

Could you please confirm that from your PoV, the following PRs are enough for an initial implementation of PEP 639:

Add initial support for license expression (PEP 639) #4706

Store license-files in licenses subfolder #4728

Update validate-pyproject to 0.23.0 #4734

As setuptools was stuck on an old metadata version, I stopped short of fully implementing it. Basically all that's missing after these are merged is

Emit License-Expression metadata instead of License (for valid license expressions)
Bump the metadata version to 2.4

Or is there anything else needed for compatibility with Core Metadata version 2.3?

Note

PEP 639 requires version 2.4, not 2.3. So we should have bumped to 2.3 before finalizing the 639 support.

In my opinion, the release of PEP 639 should be atomic. What I would like to avoid for example is "changing the license directory" without "bumping the metadata version" (although it should be possible to do that without offending the standards, we cannot guarantee how the existing tools would react to piecewise changes).

I understand your position. All #4706 does is basically allow license expressions in the metadata but emit it in the License field (valid today). In the past, I've even encouraged projects to use the official SPDX identifier with project.license.text. At worst it might be slightly surprising to people that setuptools doesn't emit 2.4 metadata yet. Though it's equally surprising that it rejects "valid" keys in pyproject.toml, #4828.

The different license files directory isn't a big deal either. The current implementation is broken, especially if two license files have the same name (in different directories). One would just get overridden. Tools already need to handle the correct path as well, since other backends didn't implement the setuptools version in the first place. It would be different if the change would be to another not spec-compliant implementation, but that isn't the case here.

To summarize, I think the current PRs would be fine as is even in main if released today. Using a temporary branch works as well, just complicates it a bit IMO. It might be necessary to rebase and resolve merge conflicts which always pose the risk of adding additional issues.

Switch to hatchling due to setuptools issue with License-File

Setuptools workaround to publish to PyPI

Workaround setuptools metadata issue pypa/setuptools#4759

Build: Clear "license-files" option in setup metadata due to

It looks like this issue can now be closed (I believe since v77.0.0).

The reproducible example in the original problem seems to run fine now without errors:

# docker run --rm -it python:3.12-bookworm /bin/bash

mkdir /tmp/project

cd /tmp/project

cat >pyproject.toml <<EOF
[build-system]
requires = ["setuptools"]
build-backend = "setuptools.build_meta"

[project]
name = "foo"
version = "1.2.3"

[tool.setuptools]
license-files = ["LICENSE"]
EOF

touch LICENSE

python -m venv .venv
.venv/bin/python -m pip install -U "build==1.3.0"
.venv/bin/python -m build .

.venv/bin/python -m pip install -U "packaging==25.0"
tar xf dist/foo-1.2.3.tar.gz
.venv/bin/python -c '
import pathlib
import packaging.metadata
print(packaging.metadata.Metadata.from_email(pathlib.Path("foo-1.2.3/PKG-INFO").read_text()))
'
# <packaging.metadata.Metadata object at 0x798366850440>

We can also use project.license-files:

# docker run --rm -it python:3.12-bookworm /bin/bash

mkdir /tmp/project2

cd /tmp/project2

cat >pyproject.toml <<EOF
[build-system]
requires = ["setuptools"]
build-backend = "setuptools.build_meta"

[project]
name = "foo"
version = "1.2.3"
license-files = ["LICENSE"]
EOF

touch LICENSE

python -m venv .venv
.venv/bin/python -m pip install -U "build==1.3.0"
.venv/bin/python -m build .

.venv/bin/python -m pip install -U "packaging==25.0"
tar xf dist/foo-1.2.3.tar.gz
.venv/bin/python -c '
import pathlib
import packaging.metadata
print(packaging.metadata.Metadata.from_email(pathlib.Path("foo-1.2.3/PKG-INFO").read_text()))
'
# <packaging.metadata.Metadata object at 0x723e954dc440>

Disable setuptools automatic license-files inclusion

remove license file metadata from pyproject.toml

Disable setuptools automatic license-files inclusion

Navigation Menu

Repository navigation

[BUG] `tool.setuptools.license-files` results in invalid metadata #4759

setuptools version

Python version

OS

Additional environment information

Description

Expected behavior

How to Reproduce

Output

77 remaining items

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

[BUG] tool.setuptools.license-files results in invalid metadata #4759

Description

setuptools version

Python version

OS

Additional environment information

Description

Expected behavior

How to Reproduce

Output

Activity

abravalheri commented on Nov 26, 2024

dnicolodi commented on Nov 26, 2024

cdce8p commented on Nov 27, 2024

abravalheri commented on Nov 27, 2024

dnicolodi commented on Nov 27, 2024

abravalheri commented on Nov 27, 2024

dnicolodi commented on Nov 27, 2024

cdce8p commented on Nov 27, 2024

dnicolodi commented on Nov 27, 2024

dnicolodi commented on Nov 27, 2024

cdce8p commented on Nov 27, 2024

abravalheri commented on Nov 27, 2024

Footnotes

77 remaining items

cdce8p commented on Feb 13, 2025

abravalheri commented on Feb 14, 2025

cdce8p commented on Feb 16, 2025

abravalheri commented on Nov 20, 2025

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions

[BUG] `tool.setuptools.license-files` results in invalid metadata #4759