Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
38 commits
Select commit Hold shift + click to select a range
3ef53b2
refactor(pypi): select packages from the pyproject.toml dependency gr…
aignas Nov 7, 2026
7557370
refactor(pypi): support hiding the unexposed packages of a hub
aignas Nov 7, 2026
23127e5
feat(pypi): add pip.configure to select the exposed packages of a hub
aignas Nov 7, 2026
6bb7f83
fix(pypi): only add the exposed packages of a configured hub to @pypi
aignas Nov 7, 2026
b31a734
fix(pypi): treat exposing all packages as not configuring the hub
aignas Nov 8, 2026
9d97360
refactor(pypi): use explicit loops and simpler conditions in pip.conf…
aignas Nov 8, 2026
35c0660
test(pypi): test the pip.default pyproject_toml with pip.configure
aignas Nov 8, 2026
a2049ee
Merge branch 'main' into aignas.feat.dep-group-selectors
aignas Nov 9, 2026
2873e71
feat(pypi): add strip_extras to remove the extras from requirement lines
aignas Nov 13, 2026
2f92d4e
feat(pypi): add keep_extras and dependency excludes to pep508_deps
aignas Nov 13, 2026
efc64f0
feat(pypi): add extra_label and group_member label helpers
aignas Nov 13, 2026
552d8f1
feat(pypi): create a target for every provided extra in the whl repos…
aignas Nov 13, 2026
ace0065
feat(pypi): record the requested extras in whl_config_setting
aignas Nov 13, 2026
c0bd9cc
feat(pypi): add the per-platform extra targets to the hub packages
aignas Nov 13, 2026
998146c
feat(pypi): use a single whl repository for the different extras of a…
aignas Nov 13, 2026
dfaa556
feat(pypi): support extras in the requirement macro and add the requi…
aignas Nov 13, 2026
6e77448
refactor(pypi): extract the failing action of missing_package_error
aignas Nov 13, 2026
3284430
feat(pypi): fail the build when a required extra of a dependency is m…
aignas Nov 13, 2026
cca8d1b
feat(pypi): support the extra targets in WORKSPACE
aignas Nov 13, 2026
181ddc9
feat(pypi): support extras as members of the requirement cycle groups
aignas Nov 13, 2026
cf3b4b6
refactor(pypi): pass the requirement line only to the repositories us…
aignas Nov 13, 2026
be422b2
test(pypi): check the extras of the dependencies from the METADATA
aignas Nov 13, 2026
646f270
docs(pypi): document the extra targets
aignas Nov 13, 2026
dafa838
chore(examples): update the vendored requirements.bzl
aignas Nov 13, 2026
6431f8b
fix(pypi): parse the whl METADATA with CRLF line endings
aignas Nov 13, 2026
c2e07df
test(pypi): drop the requirement from the whl repositories in the int…
aignas Nov 13, 2026
9f9fc9f
refactor(pypi): keep the original hub_builder group map and repo mapp…
aignas Nov 13, 2026
a24fac6
refactor(pypi): minimize the changes to the group library generation
aignas Nov 13, 2026
8f57a4f
test(pypi): remove the overlapping extras tests
aignas Nov 13, 2026
330eb71
Merge remote-tracking branch 'upstream/main' into aignas.feat.dep-gro…
aignas Nov 15, 2026
c488c62
Merge remote-tracking branch 'upstream/main' into feat.aignas.target-…
aignas Nov 15, 2026
65a8f92
Merge remote-tracking branch 'upstream/main' into feat.aignas.target-…
aignas Nov 17, 2026
7765a28
Merge remote-tracking branch 'upstream/main' into aignas.feat.dep-gro…
aignas Nov 17, 2026
b014361
Merge branch 'feat.aignas.target-per-extra' into feat.aignas.pip-conf…
aignas Nov 17, 2026
df890f2
feat(pypi): collect the extras of the selected dependency groups
aignas Nov 17, 2026
c7b28a6
feat(pypi): expose the pyproject.toml extras with pip.configure
aignas Nov 17, 2026
45c1c82
docs(pypi): document the pip.configure extras
aignas Nov 17, 2026
b514331
docs(pypi): name the news entry after the PR
aignas Nov 17, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions MODULE.bazel
Original file line number Diff line number Diff line change
Expand Up @@ -335,6 +335,7 @@ use_repo(
"pkgutil_nspkg2",
"rules_python_runtime_env_tc_info",
"somepkg_with_build_files",
"whl_library_extras_consumer",
"whl_library_extras_direct_dep",
"whl_with_build_files",
"whl_with_data1",
Expand Down
26 changes: 26 additions & 0 deletions docs/pypi/circular-dependencies.md
Original file line number Diff line number Diff line change
Expand Up @@ -65,6 +65,32 @@ be a part of the `airflow` cycle. For instance:
)
```

Each extra of a package has its own target, so a cycle that goes through an
extra back to the package without extras, e.g. `foo[bar]` depending on `baz`
which depends on `foo`, is not a cycle in Bazel and does not need a group. If
the cycle includes the extra itself, e.g. `baz` depends on `foo[bar]`, then
only the extra needs to be a member of the group:

```starlark
...
experimental_requirement_cycles = {
"foo": [
"foo[bar]",
"baz",
]
}
)
```

Only the targets of the members, `@pypi//foo:[bar]` and `@pypi//baz`, are
then part of the group and the other targets of `foo`, e.g. `@pypi//foo:[]`,
can be used without the group. The `whl` targets do not use the extras, so
only the packages without extras are part of the group for the `whl` targets.

:::{versionadded} VERSION_NEXT_FEATURE
Support for the extras as members of the groups.
:::

Alternatively, one could resolve the cycle by removing one leg of it.

For example, while `apache-airflow-providers-sqlite` is "baked into" the Airflow
Expand Down
66 changes: 66 additions & 0 deletions docs/pypi/download.md
Original file line number Diff line number Diff line change
Expand Up @@ -134,6 +134,72 @@ rename, and `requirement("numpy")` routes through the active hub like
fixed at loading time, before the active hub is known, so load them from a
concrete hub instead. Prefer `@pypi//<pkg>` labels in new code.

(pip-configure)=
### Configuring the Exposed Packages (pip.configure)

By default a hub exposes all of the packages from its requirements files or
`uv.lock` file, including the transitive dependencies. A `pip.configure` tag
can limit the hub to the packages listed in a `pyproject.toml` file: the
`[project] dependencies`, `[project.optional-dependencies]` and PEP 735
`[dependency-groups]` tables.

```toml
[project]
dependencies = ["numpy"]

[dependency-groups]
dev = ["pytest"]
```

```starlark
pip.parse(
hub_name = "pypi_hub",
pyproject_toml = "//:pyproject.toml",
requirements_lock = "//:requirements_lock.txt",
)

pip.configure(
hub_name = "pypi_hub",
dependency_groups = ["default", "group:dev"],
)
```

Here `@pypi_hub//numpy` and `@pypi_hub//pytest` can be used, but the
dependencies of `numpy` are only used through `@pypi_hub//numpy`. The
`all_requirements` lists in the hub's `requirements.bzl` and the
[Unified @pypi Hub](unified-pypi-hub) only contain the exposed packages.

The {attr}`pip.configure.dependency_groups` attribute defaults to `["*"]`,
which selects every group. Use `["*", "transitive"]` to expose all of the
packages of the hub, including the transitive dependencies, which is the same
as not configuring the hub. Selecting the transitive dependencies of only some
of the groups is not supported.

Multiple tags for the same hub are combined. Only the root module and the
module that created the hub can configure it; if both do, the root module
wins. A self reference such as `my-project[test]` in the `pyproject.toml`
selects the optional dependencies of the project. The packages that are not
part of the hub are ignored, so a single `pyproject.toml` can list tools that
are missing from the lock file.

The extras requested in the selected groups are exposed by the hub even if the
lock file does not request them, for example because the lock file was
generated with `--strip-extras`. With the following `pyproject.toml`:

```toml
[project]
dependencies = ["requests[socks]"]
```

the hub has the `@pypi_hub//requests:[socks]` target and
`@pypi_hub//requests` includes the `socks` extra on all of the platforms. The
environment markers of the requirements are ignored when selecting the extras.
The dependencies of the extras, such as `pysocks`, still have to be in the lock
file. See [extras](./use.md#extras) for the targets of the extras.

:::{versionadded} VERSION_NEXT_FEATURE
:::

### Declaring Abstract Dependencies (pip.dep)

:::{versionadded} 2.2.0
Expand Down
80 changes: 77 additions & 3 deletions docs/pypi/use.md
Original file line number Diff line number Diff line change
Expand Up @@ -57,11 +57,85 @@ which can help you create a `py_binary` target for a particular console script e

[whl_ep]: https://packaging.python.org/specifications/entry-points/

{#extras}
## 'Extras' dependencies

Any "extras" specified in the requirements lock file will be automatically added
as transitive dependencies of the package. In the example above, you'd just put
`requirement("useful_dep")` or `@pypi//useful_dep`.
Any "extras" specified in the requirements lock file or the `uv.lock` file will
be automatically added as transitive dependencies of the package. In the
example above, you'd just put `requirement("useful_dep")` or
`@pypi//useful_dep`.

If you need only some of the extras, the hub repo also has a target for the
package without any extras and a target for each extra used in the lock file.
For example, if the lock file has `google-api-core[async_rest,grpc]`, then the
hub has:

```
@pypi//google_api_core # The package with all of the extras
@pypi//google_api_core:[] # The package without any extras
@pypi//google_api_core:[async_rest] # The package with the async_rest extra
@pypi//google_api_core:[grpc] # The package with the grpc extra
```

Extra names are normalized in the same way as the package names, so
`async-rest` becomes `async_rest`.

The `requirement` helper supports the same syntax:

```starlark
load("@pypi//:requirements.bzl", "requirement")

py_library(
name = "my_lib",
deps = [
requirement("google-api-core[grpc]"),
],
)
```

There are no targets for combinations of extras, because each `[extra]` target
already includes the package. The `requirement` helper accepts at most one
extra. To request more than one extra, use the `requirements` helper, which
accepts any number of packages and returns a list with a label for each
package and each extra:

```starlark
load("@pypi//:requirements.bzl", "requirements")

py_library(
name = "my_lib",
deps = [
":other_lib",
] + requirements(
"google-api-core[async_rest,grpc]",
"numpy",
),
)
```

If the lock file requests different extras for a package on different
platforms, for example `foo[bar]` on Linux and `foo[baz]` on macOS, then
`@pypi//foo` includes only the extras requested for the target platform and
`@pypi//foo:[bar]` on macOS is the same as `@pypi//foo:[]`.

The dependencies of the packages use the extras written in the wheel
`METADATA`, so if `foo` depends on `bar[socks]`, then `@pypi//foo` depends on
`@pypi//bar:[socks]`. If the extra is not in the lock file, then building
`@pypi//foo` fails with a message asking you to add `bar[socks]` to the
requirements or the lock file. For example, when compiling the requirements
with `uv pip compile`, pass `--no-strip-extras` to keep the extras in the
output.

The `whl` targets, e.g. `@pypi//foo:whl`, do not use the extras and include
only the wheels of the package and its dependencies without extras.

The packages in a dependency group can also have only some of their extras in
the group, see the [circular dependencies](./circular-dependencies) docs.

:::{versionadded} VERSION_NEXT_FEATURE
The `[]` and `[extra]` targets, the support for extras in the `requirement`
helper and the `requirements` helper were added.
:::

## Consuming Wheel Dists Directly

Expand Down
28 changes: 26 additions & 2 deletions examples/pip_parse_vendored/requirements.bzl
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@

load("@rules_python//python:pip.bzl", "pip_utils")
load("@rules_python//python/pip_install:pip_repository.bzl", "whl_config_repo", "whl_library")
load("@rules_python//python/private/pypi:labels.bzl", "requirement_label", "requirement_labels") # buildifier: disable=bzl-visibility

all_requirements = [
"@my_project_pip_deps_vendored_certifi//:pkg",
Expand Down Expand Up @@ -79,7 +80,28 @@ _config = {
_annotations = {}

def requirement(name):
return "@my_project_pip_deps_vendored_{}//:{}".format(pip_utils.normalize_name(name), "pkg")
"""Return the label for the package.

Args:
name: The name of the package with at most one extra, e.g. `foo`,
`foo[]` or `foo[bar]`. Use `requirements` for multiple extras.

Returns:
The label of the package.
"""
return requirement_label("@my_project_pip_deps_vendored_{}//:{}", name)

def requirements(*names):
"""Return the labels for the packages.

Args:
*names: The names of the packages with optional extras, e.g. `foo`,
`foo[]`, `foo[bar]` or `foo[bar,baz]`.

Returns:
The list of labels, with a label for each extra of a package.
"""
return requirement_labels("@my_project_pip_deps_vendored_{}//:{}", *names)

def whl_requirement(name):
return "@my_project_pip_deps_vendored_{}//:{}".format(pip_utils.normalize_name(name), "whl")
Expand Down Expand Up @@ -107,8 +129,10 @@ def install_deps(**whl_library_kwargs):
# Set up the requirement groups
all_requirement_groups = {}

# The members are the packages, e.g. `foo`, or the extras of the packages,
# e.g. `foo[bar]`.
requirement_group_mapping = {
requirement: group_name
requirement.partition("[")[0]: group_name
for group_name, group_requirements in all_requirement_groups.items()
for requirement in group_requirements
}
Expand Down
17 changes: 17 additions & 0 deletions news/2786.added.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
(pypi) The hub repositories now have a `[]` target for each package without
any extras and an `[extra]` target for each extra used in the lock file, e.g.
`@pypi//google_api_core:[grpc]`, in both bzlmod and `WORKSPACE`. The
`requirement` macro accepts a single extra, e.g.
`requirement("google-api-core[grpc]")`, and the new `requirements` macro
returns a list of labels for any number of packages and extras, e.g.
`requirements("google-api-core[async_rest,grpc]", "numpy")`. The whl
repositories now create the targets for every extra that the package provides
and the requested extras are no longer passed to them, so changing the
requested extras does not refetch them. The dependencies use the extras from
the wheel `METADATA` and the build fails with a helpful message if an extra is
missing in the lock file. The `whl` targets no longer include the dependencies
of the extras. The members of `experimental_requirement_cycles` can now be the
extras of the packages, e.g. `foo[bar]`. See
[the docs](https://rules-python.readthedocs.io/en/latest/pypi/use.html#extras)
([#2786](https://github.com/bazel-contrib/rules_python/issues/2786),
[#4168](https://github.com/bazel-contrib/rules_python/issues/4168)).
3 changes: 3 additions & 0 deletions news/4395.added.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
(pypi) Added {obj}`pip.configure` to limit the packages that a hub exposes to
the ones listed in the dependency groups of a `pyproject.toml` file
([#3413](https://github.com/bazel-contrib/rules_python/issues/3413)).
5 changes: 5 additions & 0 deletions news/4522.added.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
(pypi) {obj}`pip.configure` now exposes the extras requested in the selected
`pyproject.toml` dependency groups, e.g. `requests[socks]`, as the
`@pypi//requests:[socks]` target and includes them in `@pypi//requests`, even
if the lock file does not request them
([#2786](https://github.com/bazel-contrib/rules_python/issues/2786)).
26 changes: 16 additions & 10 deletions python/private/internal_dev_deps.bzl
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,6 @@ def _internal_dev_deps_impl(mctx):
whl_library(
name = "somepkg_with_build_files",
whl_file = "@whl_with_build_files//:somepkg-1.0-any-none-any.whl",
requirement = "somepkg",
)

# Setup for //tests/implicit_namespace_packages
Expand All @@ -51,7 +50,6 @@ def _internal_dev_deps_impl(mctx):
whl_library(
name = "implicit_namespace_ns_sub1",
whl_file = "@implicit_namespace_ns_sub1_whl//:ns_sub1-1.0-any-none-any.whl",
requirement = "ns-sub1",
enable_implicit_namespace_pkgs = False,
)

Expand All @@ -63,7 +61,6 @@ def _internal_dev_deps_impl(mctx):
whl_library(
name = "implicit_namespace_ns_sub2",
whl_file = "@implicit_namespace_ns_sub2_whl//:ns_sub2-1.0-any-none-any.whl",
requirement = "ns-sub2",
enable_implicit_namespace_pkgs = False,
)

Expand All @@ -75,7 +72,6 @@ def _internal_dev_deps_impl(mctx):
whl_library(
name = "pkgutil_nspkg1",
whl_file = "@pkgutil_nspkg1_whl//:pkgutil_nspkg1-1.0-any-none-any.whl",
requirement = "pkgutil_nspkg1",
enable_implicit_namespace_pkgs = False,
)

Expand All @@ -87,7 +83,6 @@ def _internal_dev_deps_impl(mctx):
whl_library(
name = "pkgutil_nspkg2",
whl_file = "@pkgutil_nspkg2_whl//:pkgutil_nspkg2-1.0-any-none-any.whl",
requirement = "pkgutil_nspkg2",
enable_implicit_namespace_pkgs = False,
)

Expand All @@ -99,7 +94,6 @@ def _internal_dev_deps_impl(mctx):
whl_library(
name = "whl_with_data1",
whl_file = "@whl_with_data1_whl//:whl_with_data1-1.0-any-none-any.whl",
requirement = "whl-with-data1",
)

whl_from_dir_repo(
Expand All @@ -110,14 +104,28 @@ def _internal_dev_deps_impl(mctx):
whl_library(
name = "whl_with_data2",
whl_file = "@whl_with_data2_whl//:whl_with_data2-1.0-any-none-any.whl",
requirement = "whl-with-data2",
)

_whl_library_from_dir(
name = "whl_library_extras_direct_dep",
root = "//tests/pypi/whl_library/testdata/pkg:BUILD.bazel",
output = "pkg-1.0-any-none-any.whl",
requirement = "pkg[optional]",
# The following is necessary to enable pipstar and make tests faster
config_load = "@rules_python//tests/pypi/whl_library/testdata:packages.bzl",
dep_template = "@whl_library_extras_{name}//:{target}",
)
_whl_library_from_dir(
name = "whl_library_extras_pkg",
root = "//tests/pypi/whl_library/testdata/pkg:BUILD.bazel",
output = "pkg-1.0-any-none-any.whl",
# The following is necessary to enable pipstar and make tests faster
config_load = "@rules_python//tests/pypi/whl_library/testdata:packages.bzl",
dep_template = "@whl_library_extras_{name}//:{target}",
)
_whl_library_from_dir(
name = "whl_library_extras_consumer",
root = "//tests/pypi/whl_library/testdata/consumer:BUILD.bazel",
output = "consumer-1.0-any-none-any.whl",
# The following is necessary to enable pipstar and make tests faster
config_load = "@rules_python//tests/pypi/whl_library/testdata:packages.bzl",
dep_template = "@whl_library_extras_{name}//:{target}",
Expand All @@ -126,7 +134,6 @@ def _internal_dev_deps_impl(mctx):
name = "whl_library_extras_optional_dep",
root = "//tests/pypi/whl_library/testdata/optional_dep:BUILD.bazel",
output = "optional_dep-1.0-any-none-any.whl",
requirement = "optional_dep",
# The following is necessary to enable pipstar and make tests faster
config_load = "@rules_python//tests/pypi/whl_library/testdata:packages.bzl",
)
Expand All @@ -140,7 +147,6 @@ def _internal_dev_deps_impl(mctx):
whl_library(
name = "patch_whl_pkg",
whl_file = "@patch_whl_pkg_whl//:pkg-1.0-any-none-any.whl",
requirement = "pkg",
whl_patches = {
"//tests/pypi/patch_whl/testdata/patches:modify_pkg.patch": json.encode({
"patch_strip": 0,
Expand Down
Loading
Loading