Repository navigation
[CI] Update test fetcher script #14607
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Changes from all commits
341e55f
c7c8cc4
4242566
bd8d8b8
b19052c
7e674e0
dbc8d8a
51e0728
e258130
a8b2c8e
a5af804
bc5488a
bcfeaa9
7e67c09
8b5edc1
bc0e8cb
fee0b30
e44629d
8668474
966349c
5835f5f
40470d3
45905d6
14e737c
e409a9f
2bd7b43
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -30,7 +30,6 @@ jobs: | |
| shell: bash | ||
| outputs: | ||
| matrix: ${{ steps.set_matrix.outputs.matrix }} | ||
| test_map: ${{ steps.set_matrix.outputs.test_map }} | ||
| steps: | ||
| - name: Checkout diffusers | ||
| uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 | ||
|
|
@@ -44,42 +43,46 @@ jobs: | |
| diffusers-cli env | ||
| echo $(git --version) | ||
| - name: Fetch Tests | ||
| # The checkout step only marks the workspace safe in a temporary git config, so git processes the | ||
| # fetcher spawns in the container would otherwise refuse the repo as dubiously owned. | ||
| run: | | ||
| git config --global --add safe.directory "$GITHUB_WORKSPACE" | ||
| python utils/tests_fetcher.py | tee test_preparation.txt | ||
| - name: Report selection in job summary | ||
| run: | | ||
| if [ -f test_fetcher_summary.md ]; then | ||
| cat test_fetcher_summary.md >> $GITHUB_STEP_SUMMARY | ||
| fi | ||
| - name: Report fetched tests | ||
| uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1 | ||
| with: | ||
| name: test_fetched | ||
| path: test_preparation.txt | ||
| - id: set_matrix | ||
| name: Create Test Matrix | ||
| # The `keys` is used as GitHub actions matrix for jobs, i.e. `models`, `pipelines`, etc. | ||
| # The `test_map` is used to get the actual identified test files under each key. | ||
| # If no test to run (so no `test_map.json` file), create a dummy map (empty matrix will fail) | ||
| # `test_map.json` is a list of matrix entries `{"name", "paths", "markers"}`, one job each. | ||
| # If no test to run (so no `test_map.json` file), emit an empty list and skip the test job. | ||
| run: | | ||
| if [ -f test_map.json ]; then | ||
| keys=$(python3 -c 'import json; fp = open("test_map.json"); test_map = json.load(fp); fp.close(); d = list(test_map.keys()); print(json.dumps(d))') | ||
| test_map=$(python3 -c 'import json; fp = open("test_map.json"); test_map = json.load(fp); fp.close(); print(json.dumps(test_map))') | ||
| matrix=$(python3 -c 'import json; print(json.dumps(json.load(open("test_map.json"))))') | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Is it possible to have an example matrix for quick references?
Collaborator
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. [
{
"name": "others",
"paths": "tests/others/test_flashpack.py",
"markers": ""
},
{
"name": "pipelines-core",
"paths": "tests/pipelines/kandinsky3/test_kandinsky3.py tests/pipelines/kandinsky3/test_kandinsky3_img2img.py tests/pipelines/pag/test_pag_sd.py tests/pipelines/pag/test_pag_sd3_img2img.py tests/pipelines/pag/test_pag_sd_img2img.py tests/pipelines/pag/test_pag_sd_inpaint.py tests/pipelines/pag/test_pag_sdxl.py tests/pipelines/pag/test_pag_sdxl_img2img.py tests/pipelines/pag/test_pag_sdxl_inpaint.py tests/pipelines/stable_diffusion/test_stable_diffusion.py tests/pipelines/stable_diffusion_xl/test_stable_diffusion_xl.py",
"markers": "core"
},
{
"name": "pipelines-lora",
"paths": "<same 11 files>",
"markers": "lora"
},
{
"name": "pipelines-memory",
"paths": "<same 11 files>",
"markers": "memory or cpu_offload or group_offload"
},
{
"name": "pipelines-ip_adapter",
"paths": "<same 11 files>",
"markers": "ip_adapter"
}
] |
||
| else | ||
| keys=$(python3 -c 'keys = ["dummy"]; print(keys)') | ||
| test_map=$(python3 -c 'test_map = {"dummy": []}; print(test_map)') | ||
| matrix='[]' | ||
| fi | ||
| echo $keys | ||
| echo $test_map | ||
| echo "matrix=$keys" >> $GITHUB_OUTPUT | ||
| echo "test_map=$test_map" >> $GITHUB_OUTPUT | ||
| echo "matrix=$matrix" >> $GITHUB_OUTPUT | ||
| echo "Matrix: $matrix" | ||
|
|
||
| run_pr_tests: | ||
| permissions: | ||
| contents: read | ||
| name: Run PR Tests | ||
| id-token: write | ||
| name: ${{ matrix.name }} | ||
| needs: setup_pr_tests | ||
| if: contains(fromJson(needs.setup_pr_tests.outputs.matrix), 'dummy') != true | ||
| if: needs.setup_pr_tests.outputs.matrix != '[]' | ||
| strategy: | ||
| fail-fast: false | ||
| max-parallel: 2 | ||
| matrix: | ||
| modules: ${{ fromJson(needs.setup_pr_tests.outputs.matrix) }} | ||
| include: ${{ fromJson(needs.setup_pr_tests.outputs.matrix) }} | ||
| runs-on: | ||
| group: aws-general-8-plus | ||
| container: | ||
|
|
@@ -104,21 +107,27 @@ jobs: | |
| diffusers-cli env | ||
|
|
||
| - name: Run all selected tests on CPU | ||
| # The gated single-file checkpoints need a Hub token, obtained through OIDC like the nightly and release | ||
| # workflows. Only runs that can mint an id token get the resource: a fork PR cannot, and huggingface_hub | ||
| # raises rather than falling back when `HF_OIDC_RESOURCE` is set with no id token, so there it is left | ||
| # empty and the `require_hf_token` tests skip instead. | ||
| env: | ||
| HF_OIDC_RESOURCE: ${{ (github.event_name != 'pull_request' || github.event.pull_request.head.repo.full_name == github.repository) && 'Diffusers-CI-Bot' || '' }} | ||
| run: | | ||
| pytest -n 2 --dist=loadfile -v --make-reports=${{ matrix.modules }}_tests_cpu ${{ fromJson(needs.setup_pr_tests.outputs.test_map)[matrix.modules] }} | ||
| pytest -n 2 --dist=loadfile -m "${{ matrix.markers }}" --make-reports=${{ matrix.name }}_tests_cpu ${{ matrix.paths }} | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Possible to include a comment to show example markers and name? If possible, I would also increase the number of workers to 4. Doesn't hurt I guess. |
||
|
|
||
| - name: Failure short reports | ||
| if: ${{ failure() }} | ||
| continue-on-error: true | ||
| run: | | ||
| cat reports/${{ matrix.modules }}_tests_cpu_stats.txt | ||
| cat reports/${{ matrix.modules }}_tests_cpu_failures_short.txt | ||
| cat reports/${{ matrix.name }}_tests_cpu_stats.txt | ||
| cat reports/${{ matrix.name }}_tests_cpu_failures_short.txt | ||
|
|
||
| - name: Test suite reports artifacts | ||
| if: ${{ always() }} | ||
| uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1 | ||
| with: | ||
| name: ${{ matrix.modules }}_test_reports | ||
| name: ${{ matrix.name }}_test_reports | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I think we're still not fetching the GPU tests for new pipeline PRs? For example, on MiniMax H3 Image (it's not limited to MiniMax H3 Image), I had to trigger the fast GPU tests manually. This should not be the case right?
Collaborator
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Yes, the changes here are specifically for CPU PR tests. GPU test support will be added in a follow up. |
||
| path: reports | ||
|
|
||
| run_staging_tests: | ||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -7,12 +7,29 @@ extend-exclude = [ | |
| [tool.ruff.lint] | ||
| # Never enforce `E501` (line length violations). | ||
| ignore = ["C901", "E501", "E721", "E741", "F402", "F823"] | ||
| select = ["C", "E", "F", "I", "W"] | ||
| select = ["C", "E", "F", "I", "W", "TID251"] | ||
|
|
||
| # `utils/tests_fetcher.py` selects the tests a PR runs from the import graph. `auto_pipeline.py` imports every | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Does it interfere with the actual testing of the auto pipelines? For example, if a PR concerns auto pipelines, how does the fetcher react to it?
Collaborator
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. For the actual auto-pipeline test we ignore this rule: A PR touching |
||
| # pipeline, so a test that imports an AutoPipeline is selected on every pipeline change. Tests use the concrete | ||
| # pipeline class instead; the auto pipelines' own test is the one exception. | ||
| [tool.ruff.lint.flake8-tidy-imports.banned-api] | ||
| "diffusers.AutoPipelineForText2Image".msg = "AutoPipeline imports are not allowed in tests because they interfere with the test fetcher's selection logic. Use the specific pipeline class instead (e.g. `FluxPipeline`)." | ||
| "diffusers.AutoPipelineForImage2Image".msg = "AutoPipeline imports are not allowed in tests because they interfere with the test fetcher's selection logic. Use the specific pipeline class instead (e.g. `FluxPipeline`)." | ||
| "diffusers.AutoPipelineForInpainting".msg = "AutoPipeline imports are not allowed in tests because they interfere with the test fetcher's selection logic. Use the specific pipeline class instead (e.g. `FluxPipeline`)." | ||
| "diffusers.AutoPipelineForText2Audio".msg = "AutoPipeline imports are not allowed in tests because they interfere with the test fetcher's selection logic. Use the specific pipeline class instead (e.g. `FluxPipeline`)." | ||
|
|
||
| # Ignore import violations in all `__init__.py` files. | ||
| [tool.ruff.lint.per-file-ignores] | ||
| "__init__.py" = ["E402", "F401", "F403", "F811"] | ||
| "src/diffusers/utils/dummy_*.py" = ["F401"] | ||
| # The AutoPipeline ban only concerns test selection; everything outside tests/ may use them. | ||
| "tests/pipelines/test_pipelines_auto.py" = ["TID251"] | ||
| "src/**" = ["TID251"] | ||
| "examples/**" = ["TID251"] | ||
| "benchmarks/**" = ["TID251"] | ||
| "scripts/**" = ["TID251"] | ||
| "utils/**" = ["TID251"] | ||
| "docs/**" = ["TID251"] | ||
|
|
||
| [tool.ruff.lint.isort] | ||
| lines-after-imports = 2 | ||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -15,10 +15,13 @@ | |
| # tests directory-specific settings - this file is run automatically | ||
| # by pytest before any tests are run | ||
|
|
||
| import ast | ||
| import sys | ||
| import warnings | ||
| from os.path import abspath, dirname, join | ||
|
|
||
| import pytest | ||
|
|
||
|
|
||
| # allow having multiple repository checkouts and not needing to remember to rerun | ||
| # 'pip install -e .[dev]' when switching between checkouts and running tests. | ||
|
|
@@ -30,7 +33,49 @@ | |
| warnings.simplefilter(action="ignore", category=FutureWarning) | ||
|
|
||
|
|
||
| def marker_decorators(): | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Consider providing an example of how a minimal |
||
| """Map each `is_*` decorator in `tests/testing_utils.py` to the pytest marker it applies. | ||
|
|
||
| Only that file is read: it is where the marker decorators are defined. The `testing_utils/` packages | ||
| under `tests/models/` and `tests/pipelines/` hold the tester mixins that *use* these decorators, and the | ||
| per-pipeline `testing_utils.py` files hold test helpers; neither defines markers. Read from the source | ||
| with `ast` rather than imported, so `utils/tests_fetcher.py` can call this without importing torch. | ||
| Matches `def is_x(test_case): return pytest.mark.<marker>(test_case)`, so with | ||
|
|
||
| def is_lora(test_case): | ||
| return pytest.mark.lora(test_case) | ||
|
|
||
| def is_torch_compile(test_case): | ||
| return pytest.mark.compile(test_case) | ||
|
|
||
| in `tests/testing_utils.py` the result is `{"is_lora": "lora", "is_torch_compile": "compile"}`. A new `is_*` | ||
| decorator is picked up automatically; nothing else needs registering. | ||
| """ | ||
| tree = ast.parse(open(join(dirname(__file__), "testing_utils.py"), encoding="utf-8").read()) | ||
| decorators = {} | ||
| for node in ast.walk(tree): | ||
| if not isinstance(node, ast.FunctionDef) or not node.name.startswith("is_"): | ||
| continue | ||
| for ret in ast.walk(node): | ||
| if isinstance(ret, ast.Return) and isinstance(ret.value, ast.Call): | ||
| func = ret.value.func | ||
| if ( | ||
| isinstance(func, ast.Attribute) | ||
| and isinstance(func.value, ast.Attribute) | ||
| and func.value.attr == "mark" | ||
| ): | ||
| decorators[node.name] = func.attr | ||
| return decorators | ||
|
|
||
|
|
||
| # Every marker an `is_*` decorator applies takes a test out of the `core` set. CI runs each feature in its | ||
| # own job (`-m lora`, ...) and everything else as `-m core`, so a test carrying none of these is marked | ||
| # `core` at collection time. `utils/tests_fetcher.py` uses the same set to decide which jobs to schedule. | ||
| NON_CORE_MARKERS = set(marker_decorators().values()) | ||
|
|
||
|
|
||
| def pytest_configure(config): | ||
| config.addinivalue_line("markers", "core: marks tests carrying none of the feature markers (added automatically)") | ||
| config.addinivalue_line("markers", "big_accelerator: marks tests as requiring big accelerator resources") | ||
| config.addinivalue_line("markers", "lora: marks tests for LoRA/PEFT functionality") | ||
| config.addinivalue_line("markers", "ip_adapter: marks tests for IP Adapter functionality") | ||
|
|
@@ -54,6 +99,21 @@ def pytest_configure(config): | |
| config.addinivalue_line("markers", "nightly: mark test as nightly") | ||
|
|
||
|
|
||
| @pytest.hookimpl(tryfirst=True) | ||
| def pytest_collection_modifyitems(items): | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. What is this for?
Collaborator
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. It's a hook that marks any test without an explicit feature marker as
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. But the corresponding test could be a non-core. Just trying to understand situations where this might be helpful. I also don't see it used anywhere in the changeset of the PR.
Collaborator
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. The alternative is to manually mark With this approach, if non-core tests happen to fall into This is how we apply
Example matrix: [
{
"name": "others",
"paths": "tests/others/test_flashpack.py",
"markers": ""
},
{
"name": "pipelines-core",
"paths": "tests/pipelines/kandinsky3/test_kandinsky3.py tests/pipelines/kandinsky3/test_kandinsky3_img2img.py tests/pipelines/pag/test_pag_sd.py tests/pipelines/pag/test_pag_sd3_img2img.py tests/pipelines/pag/test_pag_sd_img2img.py tests/pipelines/pag/test_pag_sd_inpaint.py tests/pipelines/pag/test_pag_sdxl.py tests/pipelines/pag/test_pag_sdxl_img2img.py tests/pipelines/pag/test_pag_sdxl_inpaint.py tests/pipelines/stable_diffusion/test_stable_diffusion.py tests/pipelines/stable_diffusion_xl/test_stable_diffusion_xl.py",
"markers": "core"
},
{
"name": "pipelines-lora",
"paths": "<same 11 files>",
"markers": "lora"
},
{
"name": "pipelines-memory",
"paths": "<same 11 files>",
"markers": "memory or cpu_offload or group_offload"
},
{
"name": "pipelines-ip_adapter",
"paths": "<same 11 files>",
"markers": "ip_adapter"
}
]A sample run: |
||
| """Mark every test carrying none of `NON_CORE_MARKERS` as `core`. | ||
|
|
||
| CI runs the feature jobs with `-m lora`, `-m memory`, ... and everything else with `-m core`. A positive | ||
| marker is the only way to say "everything else" to pytest: `-m "not (lora or memory or ...)"` would have | ||
| to list every feature marker and silently go stale. Adding the marker here, to every collected test | ||
| without a feature marker, keeps `core` exhaustive by construction. `tryfirst` runs this before pytest's | ||
| own `-m` deselection, which happens in the same hook. | ||
| """ | ||
| for item in items: | ||
| if not any(m.name in NON_CORE_MARKERS for m in item.iter_markers()): | ||
| item.add_marker("core") | ||
|
|
||
|
|
||
| def pytest_addoption(parser): | ||
| from .testing_utils import pytest_addoption_shared | ||
|
|
||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -22,8 +22,7 @@ | |
| from transformers import AutoConfig, AutoTokenizer, T5EncoderModel | ||
|
|
||
| from diffusers import ( | ||
| AutoPipelineForImage2Image, | ||
| AutoPipelineForText2Image, | ||
|
Comment on lines
-25
to
-26
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I think it's then better to replace them with their exact pipeline class so that we don't have to do the inline import. Doing it this way also preserves the expected behaviour in the dependency map? Applies elsewhere, too. |
||
| Kandinsky3Img2ImgPipeline, | ||
| Kandinsky3Pipeline, | ||
| Kandinsky3UNet, | ||
| VQModel, | ||
|
|
@@ -177,7 +176,7 @@ def cleanup(self): | |
| backend_empty_cache(torch_device) | ||
|
|
||
| def test_kandinskyV3(self): | ||
| pipe = AutoPipelineForText2Image.from_pretrained( | ||
| pipe = Kandinsky3Pipeline.from_pretrained( | ||
| "kandinsky-community/kandinsky-3", variant="fp16", torch_dtype=torch.float16 | ||
| ) | ||
| pipe.enable_model_cpu_offload(device=torch_device) | ||
|
|
@@ -203,7 +202,7 @@ def test_kandinskyV3(self): | |
| assert np.allclose(image_np, expected_image_np, atol=5e-2) | ||
|
|
||
| def test_kandinskyV3_img2img(self): | ||
| pipe = AutoPipelineForImage2Image.from_pretrained( | ||
| pipe = Kandinsky3Img2ImgPipeline.from_pretrained( | ||
| "kandinsky-community/kandinsky-3", variant="fp16", torch_dtype=torch.float16 | ||
| ) | ||
| pipe.enable_model_cpu_offload(device=torch_device) | ||
|
|
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The fetcher is currently in
workflow_dispatch. Should it have apull_requesttrigger? And then we should also make sure that the fetching script is always run from the base upstream (for security reasons)?There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
So I think we would run this workflow manually for a little bit to verify that everything is working as expected and then run it by default on PRs and remove the current PR tests.