Repository navigation
Conversation
If creating a symlink fails, e.g. on Windows without the privilege to create symlinks, the venv that is created at runtime uses hard links, junctions or copies instead, and its pyvenv.cfg sets home, because Python can't find it from the interpreter otherwise. The RULES_PYTHON_TESTING_DISABLE_SYMLINKS environment variable forces the fallback, so that it can be tested where symlinks work.
|
This PR was written by an AI agent (Claude Code) working for @aignas. Review comments are read and addressed by the agent, and every reply and change is reviewed and approved by @aignas before it is posted or pushed. The new Windows CI job is experimental: it checks whether This comment was generated by an AI agent and approved by @aignas. |
Revert this before the PR leaves draft.
This reverts commit 0a739bf.
On Windows, the venv paths are Windows paths, which the bash checks couldn't handle.
Use the `? "--flag"` keys that presubmit_test requires, and download the remote cache outputs, because without symlinks Bazel creates a junction in the runfiles tree for an output that isn't downloaded yet.
|
I am not sure if it is the smallest possible change to fix the issue, but it is pretty small and if it works, I'll be chuffed. |
The fallback for when symlinks can't be created is now Windows only. Windows refuses to delete a read-only file and the attribute is shared by all the hard links of a file, so removing the venv made the hard linked Bazel outputs writable. The hard links are now removed first, and the attribute is restored on their originals.
Instead of restoring the read-only attribute of hard linked Bazel outputs after clearing it, delete the read-only files of the venv with FILE_DISPOSITION_FLAG_IGNORE_READONLY_ATTRIBUTE, which doesn't change the other hard links of a file. It needs Windows 10 1809 or Windows Server 2019, so older versions still require symlinks.
rickeylev
left a comment
There was a problem hiding this comment.
Several lines in the PR description reach 73–76 columns, exceeding the
72-column limit for merge commit messages in .editorconfig.
Suggested fix: Re-wrap the PR description body at 72 columns while preserving
the existing wording.
This is an agent-generated comment reviewed and approved by the user.
… globals Move them next to the other top-level globals and use a lower case name for the one that changes at runtime.
When Bazel could create the runfiles symlinks, but the venv falls back to hard links, os.link() hard linked the symlink itself, so sys.executable was still a symlink. Resolve the target first.
When processes share RULES_PYTHON_EXTRACT_ROOT, another one can create the entry between the existence check and creating the link, which made the fallback copy onto it. Also only catch OSError from _winapi.CreateJunction, which every supported Python on Windows has.
They were set up again for every read-only file that is removed. Also declare the return type of CloseHandle.
The fallback creates junctions for directories, and islink() is False for them, so a dangling one wasn't removed and makedirs() failed.
Remove the CR of the Windows output, so that the check that the venv was removed doesn't always pass, follow the runfiles symlinks when looking for read-only outputs and require that there are some, create the venvs under TEST_TMPDIR, and quote the paths.
…oved Check the behavior of lexists(), exists() and unlink() for a dangling junction, which the bootstrap relies on, and run the cached venv test with the fallback for when symlinks can't be created.
…ed venv test Windows can't delete a hard link to a running executable, so cleaning up the temporary directory failed after the fallback hard linked sys.executable into the venv. The venv isn't run, so link to a stand-in file instead.
rickeylev
left a comment
There was a problem hiding this comment.
LGTM in concept. If we can't create symlinks at build time, then we don't have much choice but to copy at build time, or do something at runtime. The runtime option is cheaper, so I'm in favor of that.
I suppose we could experiment with more exotic strategies, too. e.g. a custom importer, patching rpath on the interpreter, etc. Windows might be a decent place to prove that out, since our build time options are limited.
Pass `--test_env=RULES_PYTHON_TESTING_DISABLE_SYMLINKS=1` in `build_flags` too and share the mapping with `test_flags` via a YAML anchor so `bazel test` reuses the build step's analysis cache.
On Windows,
py_binaryandpy_testcreate a venv at runtime bysymlinking the interpreter, its
.dllfiles and thesite-packagesfiles into a temporary directory. Without the privilege to create
symlinks, this fails with
WinError 1314, which is the default forusers that can't enable Developer Mode.
On Windows, if creating a symlink fails, the bootstrap now uses hard
links for files, junctions for directories, and copies if those aren't
possible, e.g. when the temporary directory is on a different volume.
The runfiles files are already copies when Bazel can't create symlinks,
so they aren't moved, as that would break the next run that uses the
same runfiles tree. Other platforms are unchanged.
Windows refuses to delete read-only files, such as hard links to Bazel
outputs, and the read-only attribute is shared by all the hard links of
a file, so the venv's read-only files are deleted with
FILE_DISPOSITION_FLAG_IGNORE_READONLY_ATTRIBUTEinstead of clearingthe attribute. This needs Windows 10 1809 or Windows Server 2019, so
older versions still require the privilege to create symlinks.
The
RULES_PYTHON_TESTING_DISABLE_SYMLINKSenvironment variable forcesthe fallback on Windows, which a new test uses, and a new CI job uses
to build and test
examples/bzlmodwith--nowindows_enable_symlinks.Fixes #3792