Skip to content

Revert "Add git gc step to the end of every build (#790)" - #833

Merged
zware merged 1 commit into
python:mainfrom
zware:remove_git_gc
Nov 17, 2026
Merged

zware merged 1 commit into
python:mainfrom
zware:remove_git_gc

Conversation

@zware

@zware zware commented Nov 13, 2026

Copy link
Copy Markdown
Member

This reverts (most of) commit b082fae.

@vstinner

Copy link
Copy Markdown
Member

Can you elaborate why you remove the git gc command? I added it to save a lot of disk space on buildbot workers.

@mhsmith

mhsmith commented Nov 14, 2026 •

Copy link
Copy Markdown
Member

Since the buildbot master was updated to version 4.4.0 on November 12, the iOS and both Android workers have started taking up to 15 minutes to run git gc, where previously it was less than 1:

I think this may be something to do with the "blobless clone" feature mentioned in the b082fae commit message.

@zware
zware merged commit 4c662a6 into python:main Nov 17, 2026
1 check passed
@zware
zware deleted the remove_git_gc branch November 17, 2026 15:02
@mhsmith

mhsmith commented Nov 18, 2026 •

Copy link
Copy Markdown
Member

I don't think this has worked as expected. Since this PR was merged, the mhsmith-android-x86_64 worker has done a build on every builder except 3.13. And every worker except 3.13, and pull_request for some reason, is now consuming 6-7 GB in its .git directory:

buildbot@chaquo:~/buildarea$ du -scm */*/.git
256     3.13.mhsmith-android-x86_64/build/.git
6375    3.14.mhsmith-android-x86_64/build/.git
6752    3.15.mhsmith-android-x86_64/build/.git
7083    3.x.mhsmith-android-x86_64/build/.git
285     pull_request.mhsmith-android-x86_64/build/.git
20748   total

Let me know if you need any more information.

@zware

zware commented Nov 18, 2026

Copy link
Copy Markdown
Member Author

It occurs to me that I'm not sure; @stratakis, was the shallow clone fix on the master or worker side of buildbot?

@mhsmith

mhsmith commented Nov 18, 2026

Copy link
Copy Markdown
Member

Looking at https://buildbot.python.org/#/builders/1591/builds/5705/steps/1/logs/stdio, the blob:none does appear in the log, but so does this message:

Auto packing the repository in background for optimum performance.
See "git help gc" for manual housekeeping.
warning: The last gc run reported the following. Please correct the root cause
and remove .git/gc.log
Automatic cleanup will not be performed until the file is removed.

warning: There are too many unreachable loose objects; run 'git prune' to remove them.

I removed .git/gc.log and ran git prune in 3.x.mhsmith-android-x86_64, but it only reduced the size of .git from 7083 to 7004 MB. Then I ran git gc, which took more than 5 minutes but still only reduced it to 6586 MB.

I'll try doing the same in 3.15.mhsmith-android-x86_64 but without running git gc, and report back with what happens to each directory after the next build.

@zware

zware commented Nov 18, 2026

Copy link
Copy Markdown
Member Author

Could you also experiment with completely removing one of the workdirs to see if it still balloons up when starting from scratch?

@mhsmith

mhsmith commented Nov 18, 2026

Copy link
Copy Markdown
Member

Good idea: I've removed the 3.14 directory.

@vstinner

Copy link
Copy Markdown
Member

Could you also experiment with completely removing one of the workdirs to see if it still balloons up when starting from scratch?

I just removed all build directories from Fedora Stable x64-64. I wanted to do it yesterday: I stopped the systemd service and... ooops, I forgot to remove directories and restart the service :-( It's now done.

@vstinner

Copy link
Copy Markdown
Member

I just removed all build directories from Fedora Stable x64-64.

And 1h later: 3.x.cstratak-fedora-stable-x86_64.refleak/ build directory has been created. Its .git/ sub-directory only take 278 MB, it's similar to what we had before on this machine.

The whole build directory takes 830 MB: that's more surprising, I measured 413 MB before (but I measured cstratak-fedora-stable-x86_64/ before which uses different configure options).

This Fedora 45 machine is running git version 2.56.1.

@mhsmith

mhsmith commented Nov 18, 2026 •

Copy link
Copy Markdown
Member

The directories where I only ran git prune or git gc did not get any smaller after their next run. But the one I removed has now been recreated and is only 393 MB after two runs. So I guess that's what needs to be done.

It might be a good idea to email the buildbot owners mailing list to advise everyone to do the same.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants