Repository navigation
fix(bitbucket): extract only new raw rows in Extract Pipelines - #9431
Open
eahmad-fhz wants to merge 1 commit into
Open
eahmad-fhz wants to merge 1 commit into
eahmad-fhz wants to merge 1 commit into
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Part of #9430 (extractor half).
Bitbucket Cloud ignores the
updated_onfilter on/pipelines, so every collection appends a full copy of all pipelines to_raw_bitbucket_api_pipelines.Extract PipelinesusedNewApiExtractor, which re-reads every raw row, so the subtask grows without bound (485 of 536 minutes in the reporting instance).This switches it to
NewStatefulApiExtractor, the same change as #9313 (CircleCI):SubtaskConfig, so changing them triggers a full re-extract (they settypeandenvironmenton every row).The collector is unchanged; raw table growth stays open in #9430.
Does this close any open issues?
No. This fixes the extractor half of #9430; the collector half (raw table growth) stays open there.
Tests
New e2e test
TestBitbucketPipelineExtractorIncremental(same shape as the CircleCI one in #9313):SubtaskConfigis removed.Other Information
go vetandgofmtare clean on the touched packages.