Skip to content

fix(bitbucket): extract only new raw rows in Extract Pipelines - #9431

Open
eahmad-fhz wants to merge 1 commit into
apache:mainfrom
eahmad-fhz:fix/bitbucket-pipeline-extractor-incremental
Open

eahmad-fhz wants to merge 1 commit into
apache:mainfrom
eahmad-fhz:fix/bitbucket-pipeline-extractor-incremental

Conversation

@eahmad-fhz

Copy link
Copy Markdown

Summary

Part of #9430 (extractor half).

Bitbucket Cloud ignores the updated_on filter on /pipelines, so every collection appends a full copy of all pipelines to _raw_bitbucket_api_pipelines. Extract Pipelines used NewApiExtractor, which re-reads every raw row, so the subtask grows without bound (485 of 536 minutes in the reporting instance).

This switches it to NewStatefulApiExtractor, the same change as #9313 (CircleCI):

  • An incremental run extracts only the raw rows added since the previous run.
  • The deployment/production patterns are passed as SubtaskConfig, so changing them triggers a full re-extract (they set type and environment on every row).
  • A full sync still reads all raw rows.

The collector is unchanged; raw table growth stays open in #9430.

Does this close any open issues?

No. This fixes the extractor half of #9430; the collector half (raw table growth) stays open there.

Tests

New e2e test TestBitbucketPipelineExtractorIncremental (same shape as the CircleCI one in #9313):

  1. A first run extracts every pipeline.
  2. A tool row is deleted and a newer raw row is appended for another pipeline. The incremental run must not restore the deleted row, and must apply the newer row.
  3. Changing the production pattern must re-extract all rows.
  • Fails on main at step 2 ("an incremental run must not read old raw rows again").
  • Fails at step 3 if SubtaskConfig is removed.
  • Passes with this change, along with the existing Bitbucket e2e tests (PostgreSQL 14).

Other Information

go vet and gofmt are clean on the touched packages.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant