Skip to content

fix(github_graphql): extract only new raw rows in Extract Jobs - #9427

Open
shaynhornik wants to merge 1 commit into
apache:mainfrom
shaynhornik:fix/github-graphql-stateful-job-extractor
Open

shaynhornik wants to merge 1 commit into
apache:mainfrom
shaynhornik:fix/github-graphql-stateful-job-extractor

Conversation

@shaynhornik

Copy link
Copy Markdown

Summary

Extract Jobs in github_graphql was built with api.NewApiExtractor, which deletes and saves again every job of the repo on every run. The run time grew with the CI history, and because every _tool_github_jobs row got a new updated_at, the incremental Convert Jobs converted all of them again too (#9421).

It now uses api.NewStatefulApiExtractor, like plugins/github/tasks/cicd_job_extractor.go, so an incremental run reads only the raw rows the collector added since the previous run. The field mapping is unchanged.

One addition compared with the REST extractor: SubtaskConfig: data.RegexEnricher.PlainMap(). A job's Type and Environment come from the scope config's deployment and production patterns, so a change of those patterns extracts every job again. Without it, a re-transform after a pattern change would leave old jobs classified the old way.

When every row is still extracted:

  • a full sync;
  • the first run after the upgrade, once (there is no extractor state yet);
  • after a change of the deployment or production pattern.

Why incremental is safe here: Collect Jobs is a stateful GraphQL collector that only deletes raw rows on a non-incremental run and otherwise inserts a new raw row for a changed job. Rows are read in id order and upserted, so the newest raw row wins.

Does this close any open issues?

Closes #9421

Tests

New e2e test TestGithubGraphqlJobExtractorIncremental with a raw fixture of 5 check runs:

  1. With no stored state, every raw row is extracted and matches the snapshot.
  2. After a new raw row is added for a job that finished, only that job is written; the other four keep their updated_at.
  3. After the production pattern changes, every row is extracted again and Environment changes on an old job.
  4. A deleted tool row is not restored by an incremental run, and is restored by a full sync.

Against the old extractor the test fails on the updated_at checks.

Run locally:

  • go test ./plugins/github_graphql/e2e/... on MySQL 8.4 and PostgreSQL 18, and the new test 10 times on each.
  • go test ./plugins/github_graphql/tasks/... ./plugins/github_graphql/impl/... ./helpers/pluginhelper/api/...
  • go vet and golangci-lint v2.15.0 on the plugin.

Not run: the full unit and e2e suites, and a real collection against GitHub.

Other Information

  • The issue also suggests the plugin's other extractors. I kept this PR to Extract Jobs. Releases and deployments look like the same one-step change; issues and pull requests write child rows and need more than that. I can follow up if you want them.

Generated-by: Claude Code (Claude Fable 5.2). I reviewed and tested the change.

🤖 Generated with Claude Code

Extract Jobs used api.NewApiExtractor, which deletes and saves again
every job of the repo on every run. The run time grew with the CI
history, and because every job got a new updated_at, the incremental
Convert Jobs converted all of them again as well.

Use api.NewStatefulApiExtractor, like the REST plugin does, so an
incremental run reads only the raw rows that the collector added since
the previous run. A full sync, the first run after the upgrade and a
change of the deployment or production pattern still extract every row.

Generated-by: Claude Code (Claude Fable 5.2)
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug][GitHub GraphQL] extractJobs reads/rewrites the full raw table on every run, so collection time grows without bound

1 participant