Skip to content

GH-55292: [C++][Dataset] Prune Parquet row groups for is_in predicates with large value sets - #55293

Open
rok wants to merge 17 commits into
apache:mainfrom
rok:GH-55292-is-in-row-group-pruning
Open

rok wants to merge 17 commits into
apache:mainfrom
rok:GH-55292-is-in-row-group-pruning

Conversation

@rok

@rok rok commented Oct 27, 2026 •

Copy link
Copy Markdown
Member

Rationale for this change

Since #46859, is_in filters with more than 50 values are not checked against Parquet row group statistics, so they read every row group. Separately, row groups that may contain nulls are never pruned by any filter, because their statistics guarantee, (x >= min and x <= max) or is_null(x), is not recognized. See #55292.

What changes are included in this PR?

  • Large is_in. The is_in kernel state keeps its value set and sorts it the first time a filter is simplified against a guarantee, so plain kernel calls never sort. For each row group, two binary searches (search_sorted) tell whether any value lies between its min and max. Value sets longer than 2^17 are not sorted, only bounded by their smallest and largest value. Value sets of up to 50 values keep the existing path.
  • Kernel interface. SimplifyWithGuarantee asks the kernel what the call evaluates to over a range through a small internal interface (RangeEvaluator), so the null matching rules of is_in stay in the kernel.
  • Row groups with nulls. The nullable range guarantee is recognized, so comparisons and is_in prune such row groups too.
  • One pass. All bounds on a field are applied together, in a single pass over the filter. This is what lets a row group be skipped when it lies inside the span of the value set but holds none of its values.
  • Types must agree. The two additions only apply when the filter is on the column itself and the statistics (and a small value set) have the column's type. The existing handling of casts is unchanged.
  • Small is_in value sets: the deprecated skip_nulls option is honored, and sets containing NaN are left alone, since statistics cannot bound NaN.

Are these changes tested?

Yes, with new expression tests and two Parquet dataset tests that check which row groups are read. The SimplifyFilterWithGuarantee benchmark gains cases for a 100,000-value is_in against a row group guarantee: simplifying takes about 1.8 µs per row group, against about 0.3 µs on main, which skips such filters.

Scan timings on a 49.3M row file with 47 row groups of 2^20 rows and a sorted int64 id column (release build, macOS arm64, median of 5 scans, each binding a fresh filter):

is_in value set Row groups holding ids main This PR
100 ids 2 adjacent 411 ms 19 ms
100K ids 2 adjacent 573 ms 37 ms
1M ids 2 adjacent 1489 ms 79 ms
1M ids 2, twenty row groups apart 1443 ms 682 ms
10M ids 11 adjacent 1232 ms 481 ms
1K ids spread over all 47 563 ms 552 ms

The fourth row shows the limit for value sets longer than 2^17: the row groups in between are read too.

Are there any user-facing changes?

No API changes. Large is_in filters, and filters on row groups with nulls, now skip row groups that cannot match. Not covered: decimal, fixed size binary and view value sets, dictionary columns, and columns read as another type than they are stored as.

Was AI used for this PR?

PR code and description written by:

  • Human
  • AI

Reviewed before submission by:

  • Human
  • AI
  • Not reviewed

The code was drafted with an AI assistant; I reviewed, tested and benchmarked it and iterated on the design.

@github-actions

Copy link
Copy Markdown

⚠️ GitHub issue #55292 has been automatically assigned in GitHub to PR creator.

rok added 4 commits November 3, 2026 18:12
… sets using a sorted value set

Since apacheGH-46777 is_in predicates with more than 50 values are not simplified
against row group statistics at all, because filtering the value set and
rebinding the call for every guarantee was prohibitively expensive.

Large value sets are now simplified with a binary search instead:

- The is_in kernel init keeps a copy of the value set sorted in the order of
  the comparison kernels, with nulls removed. An already sorted value set is
  reused as is after a single O(N) check; an unsorted one is sorted lazily and
  thread-safely on the first simplification, so eager kernel calls never pay
  for sorting. Value sets containing NaN are not searchable, since statistics
  cannot bound NaN.
- SimplifyWithGuarantee applies all inequality guarantees on a field to a
  large is_in call at once, narrowing an index range into the sorted value set
  with lower/upper bound searches. If the range is empty the call simplifies to
  false (subject to the null matching behavior), otherwise it is left
  unchanged. This is exact for any number of values, costs O(log N) per
  guarantee and never rebinds. Applying both bounds together is what allows
  pruning a row group inside the hull of the value set but holding none of its
  values, which no per-bound check can do.

Value sets up to 50 values keep using the existing filter-and-rebind path.
…d of via sort_indices + take

Sorting a copy of the values with std::sort is about 14x faster than the
generic sort_indices + take path for a shuffled set of 10M int64 values
(0.19s vs 2.6s). Nulls are dropped while copying and a set containing NaN is
reported as not searchable before sorting, since NaN would break the strict
weak ordering std::sort requires. Non-primitive types (binary, decimal) keep
using the generic path.
…in is_in pruning

Extract nullable min/max bounds from Parquet statistics for large value sets. Respect GetNullMatchingBehavior() in both simplification paths and when rebinding narrowed sets.

Add expression and Parquet regressions for nullable ranges and conflicting deprecated skip_nulls options.
…and prune nullable ranges in every simplification path

Inequality guarantees are now restated through the order preserving casts
wrapping a filter's argument before they are applied: strict bounds are
weakened to inclusive ones and the bound is put through the same casts,
since e.g. f64 = 51.00000001 satisfies f64 > 51 but casts to 51f. This
applies to plain comparisons and small is_in value sets as well as the
large value set path, which previously was the only one handling it.

The nullable range guarantee produced by Parquet statistics for row groups
with nulls, (x >= min and x <= max) or is_null(x), is now extracted for all
simplifications rather than only for large is_in value sets, and an
is_valid(x) member of the guarantee rules out nulls for the inequalities on
x.
@rok
rok force-pushed the GH-55292-is-in-row-group-pruning branch from a3f317c to 79750a7 Compare November 3, 2026 17:12
rok added 4 commits November 3, 2026 19:23
…L and INCONCLUSIVE calls

- Defer concatenating, checking and sorting the value set to the first
  GetSortedValueSet() call (std::call_once), so plain is_in kernel calls pay
  nothing extra at init.
- Simplify an is_in call without possible matches to its exact result for
  every null matching behavior: invert(true_unless_null(arg)) for EMIT_NULL
  and INCONCLUSIVE, or a null literal for INCONCLUSIVE with a null in the
  value set, so these prune row groups too.
- Factor out Comparison::IsOrderPreservingCast, derive
  IsSearchableValueSetType from the accessor trait, and drop unreachable
  NaN, type and comparison checks.
- Test chunked value sets and the new null matching results.
…pruning and condense tests

- Sort the value set with sort_indices + take and search it with
  search_sorted instead of hand-written visitors. Decimal and fixed size
  binary value sets are no longer searchable since search_sorted does not
  support them.
- Make the large is_in simplification tests table-driven, merge the null
  matching tests, and share a helper which checks that simplified filters
  evaluate like the originals.
- Merge the two Parquet row group pruning tests into one file whose row
  groups contain nulls.
…and max instead of sorting them

Sorting a value set costs 1.5-3x as much as building the is_in lookup
table, so for 10M shuffled ids the one-time sort outweighed the row groups
it pruned and scans got slower than without pruning. Value sets longer
than kIsInMaxSortedValueSet (2^17) are now only bounded by their smallest
and largest value (min_max), which is cheap at any size and prunes as well
when the values are clustered.
…ith a cast

MSVC flags constructing a std::vector<int32_t> from int64_t iterators as a
narrowing conversion (C4244), which fails the Windows build.
rok added 4 commits November 10, 2026 13:57
…ered alike and keep NaN in small is_in value sets

A guarantee bound of another type was brought into the field's type whenever a
safe cast round tripped. Strings and numbers round trip but are ordered
differently, so the statistics of a string column read as int64 pruned row
groups holding matches of a large is_in. Bounds are now only converted between
types which are ordered alike.

Value sets of up to 50 values containing NaN are no longer filtered against
inequality guarantees: statistics cannot bound NaN, which is_in matches.
…ge and simplify against per-field ranges

Expression simplification no longer reaches into the is_in kernel state.
A kernel state may implement the new internal RangeEvaluator interface to
tell what its call evaluates to when the argument lies in a range; is_in
implements it with its lazily sorted value set and its null matching rules,
and scalar_set_lookup_internal.h is removed.

SimplifyWithGuarantee folds the guarantee once into the inequalities and
nullability of each field and simplifies every call in a single pass.
Guarantee bounds are converted by one rule (Inequality::Restate) for
comparisons, small is_in value sets and range evaluation alike: a bound of
another type must be ordered like the field's type and convert to it exactly.
…st the ones which matter

Removing each piece of the simplification in turn showed that the NaN check
on guarantee bounds in expression.cc duplicates the kernel's, and that decimal
bounds never converted to an integer or floating point field's type; both are
removed.

Guards which no test covered now have one: NaN in a sorted value set and in a
value set bounded by its smallest and largest value, a strict bound next to an
inclusive one at the same value, date bounds on a timestamp field, and index_in
never being simplified.
… instead of ignoring them

A bound of another type than the field's (the statistics of a float64 column
read as float32, say) was ignored unless it converted exactly, so comparisons
against such statistics were no longer simplified. The bound is now cast
safely and, unless it round trips, strict comparisons are weakened to
inclusive ones: 0.2 rounds up to 0.2f, which a stored 0.2 is read as.

Also apply clang-format.
@rok
rok marked this pull request as ready for review November 11, 2026 00:04
@rok
rok requested review from adamreeve and a balanced review from Copilot November 11, 2026 00:04

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The simplification compares uncast double entries against a rounded bound, potentially reducing a matching filter to false and incorrectly pruning rows.

1 open issue
What changed in this PR

Restores Parquet row-group pruning for large is_in filters without repeatedly filtering and rebinding value sets, while extending simplification to nullable statistics.

Changes:

  • Adds kernel range evaluation with lazy sorted-set lookup and min/max bounds for very large sets.
  • Combines per-field bounds and handles nullable guarantees, cast rounding, and null-matching options.
  • Adds expression-equivalence and Parquet row-group pruning tests.
File Description
cpp/​src/​arrow/​dataset/​file_parquet_test.cc Tests large-set and nullable row-group pruning.
cpp/​src/​arrow/​compute/​range_evaluation_internal.h Defines the internal kernel range-evaluation interface.
cpp/​src/​arrow/​compute/​kernels/​scalar_set_lookup.cc Adds cached value-set range evaluation for is_in.
cpp/​src/​arrow/​compute/​expression.cc Integrates per-field ranges and cast-aware simplification.
cpp/​src/​arrow/​compute/​expression_test.cc Covers ranges, coercion, null behavior, and execution equivalence.
cpp/​src/​arrow/​compute/​expression_internal.h Refines order-preserving cast detection to exclude wrapping casts.

Review effort: Balanced

Comment thread cpp/src/arrow/compute/expression.cc Outdated
std::string func_name = Comparison::GetName(cmp);
DCHECK_NE(func_name, "na");
std::vector<Datum> args{options->value_set, guarantee.bound};
std::vector<Datum> args{options->value_set, bound};

@adamreeve adamreeve left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks Rok. I'm still looking through this but have some comments/questions

Comment on lines +2121 to +2122
// Bounds which a safe cast rounds (the statistics of a float64 column read as
// float32, say) are used rounded, and then only bound inclusively: 0.2 rounds up

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this something that could happen in practice?

Comment on lines +2140 to +2141
auto string_range =
and_(greater_equal(i64, literal("10")), less_equal(i64, literal("9")));

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is going on here, is it really valid to compare an integer to a string?

This comment says "as are", but I'm confused about what this is referring to. The above test cases use rounding, but then these test cases all expect no change after simplification.

Comment thread cpp/src/arrow/compute/expression.cc Outdated
// value set must not be filtered out. Leave such value sets alone, as the
// kernel does for large ones.
const auto value_set_type_id = options->value_set.type()->id();
if (value_set_type_id == Type::FLOAT || value_set_type_id == Type::DOUBLE) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this should include HALF_FLOAT too.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually I think this isn't needed. HALF_FLOAT isn't in IsSearchableValueSetType, so this is probably fine.

// This base class enables non-templated access to the value set type
struct SetLookupStateBase : public KernelState {
/// Value sets up to this length are sorted for RangeEvaluator::EvaluateOverRange.
/// Sorting costs more than building the lookup table of the kernel, so longer value

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't follow the logic here. Why is the sorting time compared to a lookup table build? Avoiding row group reads is the cost that this is trying to avoid. It probably does make sense to fall-back to using the min/max of the set at some point, but I'd just like to better understand how the cutoff point is chosen.

…s type and test columns read as another type

The is_in kernel matches the argument against the value set cast to the
argument's type, but small value sets were filtered against a guarantee as
they were given. A value which the cast rounds could then be dropped although
it matches: is_in(cast(f64, float32), [51.00000001]) was simplified to false
given f64 <= 51.00000001. The value set is now cast first, and left alone if
it cannot be cast. This also lets a half float value set on a float32 field be
simplified, which failed for want of a comparison kernel.

IsNaNScalar in the set lookup kernel handles half floats.

SimplifyWithBoundOfAnotherType binds the guarantee to the schema a file stores
a column with and the filter to the schema it is read with, so that it no
longer looks like an integer is compared with a string, and a Parquet test
reads int32, float64 and string columns as another type.
Copilot AI balanced review requested due to automatic review settings November 16, 2026 23:12

This comment was marked as low quality.

…se nullable ranges where types agree

The issue asks for pruning with large is_in value sets and for recognizing
the guarantee of a row group which may contain nulls. The handling of casts
in the filter, of statistics of another type than the column is read as and
of value sets of another type than the column is taken out again: it fixed
bugs which exist independently of this change and will be addressed
separately. expression_internal.h is as it was.

So that those bugs do not spread to row groups with nulls, which were not
pruned at all so far, the nullable range guarantee and large value sets are
only used where no types need to be reconciled: for calls on the field
itself, with bounds and (small) value sets of the field's type. The
guarantees which were recognized before are used as before.
Copilot AI balanced review requested due to automatic review settings November 17, 2026 00:12

This comment was marked as low quality.

Groups of near-identical cases in SimplifyIsInLargeValueSet and
SimplifyWithNullableRangeGuarantee are reduced to one or two each, and the
Parquet tests drop cases which the expression tests cover and share one
helper for the row groups a scan reads. Removing any piece of the logic which
made a test fail before still does.
Copilot AI balanced review requested due to automatic review settings November 17, 2026 00:46

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

rok added 2 commits November 17, 2026 02:16
…gic added

Each remaining case fails on main or when a piece of the added logic is
removed. The cases removed passed either way: controls, cases which the
existing tests cover, and cases guarded by search_sorted itself.
…ange guarantee

Add cases to SimplifyFilterWithGuarantee for an is_in filter with 100000
values against the guarantee of a Parquet row group's statistics: a range
without and with a value of the set, and a range which may be null.
@rok
rok requested a review from zanmato1984 as a code owner November 17, 2026 01:16
Copilot AI balanced review requested due to automatic review settings November 17, 2026 01:16

This comment was marked as low quality.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants