Repository navigation
Conversation
|
|
… sets using a sorted value set Since apacheGH-46777 is_in predicates with more than 50 values are not simplified against row group statistics at all, because filtering the value set and rebinding the call for every guarantee was prohibitively expensive. Large value sets are now simplified with a binary search instead: - The is_in kernel init keeps a copy of the value set sorted in the order of the comparison kernels, with nulls removed. An already sorted value set is reused as is after a single O(N) check; an unsorted one is sorted lazily and thread-safely on the first simplification, so eager kernel calls never pay for sorting. Value sets containing NaN are not searchable, since statistics cannot bound NaN. - SimplifyWithGuarantee applies all inequality guarantees on a field to a large is_in call at once, narrowing an index range into the sorted value set with lower/upper bound searches. If the range is empty the call simplifies to false (subject to the null matching behavior), otherwise it is left unchanged. This is exact for any number of values, costs O(log N) per guarantee and never rebinds. Applying both bounds together is what allows pruning a row group inside the hull of the value set but holding none of its values, which no per-bound check can do. Value sets up to 50 values keep using the existing filter-and-rebind path.
…d of via sort_indices + take Sorting a copy of the values with std::sort is about 14x faster than the generic sort_indices + take path for a shuffled set of 10M int64 values (0.19s vs 2.6s). Nulls are dropped while copying and a set containing NaN is reported as not searchable before sorting, since NaN would break the strict weak ordering std::sort requires. Non-primitive types (binary, decimal) keep using the generic path.
…in is_in pruning Extract nullable min/max bounds from Parquet statistics for large value sets. Respect GetNullMatchingBehavior() in both simplification paths and when rebinding narrowed sets. Add expression and Parquet regressions for nullable ranges and conflicting deprecated skip_nulls options.
…and prune nullable ranges in every simplification path Inequality guarantees are now restated through the order preserving casts wrapping a filter's argument before they are applied: strict bounds are weakened to inclusive ones and the bound is put through the same casts, since e.g. f64 = 51.00000001 satisfies f64 > 51 but casts to 51f. This applies to plain comparisons and small is_in value sets as well as the large value set path, which previously was the only one handling it. The nullable range guarantee produced by Parquet statistics for row groups with nulls, (x >= min and x <= max) or is_null(x), is now extracted for all simplifications rather than only for large is_in value sets, and an is_valid(x) member of the guarantee rules out nulls for the inequalities on x.
a3f317c to
79750a7
Compare
…L and INCONCLUSIVE calls - Defer concatenating, checking and sorting the value set to the first GetSortedValueSet() call (std::call_once), so plain is_in kernel calls pay nothing extra at init. - Simplify an is_in call without possible matches to its exact result for every null matching behavior: invert(true_unless_null(arg)) for EMIT_NULL and INCONCLUSIVE, or a null literal for INCONCLUSIVE with a null in the value set, so these prune row groups too. - Factor out Comparison::IsOrderPreservingCast, derive IsSearchableValueSetType from the accessor trait, and drop unreachable NaN, type and comparison checks. - Test chunked value sets and the new null matching results.
…pruning and condense tests - Sort the value set with sort_indices + take and search it with search_sorted instead of hand-written visitors. Decimal and fixed size binary value sets are no longer searchable since search_sorted does not support them. - Make the large is_in simplification tests table-driven, merge the null matching tests, and share a helper which checks that simplified filters evaluate like the originals. - Merge the two Parquet row group pruning tests into one file whose row groups contain nulls.
…and max instead of sorting them Sorting a value set costs 1.5-3x as much as building the is_in lookup table, so for 10M shuffled ids the one-time sort outweighed the row groups it pruned and scans got slower than without pruning. Value sets longer than kIsInMaxSortedValueSet (2^17) are now only bounded by their smallest and largest value (min_max), which is cheap at any size and prunes as well when the values are clustered.
…ith a cast MSVC flags constructing a std::vector<int32_t> from int64_t iterators as a narrowing conversion (C4244), which fails the Windows build.
…ered alike and keep NaN in small is_in value sets A guarantee bound of another type was brought into the field's type whenever a safe cast round tripped. Strings and numbers round trip but are ordered differently, so the statistics of a string column read as int64 pruned row groups holding matches of a large is_in. Bounds are now only converted between types which are ordered alike. Value sets of up to 50 values containing NaN are no longer filtered against inequality guarantees: statistics cannot bound NaN, which is_in matches.
…ge and simplify against per-field ranges Expression simplification no longer reaches into the is_in kernel state. A kernel state may implement the new internal RangeEvaluator interface to tell what its call evaluates to when the argument lies in a range; is_in implements it with its lazily sorted value set and its null matching rules, and scalar_set_lookup_internal.h is removed. SimplifyWithGuarantee folds the guarantee once into the inequalities and nullability of each field and simplifies every call in a single pass. Guarantee bounds are converted by one rule (Inequality::Restate) for comparisons, small is_in value sets and range evaluation alike: a bound of another type must be ordered like the field's type and convert to it exactly.
…st the ones which matter Removing each piece of the simplification in turn showed that the NaN check on guarantee bounds in expression.cc duplicates the kernel's, and that decimal bounds never converted to an integer or floating point field's type; both are removed. Guards which no test covered now have one: NaN in a sorted value set and in a value set bounded by its smallest and largest value, a strict bound next to an inclusive one at the same value, date bounds on a timestamp field, and index_in never being simplified.
… instead of ignoring them A bound of another type than the field's (the statistics of a float64 column read as float32, say) was ignored unless it converted exactly, so comparisons against such statistics were no longer simplified. The bound is now cast safely and, unless it round trips, strict comparisons are weakened to inclusive ones: 0.2 rounds up to 0.2f, which a stored 0.2 is read as. Also apply clang-format.
There was a problem hiding this comment.
🟡 Changes recommended
The simplification compares uncast double entries against a rounded bound, potentially reducing a matching filter to false and incorrectly pruning rows.
1 open issue
What changed in this PR
Restores Parquet row-group pruning for large is_in filters without repeatedly filtering and rebinding value sets, while extending simplification to nullable statistics.
Changes:
- Adds kernel range evaluation with lazy sorted-set lookup and min/max bounds for very large sets.
- Combines per-field bounds and handles nullable guarantees, cast rounding, and null-matching options.
- Adds expression-equivalence and Parquet row-group pruning tests.
| File | Description |
|---|---|
cpp/src/arrow/dataset/file_parquet_test.cc |
Tests large-set and nullable row-group pruning. |
cpp/src/arrow/compute/range_evaluation_internal.h |
Defines the internal kernel range-evaluation interface. |
cpp/src/arrow/compute/kernels/scalar_set_lookup.cc |
Adds cached value-set range evaluation for is_in. |
cpp/src/arrow/compute/expression.cc |
Integrates per-field ranges and cast-aware simplification. |
cpp/src/arrow/compute/expression_test.cc |
Covers ranges, coercion, null behavior, and execution equivalence. |
cpp/src/arrow/compute/expression_internal.h |
Refines order-preserving cast detection to exclude wrapping casts. |
Review effort: Balanced
| std::string func_name = Comparison::GetName(cmp); | ||
| DCHECK_NE(func_name, "na"); | ||
| std::vector<Datum> args{options->value_set, guarantee.bound}; | ||
| std::vector<Datum> args{options->value_set, bound}; |
adamreeve
left a comment
There was a problem hiding this comment.
Thanks Rok. I'm still looking through this but have some comments/questions
| // Bounds which a safe cast rounds (the statistics of a float64 column read as | ||
| // float32, say) are used rounded, and then only bound inclusively: 0.2 rounds up |
There was a problem hiding this comment.
Is this something that could happen in practice?
| auto string_range = | ||
| and_(greater_equal(i64, literal("10")), less_equal(i64, literal("9"))); |
There was a problem hiding this comment.
What is going on here, is it really valid to compare an integer to a string?
This comment says "as are", but I'm confused about what this is referring to. The above test cases use rounding, but then these test cases all expect no change after simplification.
| // value set must not be filtered out. Leave such value sets alone, as the | ||
| // kernel does for large ones. | ||
| const auto value_set_type_id = options->value_set.type()->id(); | ||
| if (value_set_type_id == Type::FLOAT || value_set_type_id == Type::DOUBLE) { |
There was a problem hiding this comment.
I think this should include HALF_FLOAT too.
There was a problem hiding this comment.
Actually I think this isn't needed. HALF_FLOAT isn't in IsSearchableValueSetType, so this is probably fine.
| // This base class enables non-templated access to the value set type | ||
| struct SetLookupStateBase : public KernelState { | ||
| /// Value sets up to this length are sorted for RangeEvaluator::EvaluateOverRange. | ||
| /// Sorting costs more than building the lookup table of the kernel, so longer value |
There was a problem hiding this comment.
I don't follow the logic here. Why is the sorting time compared to a lookup table build? Avoiding row group reads is the cost that this is trying to avoid. It probably does make sense to fall-back to using the min/max of the set at some point, but I'd just like to better understand how the cutoff point is chosen.
…s type and test columns read as another type The is_in kernel matches the argument against the value set cast to the argument's type, but small value sets were filtered against a guarantee as they were given. A value which the cast rounds could then be dropped although it matches: is_in(cast(f64, float32), [51.00000001]) was simplified to false given f64 <= 51.00000001. The value set is now cast first, and left alone if it cannot be cast. This also lets a half float value set on a float32 field be simplified, which failed for want of a comparison kernel. IsNaNScalar in the set lookup kernel handles half floats. SimplifyWithBoundOfAnotherType binds the guarantee to the schema a file stores a column with and the filter to the schema it is read with, so that it no longer looks like an integer is compared with a string, and a Parquet test reads int32, float64 and string columns as another type.
…se nullable ranges where types agree The issue asks for pruning with large is_in value sets and for recognizing the guarantee of a row group which may contain nulls. The handling of casts in the filter, of statistics of another type than the column is read as and of value sets of another type than the column is taken out again: it fixed bugs which exist independently of this change and will be addressed separately. expression_internal.h is as it was. So that those bugs do not spread to row groups with nulls, which were not pruned at all so far, the nullable range guarantee and large value sets are only used where no types need to be reconciled: for calls on the field itself, with bounds and (small) value sets of the field's type. The guarantees which were recognized before are used as before.
Groups of near-identical cases in SimplifyIsInLargeValueSet and SimplifyWithNullableRangeGuarantee are reduced to one or two each, and the Parquet tests drop cases which the expression tests cover and share one helper for the row groups a scan reads. Removing any piece of the logic which made a test fail before still does.
…gic added Each remaining case fails on main or when a piece of the added logic is removed. The cases removed passed either way: controls, cases which the existing tests cover, and cases guarded by search_sorted itself.
…ange guarantee Add cases to SimplifyFilterWithGuarantee for an is_in filter with 100000 values against the guarantee of a Parquet row group's statistics: a range without and with a value of the set, and a range which may be null.

Rationale for this change
Since #46859,
is_infilters with more than 50 values are not checked against Parquet row group statistics, so they read every row group. Separately, row groups that may contain nulls are never pruned by any filter, because their statistics guarantee,(x >= min and x <= max) or is_null(x), is not recognized. See #55292.What changes are included in this PR?
is_in. Theis_inkernel state keeps its value set and sorts it the first time a filter is simplified against a guarantee, so plain kernel calls never sort. For each row group, two binary searches (search_sorted) tell whether any value lies between its min and max. Value sets longer than 2^17 are not sorted, only bounded by their smallest and largest value. Value sets of up to 50 values keep the existing path.SimplifyWithGuaranteeasks the kernel what the call evaluates to over a range through a small internal interface (RangeEvaluator), so the null matching rules ofis_instay in the kernel.is_inprune such row groups too.is_invalue sets: the deprecatedskip_nullsoption is honored, and sets containing NaN are left alone, since statistics cannot bound NaN.Are these changes tested?
Yes, with new expression tests and two Parquet dataset tests that check which row groups are read. The
SimplifyFilterWithGuaranteebenchmark gains cases for a 100,000-valueis_inagainst a row group guarantee: simplifying takes about 1.8 µs per row group, against about 0.3 µs on main, which skips such filters.Scan timings on a 49.3M row file with 47 row groups of 2^20 rows and a sorted int64
idcolumn (release build, macOS arm64, median of 5 scans, each binding a fresh filter):is_invalue setThe fourth row shows the limit for value sets longer than 2^17: the row groups in between are read too.
Are there any user-facing changes?
No API changes. Large
is_infilters, and filters on row groups with nulls, now skip row groups that cannot match. Not covered: decimal, fixed size binary and view value sets, dictionary columns, and columns read as another type than they are stored as.Was AI used for this PR?
PR code and description written by:
Reviewed before submission by:
The code was drafted with an AI assistant; I reviewed, tested and benchmarked it and iterated on the design.