Skip to content

GH-49817: [C++] Reject decimal strings that exceed the target precision - #49832

Open
SAY-5 wants to merge 4 commits into
apache:mainfrom
SAY-5:fix/decimal-fromstring-precision-overflow-49817
Open

SAY-5 wants to merge 4 commits into
apache:mainfrom
SAY-5:fix/decimal-fromstring-precision-overflow-49817

Conversation

@SAY-5

@SAY-5 SAY-5 commented Apr 21, 2026 •

Copy link
Copy Markdown

Rationale for this change

arrow::Decimal128::FromString and Decimal256::FromString (and the SimpleDecimalFromString path used by Decimal32/Decimal64) silently wrap around when the digits don't fit in the target bit width. The digit string is fed into ShiftAndAdd, which multiplies and adds into a fixed-size uint64_t array; the carry out of the top word is dropped, so callers get (value mod 2^kBitWidth) with Status::OK.

What changes are included in this PR?

ShiftAndAdd now reports a carry out of the top word. DecimalFromString returns Status::Invalid when that happens or when the magnitude doesn't fit the signed range (above 2^127 - 1 / 2^255 - 1, or exactly 2^127 / 2^255 for negative values, which are still accepted). SimpleDecimalFromString uses the same carry so a 20+ digit string can no longer wrap a uint64_t past the existing range check.

Values with more digits than kMaxPrecision that still fit in the bit width (e.g. 170141183460469231731687303715884105727, or the 39 digit decimal128 values in the 1.0.0 integration gold files) keep parsing as before.

Are these changes tested?

Yes, the previously commented out overflow cases in FromStringLimits are enabled for Decimal128 and Decimal256 (including the 2^127 / 2^255 boundaries and the reproducer from the issue), plus a Decimal32/64 wraparound test.

Are there any user-facing changes?

Decimal*::FromString now returns an error for strings whose value doesn't fit in the decimal's bit width, instead of returning a wrapped value.

Closes #49817.

Signed-off-by: SAY-5 SAY-5@users.noreply.github.com

@github-actions

Copy link
Copy Markdown

⚠️ GitHub issue #49817 has been automatically assigned in GitHub to PR creator.

@raulcd raulcd left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The reporting issue contains some tests and reproducers. Can we add tests?

@SAY-5 as per our guidelines, can you share whether the fix was AI generated and summarise what was AI-generated?
https://arrow.apache.org/docs/dev/developers/overview.html#ai-generated-code

Thanks!

@SAY-5

SAY-5 commented May 2, 2026 •

Copy link
Copy Markdown
Author

Re: AI policy, yes, this fix used an LLM coding assistant for drafting and the test cases below; I reviewed each line, ran the change against the issue's reproducer locally, and confirmed the behaviour matches SQLite/Spark semantics (precision-overflow rejected rather than silently truncated). I'll add the tests from #49817 in the next push and reply here.

For the moment, here's the disclosure summary as requested:

  • AI-generated: Decimal128::FromString precision-overflow guard and the surrounding documentation comment
  • Human-reviewed: scope (only the precision check, no related rounding changes), error message wording, alignment with the existing Status::Invalid style of nearby validation paths, and the test plan below

@SAY-5

SAY-5 commented May 10, 2026

Copy link
Copy Markdown
Author

Diagnosis: the new precision check in DecimalFromString rejects strings whose significant-digit count exceeds the target type's max precision. Existing arrow-array, arrow-json, and arrow-s3fs tests pass strings with 77 significant digits (e.g. for decimal256 = max 76) and were previously relying on the silent ShiftAndAdd overflow returning some value. The check fires correctly per the issue, but the test fixtures need updating to either:

  1. trim trailing precision so they fit kMaxPrecision exactly, or
  2. assert Status::Invalid instead of ValueOrDie.

Affected test files (from CI logs): arrow-array-test, arrow-json-test, arrow-s3fs-test. I'd prefer to leave that test surgery for a maintainer who knows which assertions reflect intended behaviour vs. accidental reliance on overflow.

@github-actions

Copy link
Copy Markdown

⚠️ GitHub issue #49817 has been automatically assigned in GitHub to PR creator.

@pitrou

pitrou commented May 12, 2026

Copy link
Copy Markdown
Member

Existing arrow-array, arrow-json, and arrow-s3fs tests pass strings with 77 significant digits (e.g. for decimal256 = max 76) and were previously relying on the silent ShiftAndAdd overflow returning some value.

You mean arrow-utility-test not arrow-s3fs-test, right?

In any case, I think these tests need to be fixed not to overflow the allowable precision.
However, one of the tests is simply testing the error message. We should try to make the error messages consistent there to minimize disruption.

@SAY-5

SAY-5 commented May 13, 2026

Copy link
Copy Markdown
Author

Yes, arrow-utility-test, sorry for the s3fs typo. I'll trim the test fixtures so they hit kMaxPrecision exactly and align the error-message test by matching whatever phrasing the new check emits, will push once it builds clean.

@SAY-5
SAY-5 force-pushed the fix/decimal-fromstring-precision-overflow-49817 branch from 838d46e to a23f17e Compare May 13, 2026 20:09
@SAY-5

SAY-5 commented May 13, 2026

Copy link
Copy Markdown
Author

Reworked: trimmed test fixtures in arrow-array-test, arrow-json-test, and arrow-utility-test to hit kMaxPrecision exactly (apologies for the s3fs typo earlier), and normalised the precision-overflow error to use the existing 'requires precision N' phrasing so the json converter test substring stays in sync.

Comment thread cpp/src/arrow/util/decimal_test.cc Outdated
TEST(Decimal128Test, PrintMaxValue) {
const std::string string_value("170141183460469231731687303715884105727");
// trimmed to kMaxPrecision (38) digits (GH-49817)
const std::string string_value("17014118346046923173168730371588410572");

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is not testing the max value anymore (which is 99999999999999999999999999999999999999). Should we test both values using a simple loop?

Comment thread cpp/src/arrow/util/decimal_test.cc Outdated
ASSERT_FALSE(TypeParam("1" + max_zeros).FitsInPrecision(TypeParam::kMaxPrecision));
ASSERT_FALSE(TypeParam("-1" + max_zeros).FitsInPrecision(TypeParam::kMaxPrecision));
// construct 10^kMaxPrecision without going through the string parser (GH-49817)
TypeParam one_past_max = TypeParam::GetMaxValue(TypeParam::kMaxPrecision) + 1;

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why not max_nines + 1?

Comment thread cpp/src/arrow/util/decimal_test.cc Outdated
check(123, 30);

ASSERT_EQ(Decimal32("1999999998"), Decimal32("999999999") << 1);
// results trimmed to kMaxPrecision (9) digits (GH-49817)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The comment is a bit confusing, it implies that it's the left shift operation that trims the digits.

Comment thread cpp/src/arrow/util/decimal_test.cc Outdated
ASSERT_EQ(Decimal64("1234567890123456"), Decimal64("1234567890123456") << 0);
ASSERT_EQ(Decimal64("2469135780246912"), Decimal64("1234567890123456") << 1);
ASSERT_EQ(Decimal64("6917529027641081856"), Decimal64("1234567890123456") << 55);
// shift 9 keeps the result within kMaxPrecision (18 digits) (GH-49817)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Similarly, not sure this comment is informative to the reader.

@SAY-5

SAY-5 commented Jun 11, 2026

Copy link
Copy Markdown
Author

Thanks for the review. Addressed all four in 0334982: PrintMaxValue/PrintMinValue now loop over both the all-nines max (99999999999999999999999999999999999999) and the truncated bit-pattern value; FitsInPrecision uses max_nines + 1 instead of GetMaxValue; and the LeftShift comments now say the operands/shift were chosen so the expected result fits in kMaxPrecision, since FromString rejects wider literals, rather than implying the shift trims.

@pitrou pitrou left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1, thanks for the update @SAY-5

SAY-5 and others added 3 commits June 16, 2026 11:20
…recision

Signed-off-by: Sai Asish Y <say.apm35@gmail.com>
Signed-off-by: Sai Asish Y <say.apm35@gmail.com>
@pitrou
pitrou force-pushed the fix/decimal-fromstring-precision-overflow-49817 branch from 0334982 to 7761dd6 Compare June 16, 2026 09:21
@pitrou

pitrou commented Jun 16, 2026

Copy link
Copy Markdown
Member

Pushed a rebase + lint fix.

@pitrou

pitrou commented Jun 16, 2026

Copy link
Copy Markdown
Member

Uh, it looks like this fails the integration CI tests:
https://github.com/apache/arrow/actions/runs/27607596960/job/81623528560?pr=49832#step:13:17036

  ################# FAILURES #################
  FAILED TEST: decimal C++ producing,  C++ consuming
  <class 'RuntimeError'>: Command failed: /build/cpp/debug/arrow-json-integration-test --integration --arrow=/arrow/testing/data/arrow-ipc-stream/integration/1.0.0-bigendian/generated_decimal.arrow_file --json=/tmp/tmpxf1pado_/1.0.0-bigendian_decimal.gold.json --mode=VALIDATE --validate_decimals=false --validate_date64=false --validate_times=false
  With output:
  --------------
  Error message: Invalid: 135374626707953463472845258829775501736 requires precision 39 which exceeds the decimal128 maximum of 38
  
  --------------
  
  
  FAILED TEST: decimal C++ producing,  C++ consuming
  <class 'RuntimeError'>: Command failed: /build/cpp/debug/arrow-json-integration-test --integration --arrow=/arrow/testing/data/arrow-ipc-stream/integration/1.0.0-littleendian/generated_decimal.arrow_file --json=/tmp/tmpxf1pado_/1.0.0-littleendian_decimal.gold.json --mode=VALIDATE --validate_decimals=false --validate_date64=false --validate_times=false
  With output:
  --------------
  Error message: Invalid: 135374626707953463472845258829775501736 requires precision 39 which exceeds the decimal128 maximum of 38
  
  --------------

This occurs in the "gold" integration files, which means we cannot change them (they were generated with Arrow 1.0.0). We'll have to find a way around this @SAY-5

@SAY-5

SAY-5 commented Aug 16, 2026

Copy link
Copy Markdown
Author

That gold value is 135374626707953463472845258829775501736, which is below 2^127, so it round trips fine as a 128 bit pattern. It just happens to be 39 digits long, so the digit count check rejects it even though nothing was lost.

Which makes me think the check is in the wrong place. What 49817 actually reports is silent wraparound, and the strings that wrap are the ones that exceed the bit width, not the ones that exceed kMaxPrecision. If I make ShiftAndAdd report a carry out of the top limb and reject on that (with the sign aware 2^127 vs 2^127-1 boundary), 170141183460469231731687303715884105727 keeps parsing like it always did, the gold files are untouched, and most of the test churn in this PR goes away. Want me to redo it that way?

@SAY-5

SAY-5 commented Aug 16, 2026

Copy link
Copy Markdown
Author

That gold value is 39 digits but only ~1.35e38, so it fits in 128 bits without wrapping. It's the digit count cap rejecting it, not a real overflow.

Which makes me think the check is in the wrong place: if I detect the actual carry out of ShiftAndAdd instead, the wrapping cases from #49817 still get rejected, values like this one still parse, and I can drop the test edits to PrintMaxValue, PrintMinValue and the Decimal256Test.WithNulls constants as well. Do you want me to redo it that way, or keep the precision cap and let the integration JSON reader parse unscaled values through a permissive path?

…dth instead of the max precision

Signed-off-by: Sai Asish Y <say.apm35@gmail.com>
@SAY-5

SAY-5 commented Nov 17, 2026

Copy link
Copy Markdown
Author

@pitrou reworked it so FromString only rejects actual wraparound: ShiftAndAdd now reports a carry out of the top word, and magnitudes past the signed range (above 2^127-1, or 2^127 for negatives, same idea for 256) are rejected. The 39 digit gold values fit in 128 bits so they parse as before, and the earlier test edits are reverted in favor of enabling the commented out overflow cases in FromStringLimits.

@github-actions

Copy link
Copy Markdown

⚠️ GitHub issue #49817 has been automatically assigned in GitHub to PR creator.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[C++] arrow::Decimal128::FromString silently truncates when the input string has more than 38 significant digits

3 participants