Skip to content

docs: explain how a record that is never acked blocks Kafka spout commits - #9280

Open
linzch3 wants to merge 2 commits into
apache:masterfrom
linzch3:docs-kafka-undecodable-record
Open

linzch3 wants to merge 2 commits into
apache:masterfrom
linzch3:docs-kafka-undecodable-record

Conversation

@linzch3

@linzch3 linzch3 commented Nov 17, 2026

Copy link
Copy Markdown
Contributor

Addresses the docs part of #9078. The design questions there (signalling the drop back to the spout, dead-lettering) are left alone.

What is the purpose of the change

docs/storm-kafka-client.md says nothing about what happens when a tuple from the Kafka spout is never acked, for example because it always fails to deserialize on the receiving worker. In at-least-once mode that quietly stops commits for the partition and lag keeps climbing, and with the default retry settings it never recovers.

I added a short "Records that are never acked" section that explains:

  • why commits stop and when polling of the partition stops (max.uncommitted.offsets)
  • that the default retry service retries forever (DEFAULT_MAX_RETRIES is Integer.MAX_VALUE)
  • the current mitigation: a KafkaSpoutRetryExponentialBackoff with a finite maxRetries, plus KafkaTupleListener.onMaxRetryReached to log or keep track of the record before the spout acks it
  • the trade-offs: each retry waits for the message timeout, and a finite maxRetries also gives up on tuples that fail for temporary reasons

Docs only; no code changes.

How was the change tested

I checked the description against KafkaSpout.fail, KafkaSpoutConfig (defaults and builder methods) and KafkaTupleListener on master, and that the example uses the actual constructor and method names. I also read the rendered Markdown.

Happy to adjust the wording or move it somewhere else in the doc if you'd prefer.

An AI assistant helped draft this change; I reviewed it and take responsibility for it.

@rzo1 rzo1 added documentation Improvements or additions to documentation skip-changelog labels Nov 18, 2026
@rzo1 rzo1 added this to the 3.3.0 milestone Nov 18, 2026

@rzo1 rzo1 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thx for the contrib. This fills a real gap, and the claims match the code (fail() → onMaxRetryReached → ack, the defaults, the constructor order). One wording fix inline. With that, +1.

Comment thread docs/storm-kafka-client.md Outdated
out, the spout fails it and schedules a retry, and the retry fails the same way. Meanwhile:

* the spout keeps polling past the bad record, but no later offset on that partition can be committed
* once `max.uncommitted.offsets` (10,000,000 by default) offsets are pending, the spout stops polling that partition

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Small correction: once the limit is reached, the spout doesn't stop polling the partition entirely. While a retriable offset is within the limit, it keeps polling to re-emit the retry (see KafkaSpout#getPollablePartitionsInfo). What stops is consumption of new records. Maybe: "...the spout stops reading new records from that partition and only polls it to retry the failed tuple".

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, fixed in 8a161bc: it now says the spout stops reading new records from that partition and only polls it to retry the failed tuple.

Comment thread docs/storm-kafka-client.md Outdated
LOG.error("Giving up on record {}-{} at offset {}", msgId.topic(), msgId.partition(), msgId.offset());
}

// open, onEmit, onAck, onPartitionsReassigned and onRetry can be left empty

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: KafkaTupleListener has no default methods, so the snippet won't compile as-is. Maybe "the other methods (open, onEmit, ...) must be implemented but can be empty".

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point, fixed in 8a161bc: the comment now says the other methods must be implemented but can be empty.

@rzo1
rzo1 requested review from GGraziadei and reiabreu November 18, 2026 08:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation skip-changelog

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants