Repository navigation
Conversation
rzo1
left a comment
There was a problem hiding this comment.
Thx for the contrib. This fills a real gap, and the claims match the code (fail() → onMaxRetryReached → ack, the defaults, the constructor order). One wording fix inline. With that, +1.
| out, the spout fails it and schedules a retry, and the retry fails the same way. Meanwhile: | ||
|
|
||
| * the spout keeps polling past the bad record, but no later offset on that partition can be committed | ||
| * once `max.uncommitted.offsets` (10,000,000 by default) offsets are pending, the spout stops polling that partition |
There was a problem hiding this comment.
Small correction: once the limit is reached, the spout doesn't stop polling the partition entirely. While a retriable offset is within the limit, it keeps polling to re-emit the retry (see KafkaSpout#getPollablePartitionsInfo). What stops is consumption of new records. Maybe: "...the spout stops reading new records from that partition and only polls it to retry the failed tuple".
There was a problem hiding this comment.
Thanks, fixed in 8a161bc: it now says the spout stops reading new records from that partition and only polls it to retry the failed tuple.
| LOG.error("Giving up on record {}-{} at offset {}", msgId.topic(), msgId.partition(), msgId.offset()); | ||
| } | ||
|
|
||
| // open, onEmit, onAck, onPartitionsReassigned and onRetry can be left empty |
There was a problem hiding this comment.
nit: KafkaTupleListener has no default methods, so the snippet won't compile as-is. Maybe "the other methods (open, onEmit, ...) must be implemented but can be empty".
There was a problem hiding this comment.
Good point, fixed in 8a161bc: the comment now says the other methods must be implemented but can be empty.
Addresses the docs part of #9078. The design questions there (signalling the drop back to the spout, dead-lettering) are left alone.
What is the purpose of the change
docs/storm-kafka-client.mdsays nothing about what happens when a tuple from the Kafka spout is never acked, for example because it always fails to deserialize on the receiving worker. In at-least-once mode that quietly stops commits for the partition and lag keeps climbing, and with the default retry settings it never recovers.I added a short "Records that are never acked" section that explains:
max.uncommitted.offsets)DEFAULT_MAX_RETRIESisInteger.MAX_VALUE)KafkaSpoutRetryExponentialBackoffwith a finitemaxRetries, plusKafkaTupleListener.onMaxRetryReachedto log or keep track of the record before the spout acks itmaxRetriesalso gives up on tuples that fail for temporary reasonsDocs only; no code changes.
How was the change tested
I checked the description against
KafkaSpout.fail,KafkaSpoutConfig(defaults and builder methods) andKafkaTupleListeneron master, and that the example uses the actual constructor and method names. I also read the rendered Markdown.Happy to adjust the wording or move it somewhere else in the doc if you'd prefer.
An AI assistant helped draft this change; I reviewed it and take responsibility for it.