Repository navigation
Conversation
When valkey is stopped ahead of cloud_controller_ng draining during an api-group rolling update, its unix socket is removed while CC-NG still serves traffic. Every request then hit RedisStore on the dead socket and raised Redis::CannotConnectError, which surfaced as a 500 with a leaked backtrace. Wrap the RedisStore metrics writes so connection errors are logged and swallowed — a metrics write must never fail a user request. Genuine errors (e.g. Redis::CommandError) still propagate.
cee89e1 to
79c5b80
Compare
philippthun
left a comment
There was a problem hiding this comment.
We should come up with a single pattern for all places where we interact with Valkey: base_rate_limiter.rb, concurrent_request_counter.rb and statsd_updater.rb. All three should rescue the same exceptions, write at the same log level and use similar wording (fail open vs. tolerate).
Both the commit message and the PR description mention a draining behavior that is wrong (i.e. caused by another bug and fixed with cloudfoundry/capi-release#698) and thus should be updated.
StatsdUpdater's RedisStore updates the outstanding-requests gauge in Valkey on every request. Any Redis error propagated out of the request-metrics middleware and failed the request with a 500. Wrap the RedisStore operations in a fail_open helper that logs and swallows Redis errors, matching the existing handling in base_rate_limiter.rb and concurrent_request_counter.rb: rescue Redis::BaseError, log at error with "Redis error: <class> - <message>".
philippthun
left a comment
There was a problem hiding this comment.
I recommend to squash this locally as the first commit is now obsolete - and people looking at the PR could get confused.
| yield | ||
| rescue Redis::BaseError => e | ||
| logger.error("Redis error: #{e.class} - #{e.message}") | ||
| nil |
There was a problem hiding this comment.
start_request and complete_request should skip sending the metric when nil is returned, otherwise an invalid gauge packet is sent (not sure what the receiver is doing with it). E.g.:
request_outstanding_gauge = store.increment_request_outstanding_gauge
@statsd.gauge('cc.requests.outstanding.gauge', request_outstanding_gauge) unless request_outstanding_gauge.nil?
StatsdUpdater's RedisStore reaches Valkey over a unix socket on every request to maintain the outstanding-requests gauge. If a Redis call failed, for example Valkey being briefly unavailable, the error propagated out of the request-metrics middleware and the request returned a 500.
This makes the metrics path fail open: a metrics write must never fail a user request. Redis errors are caught, logged, and the operation returns nil.
CC has three code paths that talk to Valkey, and they handled Redis errors three different ways. This aligns StatsdUpdater with the existing, already-merged handling in the other two so all three behave the same:
Make StatsdUpdater's RedisStore fail open on Valkey errors. The metrics store is reached over a unix socket on every request to update the outstanding-requests gauge; any Redis error used to propagate out of the request-metrics middleware and 500 the request. Now Redis errors are caught, logged, and the operation returns nil. This aligns StatsdUpdater's error handling with the existing Valkey callers base_rate_limiter.rb and concurrent_request_counter.rb: all three rescue Redis::BaseError and log at error with the shared "Redis error: - " wording. Each still fails open in the way that fits its own call site (StatsdUpdater returns nil; the rate limiter returns a non-limiting count; the concurrent counter rescues a StoreError one level up).
CC updates the StatsD/Valkey metrics gauge on every request. When Valkey is momentarily unavailable (e.g. briefly down or restarting), the Redis call raised and the request returned a 500 to the caller instead of completing, a metrics write should never fail a user request. Failing open keeps requests succeeding while Valkey is unreachable, and rescuing/logging the same way as the other two Valkey callers makes the error handling consistent across CC.
This change was developed with AI assistance; all code was reviewed and tested by me.
I have reviewed the contributing guide
I have viewed, signed, and submitted the Contributor License Agreement
I have made this pull request to the
mainbranchI have run all the unit tests using
bundle exec rakeI have run CF Acceptance Tests