Repository navigation
feat: add OpenTelemetry instrumentation - #672
kahirokunn wants to merge 2 commits into
Conversation
There was a problem hiding this comment.
So on reviewing this, to me it seems over engineered and ultimately longterm fragile due to the coupling of messaging, rpc & http conventions.
My suggestion is as follows:
- have the encode operation add the cloud event attribute values in baggage
- have the decode operation remove the cloud event baggage items
- develop either of the below which converts baggage to attributes
- cloud events log/span processor
- baggage to attribute log/span processor
With this approach we can have the following scenarios:
Integrated transport
- application calls send
- ce lib start emit process
- ce otel extension adds cloud event info to baggage
- transport lib send invoked
- transport instrumentation creates span
- processor reads baggage & adds it as attributes
- propagator attaches info to carrier
- transport sends request
- receiver transport receives request
- propagator attaches info from carrier
- processor reads baggage & adds it as attributes as well as removes it from baggage
- handling of event completes.
Encoder
- ce encoder library called for a payload
- ce otel extension adds cloud event info to baggage
- application has an encoded payload
- application invokes transport lib send
- transport instrumentation creates span
- processor reads baggage & adds it as attributes
- propagator attaches info to carrier
- transport sends request
- receiver transport receives request
- propagator attaches info from carrier
- application creates span
- processor reads baggage & adds it as attributes as well as removes it from baggage
- ce decoder library called for a payload
- application handles decoded payload
This way the solution works for any transport & the responsibility for defining the telemetry resides with the instrumentation libraries rather than cloud events.
|
I think CloudEvents instrumentation should represent the logical event lifecycle independently of the underlying transport. OpenTelemetry's CloudEvents Semantic Conventions recommend following the Messaging Span model, including span relationships and message creation contexts. I'd prefer to follow these conventions directly, keeping CloudEvents-level observability separate from transport-level instrumentation. |
This is not documented anywhere in semantic conventions and in fact it goes the opposite direction by not documenting the lifecycle ie no span but rather what attributes can be added to spans.
The conventions are for the instrumentation which in the case of sending http request that would http, it would make no sense to have a messaging publish span for http as you don't have a messaging system.
I am following the conventions, as per https://opentelemetry.io/docs/specs/semconv/cloudevents/cloudevents-spans/#span-attributes key points bei g
Which means the attributes are added to the protocol ie if you are using rabbitmq then you have your rabbitmq span and cloud events attributes added to it.
That is exactly what i am proposing, using the extension to add to context and pulling it out when needed. Key thing is, convention is not for a span but rather attributes added to applicable spans. |
|
The Go SDK already creates CloudEvents-level PRODUCER and CONSUMER spans alongside HTTP instrumentation. I’d like to follow the same separation in the JavaScript SDK rather than limit CloudEvents instrumentation to adding attributes to transport spans. |
|
The go sdk is actually quite different to both the current otel spec & this pr. Go is just creating a span with the cloud event attributes and nothing else so no connection to messaging etc. Note I can see the benefit to this approach on client side as it avoids the need to attach attributes after the payload has been encoded but on receiver side it fells un-necessary as what do you gain? Also on looking at the otel spec history it appears that Go is following a much older spec which was removed in open-telemetry/semantic-conventions#1182. In fact it was written by the same author of the go sdk, just 3 years later. |
|
Thanks for pointing out the spec history. On the receiver side, the value I’m aiming for is batch-aware processing: one processing span linked to each event’s available creation context, with batch size and per-event attributes. This follows a similar opt-in, library-level instrumentation pattern to Prisma for ORM operations and BullMQ for job processing. I’d like to provide this as reusable, opt-in instrumentation so applications don’t have to implement that logic themselves. Where existing instrumentation already covers this, the wrapper can simply be omitted. |
Should be already covered out of the box as per messaging spec https://opentelemetry.io/docs/specs/semconv/messaging/messaging-spans/#batch-receiving in particular https://opentelemetry.io/docs/specs/semconv/messaging/messaging-spans/#recording-per-message-attributes-on-batch-operations which describes how to capture the message level details. This can all be done without instrumentation & instead achieved at the propagation layer via a custom propagator.
The issue is not about providing the functionality but rather what is the responsibility of cloudevents to provide.
Existing instrumentation will only cover the standard messaging use case ie creating spans however a cloudevents propagator is needed to create the links based on the payload & processor to add attributes to the messaging send span. As such i feel that the extension should just be:
|
|
My concern is using Baggage solely to pass CloudEvents metadata to a local processor. With baggage propagation enabled, that metadata could also be sent to downstream services. Opting into instrumentation shouldn’t implicitly opt applications into propagating event metadata downstream. I’d prefer to attach the batch attributes and links directly when the opt-in wrapper creates the processing span. This keeps the integration simple, avoids introducing Baggage and an additional processor, and doesn’t depend on other instrumentation providing an appropriate processing span. |
Then have the processor remove before it sending. Alternatively we can add it to the context instead that way it is not propagated. I do like baggage so that reciever spans can have access to the properties without parsing the messages.
I don't follow how this differs to what i said, under my proposal when processing a message the propagator extracts the context from the message, once extracted links can be created with the desired attributes.
You are right it doesn't rely on other instrumentation creating the span but i see that as a major positive as that way cloud events doesn't need to duplicate implementation & avoids chance of spans being double instrumented. I don't see what is complex from an app developer side, the steps would be:
Nothing else needs to be done so it is almost as simple (1 additional line) as directly creating the sending span but significantly decreases the code base needing to be maintained by cloudevents. |
|
Thanks for clarifying. I’d like to support enriching existing spans while keeping The wrapper’s purpose is to capture the event or batch handler’s own duration and failures, with links to each event’s available creation context. That processing can be a distinct operation from the enclosing HTTP request. |
If a transport is not instrumented it would be much better to instrument that transport following otel in either a personal or otel-js-contrib repo. Note as per the spec the messaging span represents the logical operation of calling send on the transport layer.
That is what the old Otel spec said but it was collectively decided that it shouldn't be the case and instead the messaging/http conventions should be used to represent the messaging/http operation. If you are really interested in the encoding/decoding operation duration that could be expressed as dedicated internal child spans or events. My recommendation for this pr given it is not following the otel spec or cloud events as well as the fact that it is doing an extremely lot of different things, is to split into manageable/reviewable chunks. The way i would do it is dedicated pr's for:
Independent of the above if a transport you need is not instrumented instrument. Note the first 2 pr's are following the current spec. |
|
#1182 removed the separate CloudEvents span definitions, but its merged text recommends following the Messaging span structure. That recommendation remains in the current CloudEvents conventions. The Messaging conventions describe Process spans covering handler execution. They also explicitly show a Process span beneath an HTTP server span when a broker delivers messages over HTTP, linked to the message creation context. That Messaging Process span, enriched with CloudEvents attributes and links, is what I intend this opt-in helper to provide when existing instrumentation does not already cover the processing operation. Whether this SDK should maintain that helper, and how to split the PR, are separate questions. I don’t think #1182 establishes that this processing span structure should be avoided. |
Thats not a messaging processing span as the operation has already started and the description states it is to represent the callback/handler.
Yes because it is assumed there is already a span for the messaging system. Cloudevents spec even states the focus of ce is on preparing the content https://github.com/cloudevents/spec/blob/main/cloudevents/SDK.md#object-model-structure-guidelines and not wrapping the sdk.
A handler excludes decoding.
It does but key thing is the complete flow:
Key thing is you have a broker who initiates a http request to the server. This process span would be very difficult to accurately create as the same endpoint could be called from many brokers.
Whereas for me the first thing the extension needs to do is cover the cloud events use case which is how to propagate distributed tracing otel. This applies to all transports as even already instrumented transports will not support the ce distributed tracing extension, hence that needs to be implemented
It is not a question of if a distributed tracing helper should be maintained by ce but rather is it ce responsibility to instrument transports which are not instrumented. Note, the pr can't be merged without answering that question but If the pr was focussed on a single delivery it would be much more likely to progress and is not tied into that bigger discussion.
What it does specify is the narrow use case of when it should be used and tbh if you are using that architecture, it would be more beneficial to instrument the framework. |
837c614 to
d0d3575
Compare
Signed-off-by: kahirokunn <okinakahiro@gmail.com>
efdd469 to
a778b4a
Compare
Signed-off-by: kahirokunn <okinakahiro@gmail.com>
thompson-tomo
left a comment
There was a problem hiding this comment.
Thanks for greatly descoping this PR, it made it much easier to review.
| /** | ||
| * Copy a valid span context into a new CloudEvent, replacing its tracing extension. | ||
| * @param {Context} context the trace context to inject | ||
| * @param {CloudEvent} event the event to copy | ||
| * @returns {CloudEvent} an event copy for a valid span context, otherwise the original event | ||
| */ | ||
| export function injectDistributedTracingExtension<T>(context: Context, event: CloudEvent<T>): CloudEvent<T> { | ||
| const extension: Record<string, string> = {}; | ||
| propagator.inject(context, extension, defaultTextMapSetter); | ||
| if (!extension.traceparent) { | ||
| return event; | ||
| } | ||
|
|
||
| // cloneWith() merges properties, so remove the previous tracestate before copying the extension. | ||
| const properties: Partial<CloudEventV1<T>> = { ...event }; | ||
| delete properties.traceparent; | ||
| delete properties.tracestate; | ||
| Object.assign(properties, extension); | ||
| // Preserve events that were constructed without validation. | ||
| return new CloudEvent<T>(properties, false); | ||
| } | ||
|
|
||
| /** | ||
| * Extract the event's trace context while preserving other values from the supplied context. | ||
| * @param {Context} context the base context | ||
| * @param {CloudEventV1} event the event to read | ||
| * @returns {Context} the combined context, or the supplied context for a missing or invalid extension | ||
| */ | ||
| export function extractDistributedTracingExtension(context: Context, event: CloudEventV1<unknown>): Context { | ||
| return propagator.extract(context, event, getter); | ||
| } |
There was a problem hiding this comment.
Any reason not to define this as an actual otel propagator & then have ce helpers which call this propagator plus adds the links?
This way the client application can be simpler & you could have it automatically extract the ce context on Web hooks at http span creation time if w3 context is not available & headers are used.
Proposed Changes
Add explicit helpers for injecting and extracting the CloudEvents Distributed Tracing extension through
cloudevents/otel.Description
injectDistributedTracingExtension(context, event)copies a valid W3C trace context into a new event, leaving the original unchanged. Explicit injection replaces an existing tracing extension; if the supplied context has no valid span context, the original event is returned.extractDistributedTracingExtension(context, event)returns a context containing the event's valid tracing extension. A missing or invalid extension leaves the base context unchanged. The returned context is not activated automatically.The helpers do not create spans and can be used independently of transport instrumentation. OpenTelemetry dependencies are optional for applications using the package root. The README shows injection followed by sending the event with
await emit(tracedEvent).This PR is based directly on
mainand does not depend on #657. Existing-span enrichment, span-creating wrappers, metrics, and exception logs are deferred to separate follow-up work. Related to #355.Validation