With tracing at 100% (#9154), the OpenTelemetry SDK's batch span processor can't keep up on a fast topology: in my runs it exported 5–11% of the spans and dropped the rest, and tuning it raised that only to 13%. It drops spans whatever the outcome, so failures and timeouts go too.
@GGraziadei suggested sending the spans down a Storm stream to a collector bolt that keeps failures, timeouts and a sample of the rest. The bolt would also need to know when the work for a tuple ends in an async bolt; otherwise it can't tell where a timed-out tuple stopped.
Open questions: what the bolt holds and for how long, what it does under overload, and whether reducing near each worker beats one central bolt. I'd prototype it first and measure it against OTLP.
What do you think?
With tracing at 100% (#9154), the OpenTelemetry SDK's batch span processor can't keep up on a fast topology: in my runs it exported 5–11% of the spans and dropped the rest, and tuning it raised that only to 13%. It drops spans whatever the outcome, so failures and timeouts go too.
@GGraziadei suggested sending the spans down a Storm stream to a collector bolt that keeps failures, timeouts and a sample of the rest. The bolt would also need to know when the work for a tuple ends in an async bolt; otherwise it can't tell where a timed-out tuple stopped.
Open questions: what the bolt holds and for how long, what it does under overload, and whether reducing near each worker beats one central bolt. I'd prototype it first and measure it against OTLP.
What do you think?