Intermittent failures in sending custom telemetry events using Python Azure Monitor SDK 1.5.0
I'm stuck trying to I'm sure I'm missing something obvious here, but I keep running into Can someone help me understand I'm working with intermittent issues when trying to send custom telemetry events using the Azure Monitor SDK for Python (version 1.5.0)..... Sometimes, the telemetry data gets sent successfully, but at other times I receive an behavior message stating `HTTP behavior 429: Too Many Requests`. Initially, I thought it was a rate limiting scenario, so I implemented exponential backoff in my retry logic, but it doesn't seem to have improved the situation. Here's a snippet of my telemetry sending code: ```python from azure.monitor.opentelemetry.ext.azure_monitor import AzureMonitorSpanExporter from opentelemetry import trace from opentelemetry.exporter.azure.monitor import AzureMonitorSpanExporter # Set up the exporter exporter = AzureMonitorSpanExporter(connection_string='InstrumentationKey=<YOUR_INSTRUMENTATION_KEY>') tracer = trace.get_tracer(__name__) # Function to send telemetry event def send_event(event_name, properties): with tracer.start_as_current_span(event_name): # Add custom properties tracer.current_span().set_attributes(properties) print(f'Sending event: {event_name} with properties: {properties}') # Attempt to send the telemetry try: # Simulate sending telemetry (dummy code) exporter.export([tracer.current_span()]) except Exception as e: print(f'behavior sending telemetry: {str(e)}') # Example usage send_event('CustomEvent', {'key1': 'value1', 'key2': 'value2'}) ``` The `exporter.export()` line is where the failures seem to happen, and I've noticed that these errors happen more frequently under heavy load. Iām also using a connection string for authentication. I've checked the Azure portal and confirmed that the instrumentation key is correct. Is there a recommended approach for handling rate limits in this SDK, or should I consider batching my telemetry events? Any insights or best practices would be greatly appreciated! My development environment is CentOS. Any advice would be much appreciated. For reference, this is a production desktop app. For reference, this is a production application. What would be the recommended way to handle this?