basic integratation of open telemetry

[mashraf-222]

Add OpenTelemetry Integration

This PR adds comprehensive OpenTelemetry support for distributed tracing across the codebase. The implementation uses automatic instrumentation for both third-party libraries and custom application functions, requiring zero code changes.

What's Added

• Telemetry module (src/telemetry/) with configuration, setup, and auto-instrumentation
• Automatic instrumentation for NumPy, Pandas, and other supported libraries
• Custom function auto-instrumentation that automatically traces all functions in specified modules/packages
• Environment configuration via environment variables with sensible defaults

How It Works

The implementation uses OpenTelemetry's standard auto-instrumentation pattern, which is the same approach used by production applications in large open-source projects.

Two Ways to Use It

1. Using opentelemetry-instrument (Recommended)

The easiest way is to use the opentelemetry-instrument command, which automatically instruments all supported libraries:

# Console output (for development)
OTEL_TRACES_EXPORTER=console opentelemetry-instrument python examples/run_all_traces.py

# Send to Jaeger (for visualization)
OTEL_TRACES_EXPORTER=otlp OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317 opentelemetry-instrument python examples/run_all_traces.py

This automatically instruments all supported libraries (NumPy, Pandas, Requests, SQLAlchemy, etc.) plus your custom functions.

2. Using Programmatic Setup

Alternatively, initialize telemetry in your code:

from src.telemetry import setup_telemetry, auto_instrument_package

# Initialize OpenTelemetry
setup_telemetry(
    service_name="optimize-me",
    service_version="0.1.0",
    exporter_type="console",  # or "otlp" for Jaeger
)

# Auto-instrument all custom functions
auto_instrument_package('src', exclude_modules=['src.tests', 'src.telemetry'])

Custom Function Instrumentation

All custom functions are automatically traced using auto_instrument_package(), which wraps functions in specified modules without requiring decorators. This includes:

• numerical/optimization.py - gradient descent
• algorithms/dynamic_programming.py - fibonacci, matrix operations, knapsack
• algorithms/graph.py - graph traversal, clustering, path finding
• data_processing/dataframe.py - filtering, grouping, merging
• statistics/descriptive.py - statistical operations

Testing with Jaeger

For local testing and visualization, use Jaeger (no API keys required):

1. Start Jaeger:

   docker-compose -f src/telemetry/docker-compose.yml up -d jaeger

2. Run your script with OTLP exporter:

   OTEL_TRACES_EXPORTER=otlp \
   OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317 \
   opentelemetry-instrument python examples/run_all_traces.py

3. View traces in Jaeger UI:
   Open http://localhost:16686 in your browser to see the trace visualization.

Configuration

Configuration is managed through environment variables with sensible defaults:

• OTEL_SERVICE_NAME=optimize-me - Service name for traces
• OTEL_TRACES_EXPORTER=console - Exporter type (console, otlp)
• OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317 - OTLP endpoint (for Jaeger)

Documentation

For detailed information, see:

• README.md - Complete setup guide with examples
• INSTRUMENTATION_COMPARISON.md - Detailed comparison of opentelemetry-instrument vs programmatic setup

Example Usage

Run the example script to see telemetry in action:

# Console output
OTEL_TRACES_EXPORTER=console opentelemetry-instrument python examples/run_all_traces.py

# Or with Jaeger
OTEL_TRACES_EXPORTER=otlp OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317 opentelemetry-instrument python examples/run_all_traces.py

This will output trace spans showing execution of all instrumented functions including gradient descent, graph traversal, statistical operations, and more.

[Saga4]

This Repo is hardly used in any real production environment. @KRRT7 We are looking to get real logs and signals to generate live production style test use cases for users and later know the benchmarks performances over a period of time.
Here we do not even have E2E tests as well.

[KRRT7]

@Saga4 we need to know what it'd look like as a basic POC before being able to implement it in a real codebase environment