DevServer/docs/ISSUES_FIXED_AND_TROUBLESHOOTING.md

# 🔧 Issues Fixed & Troubleshooting Guide

**Date:** October 13, 2025  
**Status:** Partial Fix Applied ✅ / Traces Still Need Configuration ⚠️

---

## ✅ FIXED: Redis Batch-Set API Error

### Problem
```
POST /api/cache/batch-set
Error 500: Invalid input of type: 'dict'. Convert to a bytes, string, int or float first.
```

### Root Cause
The `batch-set` endpoint was trying to store Python dict objects directly in Redis, but Redis only accepts strings, bytes, ints, or floats.

### Solution Applied
**File:** `app/main.py` (lines 599-617)

**Changed:**
```python
for key, value in items.items():
    pipeline.set(key, value)  # ❌ Error: can't store dict
```

**To:**
```python
for key, value in items.items():
    # Convert value to JSON string if it's a dict, list, or other complex type
    if isinstance(value, (dict, list)):
        value_str = json.dumps(value)
    else:
        value_str = str(value)
    pipeline.set(key, value_str)  # ✅ Fixed: stores as JSON string
```

### Testing
```powershell
# Test the fixed endpoint
$body = '{"key1":"value1","key2":{"nested":"object"},"key3":["array","values"]}'
Invoke-WebRequest -Uri "http://localhost:8000/api/cache/batch-set" `
  -Method POST -Body $body -ContentType "application/json"
```

**Expected:** Status 200, returns success count

---

## ⚠️ ISSUE: Jaeger Shows No Traces

### Problem
- Jaeger UI shows only `jaeger-all-in-one` service
- No traces from `devserver.api.fastapi` or `devserver.celery.worker`
- Logs show `TraceID: N/A` for all requests

### Root Cause Analysis

#### 1. OpenTelemetry Auto-Instrumentation Not Initializing
**Evidence:**
```
2025-10-13 03:56:20 | DEBUG | devserver.api.fastapi | log_requests :61 | TraceID: N/A
```

The `TraceID: N/A` means no active span context exists.

**Why:**
- `opentelemetry-instrument` command is used in docker-compose
- Environment variables are set (`OTEL_EXPORTER_OTLP_ENDPOINT`, etc.)
- BUT: The tracer provider might not be initializing before the app starts

#### 2. Jaeger Configuration
**Jaeger logs show:**
```json
{"level":"info","msg":"[core] [Channel #1]parsed dial target is: \"localhost:4317\""}
```

Jaeger is trying to connect to `localhost:4317` instead of receiving traces. This is expected behavior - Jaeger has OTLP enabled and is ready to receive.

#### 3. OTEL Collector Pipeline
**Configuration looks correct:**
```yaml
pipelines:
  traces:
    receivers: [otlp]
    processors: [memory_limiter, batch]
    exporters: [debug, otlphttp/jaeger]
```

But no trace activity in logs means traces aren't reaching the collector.

---

## 🔍 Diagnosis Steps Performed

### 1. Checked OTEL Packages ✅
```toml
"opentelemetry-distro>=0.58b0"
"opentelemetry-exporter-otlp-proto-grpc>=1.37.0"
"opentelemetry-instrumentation-fastapi>=0.58b0"
"opentelemetry-instrumentation-redis>=0.58b0"
"opentelemetry-instrumentation-celery>=0.58b0"
```
All packages are installed.

### 2. Checked Environment Variables ✅
```yaml
OTEL_TRACES_EXPORTER: otlp
OTEL_METRICS_EXPORTER: otlp
OTEL_LOGS_EXPORTER: otlp
OTEL_EXPORTER_OTLP_ENDPOINT: http://otel-collector:4317
OTEL_EXPORTER_OTLP_PROTOCOL: grpc
OTEL_SERVICE_NAME: devserver.api.fastapi
```
All variables are set correctly.

### 3. Checked Container Startup Command ✅
```yaml
command: [
  "uv", "run", 
  "opentelemetry-instrument", 
  "--service_name", "devserver.api.fastapi",
  "uvicorn", "app.main:app", 
  "--host", "0.0.0.0", "--port", "8000", "--workers", "2"
]
```
Command structure is correct.

### 4. Checked Network Connectivity ✅
- All containers on `devserver_network`
- OTEL collector accessible at `otel-collector:4317`
- Jaeger accessible at `jaeger:14268`

---

## 🛠️ Solutions to Apply

### Solution 1: Explicit TracerProvider Initialization ⭐ RECOMMENDED

Create an OpenTelemetry initialization module to explicitly configure tracing before app startup.

**File:** `app/opentelemetry_config.py` (NEW FILE)

```python
"""OpenTelemetry configuration for DevServer"""
import os
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.resources import Resource, SERVICE_NAME
from opentelemetry.instrumentation.fastapi import FastAPIInstrumentor
from opentelemetry.instrumentation.redis import RedisInstrumentor
import logging

logger = logging.getLogger(__name__)

def init_opentelemetry():
    """Initialize OpenTelemetry with OTLP exporter"""
    
    # Get configuration from environment
    service_name = os.getenv("OTEL_SERVICE_NAME", "devserver.api.fastapi")
    otlp_endpoint = os.getenv("OTEL_EXPORTER_OTLP_ENDPOINT", "http://otel-collector:4317")
    
    logger.info(f"Initializing OpenTelemetry for {service_name}")
    logger.info(f"OTLP Endpoint: {otlp_endpoint}")
    
    # Create resource with service name
    resource = Resource(attributes={
        SERVICE_NAME: service_name
    })
    
    # Create tracer provider
    provider = TracerProvider(resource=resource)
    
    # Create OTLP exporter
    otlp_exporter = OTLPSpanExporter(
        endpoint=otlp_endpoint,
        insecure=True  # Use insecure channel (no TLS)
    )
    
    # Add span processor
    span_processor = BatchSpanProcessor(otlp_exporter)
    provider.add_span_processor(span_processor)
    
    # Set as global tracer provider
    trace.set_tracer_provider(provider)
    
    logger.info("✓ OpenTelemetry initialized successfully")
    
    return provider

def instrument_app(app):
    """Instrument FastAPI app with OpenTelemetry"""
    
    # Initialize OpenTelemetry
    init_opentelemetry()
    
    # Instrument FastAPI
    FastAPIInstrumentor().instrument_app(app)
    
    # Instrument Redis
    RedisInstrumentor().instrument()
    
    logger.info("✓ FastAPI and Redis instrumented")
```

**Update `app/main.py`:**
```python
# Add at the top after imports
from app.opentelemetry_config import instrument_app

# After creating the FastAPI app
app = FastAPI(...)

# Instrument with OpenTelemetry
instrument_app(app)
```

### Solution 2: Use Bootstrap Script (Alternative)

**File:** `app/bootstrap.py` (NEW FILE)
```python
#!/usr/bin/env python
"""Bootstrap script to initialize OpenTelemetry before app starts"""
import os
os.environ.setdefault("OTEL_PYTHON_LOG_LEVEL", "debug")

from opentelemetry.instrumentation.auto_instrumentation import sitecustomize

# This will run before anything else
sitecustomize.initialize()

print("✓ OpenTelemetry bootstrap complete")
```

**Update docker-compose.yml:**
```yaml
command: [
  "uv", "run", 
  "python", "-m", "app.bootstrap",  # Run bootstrap first
  "opentelemetry-instrument",
  "--service_name", "devserver.api.fastapi",
  "uvicorn", "app.main:app",
  "--host", "0.0.0.0", "--port", "8000"
]
```

### Solution 3: Debug Mode (For Troubleshooting)

Add debug logging to see what's happening:

```yaml
environment:
  OTEL_TRACES_EXPORTER: otlp
  OTEL_EXPORTER_OTLP_ENDPOINT: http://otel-collector:4317
  OTEL_PYTHON_LOG_LEVEL: debug  # Add this
  OTEL_LOG_LEVEL: debug  # And this
```

---

## 📋 Step-by-Step Fix Implementation

### Step 1: Create OpenTelemetry Config File
```powershell
cd d:\GNPL-Private\DevServer
# Create the file with the code above
```

### Step 2: Update main.py
Add the import and instrumentation call

### Step 3: Rebuild Containers
```powershell
docker-compose build --no-cache devserver celery_worker
```

### Step 4: Restart Services
```powershell
docker-compose up -d --force-recreate devserver celery_worker
```

### Step 5: Generate Test Traffic
```powershell
# Make some API calls
for ($i=1; $i -le 5; $i++) {
    Invoke-WebRequest -Uri "http://localhost:8000/api/dice/roll?session_id=test-$i"
    Start-Sleep -Milliseconds 500
}
```

### Step 6: Check Jaeger
```powershell
# Wait 10 seconds for traces to propagate
Start-Sleep -Seconds 10

# Check services
$response = Invoke-WebRequest -Uri "http://localhost:16686/api/services"
($response.Content | ConvertFrom-Json).data
```

**Expected:** Should see `devserver.api.fastapi` in the list

---

## 🧪 Verification Commands

### Check if Traces are Being Generated
```powershell
# Check FastAPI logs for trace IDs
docker logs devserver_backend --tail=20 | Select-String "TraceID"
```
**Expected:** Should see actual hex trace IDs instead of "N/A"

### Check OTEL Collector is Receiving Traces
```powershell
docker logs devserver_otel_collector --tail=50 | Select-String "span"
```
**Expected:** Should see span activity

### Check Jaeger is Receiving from OTEL
```powershell
docker logs devserver_jaeger --tail=50 | Select-String "span"
```
**Expected:** Should see span ingestion logs

### Query Jaeger API
```powershell
# Get all services
Invoke-WebRequest -Uri "http://localhost:16686/api/services"

# Get traces for a specific service
Invoke-WebRequest -Uri "http://localhost:16686/api/traces?service=devserver.api.fastapi&limit=10"
```

---

## 🎯 Expected Behavior After Fix

### 1. Logs Should Show Real Trace IDs
```
2025-10-13 XX:XX:XX | DEBUG | devserver.api.fastapi | log_requests :61 | 
    TraceID: 4bf92f3577b34da6a3ce929d0e0e4736  ✅ (not "N/A")
```

### 2. OTEL Collector Should Show Activity
```
2025-10-13 XX:XX:XX | INFO | Span received from devserver.api.fastapi
```

### 3. Jaeger Services List
```json
{
  "data": [
    "devserver.api.fastapi",  ✅
    "devserver.celery.worker",  ✅
    "jaeger-all-in-one"
  ]
}
```

### 4. Jaeger UI Should Show Traces
- Open: http://localhost:16686
- Select Service: `devserver.api.fastapi`
- See: List of traces with operations like "api.dice.roll", "api.tasks.fibonacci.create"

---

## 📝 Current Status Summary

| Component | Status | Notes |
|-----------|--------|-------|
| **Batch-Set API** | ✅ FIXED | Now converts dicts/lists to JSON strings |
| **Containers** | ✅ Running | All 11 containers up |
| **OTEL Collector** | ✅ Configured | Pipeline ready to receive traces |
| **Jaeger** | ✅ Ready | UI accessible, OTLP enabled |
| **Trace Generation** | ❌ NOT WORKING | TraceID shows "N/A" |
| **Auto-Instrumentation** | ❌ NOT WORKING | Need explicit initialization |

---

## 🚀 Next Actions

1. **Implement Solution 1** (Recommended):
   - Create `opentelemetry_config.py`
   - Update `main.py` to call `instrument_app(app)`
   - Rebuild and restart

2. **Or Use Simpler Approach**:
   - Remove `opentelemetry-instrument` wrapper
   - Use explicit instrumentation in code
   - More control and easier debugging

3. **Test Thoroughly**:
   - Generate API traffic
   - Check logs for real trace IDs
   - Verify Jaeger shows traces

---

**Last Updated:** October 13, 2025  
**Status:** Redis fix applied ✅ | Tracing fix documented, needs implementation ⏳