DevServer/docs/ISSUES_FIXED_AND_TROUBLESHOOTING.md

🔧 Issues Fixed & Troubleshooting Guide

Date: October 13, 2025
Status: Partial Fix Applied ✅ / Traces Still Need Configuration ⚠️

---

✅ FIXED: Redis Batch-Set API Error

Problem

POST /api/cache/batch-set
Error 500: Invalid input of type: 'dict'. Convert to a bytes, string, int or float first.

Root Cause

The batch-set endpoint was trying to store Python dict objects directly in Redis, but Redis only accepts strings, bytes, ints, or floats.

Solution Applied

File: app/main.py (lines 599-617)

Changed:

for key, value in items.items():
    pipeline.set(key, value)  # ❌ Error: can't store dict

To:

for key, value in items.items():
    # Convert value to JSON string if it's a dict, list, or other complex type
    if isinstance(value, (dict, list)):
        value_str = json.dumps(value)
    else:
        value_str = str(value)
    pipeline.set(key, value_str)  # ✅ Fixed: stores as JSON string

Testing

# Test the fixed endpoint
$body = '{"key1":"value1","key2":{"nested":"object"},"key3":["array","values"]}'
Invoke-WebRequest -Uri "http://localhost:8000/api/cache/batch-set" `
  -Method POST -Body $body -ContentType "application/json"

Expected: Status 200, returns success count

---

⚠️ ISSUE: Jaeger Shows No Traces

Problem

• Jaeger UI shows only jaeger-all-in-one service
• No traces from devserver.api.fastapi or devserver.celery.worker
• Logs show TraceID: N/A for all requests

Root Cause Analysis

1. OpenTelemetry Auto-Instrumentation Not Initializing

Evidence:

2025-10-13 03:56:20 | DEBUG | devserver.api.fastapi | log_requests :61 | TraceID: N/A

The TraceID: N/A means no active span context exists.

Why:

• opentelemetry-instrument command is used in docker-compose
• Environment variables are set (OTEL_EXPORTER_OTLP_ENDPOINT, etc.)
• BUT: The tracer provider might not be initializing before the app starts

2. Jaeger Configuration

Jaeger logs show:

{"level":"info","msg":"[core] [Channel #1]parsed dial target is: \"localhost:4317\""}

Jaeger is trying to connect to localhost:4317 instead of receiving traces. This is expected behavior - Jaeger has OTLP enabled and is ready to receive.

3. OTEL Collector Pipeline

Configuration looks correct:

pipelines:
  traces:
    receivers: [otlp]
    processors: [memory_limiter, batch]
    exporters: [debug, otlphttp/jaeger]

But no trace activity in logs means traces aren't reaching the collector.

---

🔍 Diagnosis Steps Performed

1. Checked OTEL Packages ✅

"opentelemetry-distro>=0.58b0"
"opentelemetry-exporter-otlp-proto-grpc>=1.37.0"
"opentelemetry-instrumentation-fastapi>=0.58b0"
"opentelemetry-instrumentation-redis>=0.58b0"
"opentelemetry-instrumentation-celery>=0.58b0"

All packages are installed.

2. Checked Environment Variables ✅

OTEL_TRACES_EXPORTER: otlp
OTEL_METRICS_EXPORTER: otlp
OTEL_LOGS_EXPORTER: otlp
OTEL_EXPORTER_OTLP_ENDPOINT: http://otel-collector:4317
OTEL_EXPORTER_OTLP_PROTOCOL: grpc
OTEL_SERVICE_NAME: devserver.api.fastapi

All variables are set correctly.

3. Checked Container Startup Command ✅

command: [
  "uv", "run", 
  "opentelemetry-instrument", 
  "--service_name", "devserver.api.fastapi",
  "uvicorn", "app.main:app", 
  "--host", "0.0.0.0", "--port", "8000", "--workers", "2"
]

Command structure is correct.

4. Checked Network Connectivity ✅

• All containers on devserver_network
• OTEL collector accessible at otel-collector:4317
• Jaeger accessible at jaeger:14268

---

🛠️ Solutions to Apply

Solution 1: Explicit TracerProvider Initialization ⭐ RECOMMENDED

Create an OpenTelemetry initialization module to explicitly configure tracing before app startup.

File: app/opentelemetry_config.py (NEW FILE)

"""OpenTelemetry configuration for DevServer"""
import os
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.resources import Resource, SERVICE_NAME
from opentelemetry.instrumentation.fastapi import FastAPIInstrumentor
from opentelemetry.instrumentation.redis import RedisInstrumentor
import logging

logger = logging.getLogger(__name__)

def init_opentelemetry():
    """Initialize OpenTelemetry with OTLP exporter"""
    
    # Get configuration from environment
    service_name = os.getenv("OTEL_SERVICE_NAME", "devserver.api.fastapi")
    otlp_endpoint = os.getenv("OTEL_EXPORTER_OTLP_ENDPOINT", "http://otel-collector:4317")
    
    logger.info(f"Initializing OpenTelemetry for {service_name}")
    logger.info(f"OTLP Endpoint: {otlp_endpoint}")
    
    # Create resource with service name
    resource = Resource(attributes={
        SERVICE_NAME: service_name
    })
    
    # Create tracer provider
    provider = TracerProvider(resource=resource)
    
    # Create OTLP exporter
    otlp_exporter = OTLPSpanExporter(
        endpoint=otlp_endpoint,
        insecure=True  # Use insecure channel (no TLS)
    )
    
    # Add span processor
    span_processor = BatchSpanProcessor(otlp_exporter)
    provider.add_span_processor(span_processor)
    
    # Set as global tracer provider
    trace.set_tracer_provider(provider)
    
    logger.info("✓ OpenTelemetry initialized successfully")
    
    return provider

def instrument_app(app):
    """Instrument FastAPI app with OpenTelemetry"""
    
    # Initialize OpenTelemetry
    init_opentelemetry()
    
    # Instrument FastAPI
    FastAPIInstrumentor().instrument_app(app)
    
    # Instrument Redis
    RedisInstrumentor().instrument()
    
    logger.info("✓ FastAPI and Redis instrumented")

Update app/main.py:

# Add at the top after imports
from app.opentelemetry_config import instrument_app

# After creating the FastAPI app
app = FastAPI(...)

# Instrument with OpenTelemetry
instrument_app(app)

Solution 2: Use Bootstrap Script (Alternative)

File: app/bootstrap.py (NEW FILE)

#!/usr/bin/env python
"""Bootstrap script to initialize OpenTelemetry before app starts"""
import os
os.environ.setdefault("OTEL_PYTHON_LOG_LEVEL", "debug")

from opentelemetry.instrumentation.auto_instrumentation import sitecustomize

# This will run before anything else
sitecustomize.initialize()

print("✓ OpenTelemetry bootstrap complete")

Update docker-compose.yml:

command: [
  "uv", "run", 
  "python", "-m", "app.bootstrap",  # Run bootstrap first
  "opentelemetry-instrument",
  "--service_name", "devserver.api.fastapi",
  "uvicorn", "app.main:app",
  "--host", "0.0.0.0", "--port", "8000"
]

Solution 3: Debug Mode (For Troubleshooting)

Add debug logging to see what's happening:

environment:
  OTEL_TRACES_EXPORTER: otlp
  OTEL_EXPORTER_OTLP_ENDPOINT: http://otel-collector:4317
  OTEL_PYTHON_LOG_LEVEL: debug  # Add this
  OTEL_LOG_LEVEL: debug  # And this

---

📋 Step-by-Step Fix Implementation

Step 1: Create OpenTelemetry Config File

cd d:\GNPL-Private\DevServer
# Create the file with the code above

Step 2: Update main.py

Add the import and instrumentation call

Step 3: Rebuild Containers

docker-compose build --no-cache devserver celery_worker

Step 4: Restart Services

docker-compose up -d --force-recreate devserver celery_worker

Step 5: Generate Test Traffic

# Make some API calls
for ($i=1; $i -le 5; $i++) {
    Invoke-WebRequest -Uri "http://localhost:8000/api/dice/roll?session_id=test-$i"
    Start-Sleep -Milliseconds 500
}

Step 6: Check Jaeger

# Wait 10 seconds for traces to propagate
Start-Sleep -Seconds 10

# Check services
$response = Invoke-WebRequest -Uri "http://localhost:16686/api/services"
($response.Content | ConvertFrom-Json).data

Expected: Should see devserver.api.fastapi in the list

---

🧪 Verification Commands

Check if Traces are Being Generated

# Check FastAPI logs for trace IDs
docker logs devserver_backend --tail=20 | Select-String "TraceID"

Expected: Should see actual hex trace IDs instead of "N/A"

Check OTEL Collector is Receiving Traces

docker logs devserver_otel_collector --tail=50 | Select-String "span"

Expected: Should see span activity

Check Jaeger is Receiving from OTEL

docker logs devserver_jaeger --tail=50 | Select-String "span"

Expected: Should see span ingestion logs

Query Jaeger API

# Get all services
Invoke-WebRequest -Uri "http://localhost:16686/api/services"

# Get traces for a specific service
Invoke-WebRequest -Uri "http://localhost:16686/api/traces?service=devserver.api.fastapi&limit=10"

---

🎯 Expected Behavior After Fix

1. Logs Should Show Real Trace IDs

2025-10-13 XX:XX:XX | DEBUG | devserver.api.fastapi | log_requests :61 | 
    TraceID: 4bf92f3577b34da6a3ce929d0e0e4736  ✅ (not "N/A")

2. OTEL Collector Should Show Activity

2025-10-13 XX:XX:XX | INFO | Span received from devserver.api.fastapi

3. Jaeger Services List

{
  "data": [
    "devserver.api.fastapi",  ✅
    "devserver.celery.worker",  ✅
    "jaeger-all-in-one"
  ]
}

4. Jaeger UI Should Show Traces

• Open: http://localhost:16686
• Select Service: devserver.api.fastapi
• See: List of traces with operations like "api.dice.roll", "api.tasks.fibonacci.create"

---

📝 Current Status Summary

Component	Status	Notes
Batch-Set API	✅ FIXED	Now converts dicts/lists to JSON strings
Containers	✅ Running	All 11 containers up
OTEL Collector	✅ Configured	Pipeline ready to receive traces
Jaeger	✅ Ready	UI accessible, OTLP enabled
Trace Generation	❌ NOT WORKING	TraceID shows "N/A"
Auto-Instrumentation	❌ NOT WORKING	Need explicit initialization

---

🚀 Next Actions

1. Implement Solution 1 (Recommended):

   • Create opentelemetry_config.py
   • Update main.py to call instrument_app(app)
   • Rebuild and restart

2. Or Use Simpler Approach:

   • Remove opentelemetry-instrument wrapper
   • Use explicit instrumentation in code
   • More control and easier debugging

3. Test Thoroughly:

   • Generate API traffic
   • Check logs for real trace IDs
   • Verify Jaeger shows traces

---

Last Updated: October 13, 2025
Status: Redis fix applied ✅ | Tracing fix documented, needs implementation ⏳