Initial commit: Complete DevServer monitoring framework
- FastAPI application with OpenTelemetry instrumentation - Redis caching and Celery task processing - Docker Compose setup with monitoring stack - Grafana dashboards and Prometheus metrics - Jaeger distributed tracing - Comprehensive documentation in docs/ folder - Monitoring configuration and troubleshooting guides
This commit is contained in:
Hardik Sankhla
404
docs/ISSUES_FIXED_AND_TROUBLESHOOTING.md
Normal file
404
docs/ISSUES_FIXED_AND_TROUBLESHOOTING.md
Normal file
@@ -0,0 +1,404 @@
|
||||
# 🔧 Issues Fixed & Troubleshooting Guide
|
||||
|
||||
**Date:** October 13, 2025
|
||||
**Status:** Partial Fix Applied ✅ / Traces Still Need Configuration ⚠️
|
||||
|
||||
---
|
||||
|
||||
## ✅ FIXED: Redis Batch-Set API Error
|
||||
|
||||
### Problem
|
||||
```
|
||||
POST /api/cache/batch-set
|
||||
Error 500: Invalid input of type: 'dict'. Convert to a bytes, string, int or float first.
|
||||
```
|
||||
|
||||
### Root Cause
|
||||
The `batch-set` endpoint was trying to store Python dict objects directly in Redis, but Redis only accepts strings, bytes, ints, or floats.
|
||||
|
||||
### Solution Applied
|
||||
**File:** `app/main.py` (lines 599-617)
|
||||
|
||||
**Changed:**
|
||||
```python
|
||||
for key, value in items.items():
|
||||
pipeline.set(key, value) # ❌ Error: can't store dict
|
||||
```
|
||||
|
||||
**To:**
|
||||
```python
|
||||
for key, value in items.items():
|
||||
# Convert value to JSON string if it's a dict, list, or other complex type
|
||||
if isinstance(value, (dict, list)):
|
||||
value_str = json.dumps(value)
|
||||
else:
|
||||
value_str = str(value)
|
||||
pipeline.set(key, value_str) # ✅ Fixed: stores as JSON string
|
||||
```
|
||||
|
||||
### Testing
|
||||
```powershell
|
||||
# Test the fixed endpoint
|
||||
$body = '{"key1":"value1","key2":{"nested":"object"},"key3":["array","values"]}'
|
||||
Invoke-WebRequest -Uri "http://localhost:8000/api/cache/batch-set" `
|
||||
-Method POST -Body $body -ContentType "application/json"
|
||||
```
|
||||
|
||||
**Expected:** Status 200, returns success count
|
||||
|
||||
---
|
||||
|
||||
## ⚠️ ISSUE: Jaeger Shows No Traces
|
||||
|
||||
### Problem
|
||||
- Jaeger UI shows only `jaeger-all-in-one` service
|
||||
- No traces from `devserver.api.fastapi` or `devserver.celery.worker`
|
||||
- Logs show `TraceID: N/A` for all requests
|
||||
|
||||
### Root Cause Analysis
|
||||
|
||||
#### 1. OpenTelemetry Auto-Instrumentation Not Initializing
|
||||
**Evidence:**
|
||||
```
|
||||
2025-10-13 03:56:20 | DEBUG | devserver.api.fastapi | log_requests :61 | TraceID: N/A
|
||||
```
|
||||
|
||||
The `TraceID: N/A` means no active span context exists.
|
||||
|
||||
**Why:**
|
||||
- `opentelemetry-instrument` command is used in docker-compose
|
||||
- Environment variables are set (`OTEL_EXPORTER_OTLP_ENDPOINT`, etc.)
|
||||
- BUT: The tracer provider might not be initializing before the app starts
|
||||
|
||||
#### 2. Jaeger Configuration
|
||||
**Jaeger logs show:**
|
||||
```json
|
||||
{"level":"info","msg":"[core] [Channel #1]parsed dial target is: \"localhost:4317\""}
|
||||
```
|
||||
|
||||
Jaeger is trying to connect to `localhost:4317` instead of receiving traces. This is expected behavior - Jaeger has OTLP enabled and is ready to receive.
|
||||
|
||||
#### 3. OTEL Collector Pipeline
|
||||
**Configuration looks correct:**
|
||||
```yaml
|
||||
pipelines:
|
||||
traces:
|
||||
receivers: [otlp]
|
||||
processors: [memory_limiter, batch]
|
||||
exporters: [debug, otlphttp/jaeger]
|
||||
```
|
||||
|
||||
But no trace activity in logs means traces aren't reaching the collector.
|
||||
|
||||
---
|
||||
|
||||
## 🔍 Diagnosis Steps Performed
|
||||
|
||||
### 1. Checked OTEL Packages ✅
|
||||
```toml
|
||||
"opentelemetry-distro>=0.58b0"
|
||||
"opentelemetry-exporter-otlp-proto-grpc>=1.37.0"
|
||||
"opentelemetry-instrumentation-fastapi>=0.58b0"
|
||||
"opentelemetry-instrumentation-redis>=0.58b0"
|
||||
"opentelemetry-instrumentation-celery>=0.58b0"
|
||||
```
|
||||
All packages are installed.
|
||||
|
||||
### 2. Checked Environment Variables ✅
|
||||
```yaml
|
||||
OTEL_TRACES_EXPORTER: otlp
|
||||
OTEL_METRICS_EXPORTER: otlp
|
||||
OTEL_LOGS_EXPORTER: otlp
|
||||
OTEL_EXPORTER_OTLP_ENDPOINT: http://otel-collector:4317
|
||||
OTEL_EXPORTER_OTLP_PROTOCOL: grpc
|
||||
OTEL_SERVICE_NAME: devserver.api.fastapi
|
||||
```
|
||||
All variables are set correctly.
|
||||
|
||||
### 3. Checked Container Startup Command ✅
|
||||
```yaml
|
||||
command: [
|
||||
"uv", "run",
|
||||
"opentelemetry-instrument",
|
||||
"--service_name", "devserver.api.fastapi",
|
||||
"uvicorn", "app.main:app",
|
||||
"--host", "0.0.0.0", "--port", "8000", "--workers", "2"
|
||||
]
|
||||
```
|
||||
Command structure is correct.
|
||||
|
||||
### 4. Checked Network Connectivity ✅
|
||||
- All containers on `devserver_network`
|
||||
- OTEL collector accessible at `otel-collector:4317`
|
||||
- Jaeger accessible at `jaeger:14268`
|
||||
|
||||
---
|
||||
|
||||
## 🛠️ Solutions to Apply
|
||||
|
||||
### Solution 1: Explicit TracerProvider Initialization ⭐ RECOMMENDED
|
||||
|
||||
Create an OpenTelemetry initialization module to explicitly configure tracing before app startup.
|
||||
|
||||
**File:** `app/opentelemetry_config.py` (NEW FILE)
|
||||
|
||||
```python
|
||||
"""OpenTelemetry configuration for DevServer"""
|
||||
import os
|
||||
from opentelemetry import trace
|
||||
from opentelemetry.sdk.trace import TracerProvider
|
||||
from opentelemetry.sdk.trace.export import BatchSpanProcessor
|
||||
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
|
||||
from opentelemetry.sdk.resources import Resource, SERVICE_NAME
|
||||
from opentelemetry.instrumentation.fastapi import FastAPIInstrumentor
|
||||
from opentelemetry.instrumentation.redis import RedisInstrumentor
|
||||
import logging
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
def init_opentelemetry():
|
||||
"""Initialize OpenTelemetry with OTLP exporter"""
|
||||
|
||||
# Get configuration from environment
|
||||
service_name = os.getenv("OTEL_SERVICE_NAME", "devserver.api.fastapi")
|
||||
otlp_endpoint = os.getenv("OTEL_EXPORTER_OTLP_ENDPOINT", "http://otel-collector:4317")
|
||||
|
||||
logger.info(f"Initializing OpenTelemetry for {service_name}")
|
||||
logger.info(f"OTLP Endpoint: {otlp_endpoint}")
|
||||
|
||||
# Create resource with service name
|
||||
resource = Resource(attributes={
|
||||
SERVICE_NAME: service_name
|
||||
})
|
||||
|
||||
# Create tracer provider
|
||||
provider = TracerProvider(resource=resource)
|
||||
|
||||
# Create OTLP exporter
|
||||
otlp_exporter = OTLPSpanExporter(
|
||||
endpoint=otlp_endpoint,
|
||||
insecure=True # Use insecure channel (no TLS)
|
||||
)
|
||||
|
||||
# Add span processor
|
||||
span_processor = BatchSpanProcessor(otlp_exporter)
|
||||
provider.add_span_processor(span_processor)
|
||||
|
||||
# Set as global tracer provider
|
||||
trace.set_tracer_provider(provider)
|
||||
|
||||
logger.info("✓ OpenTelemetry initialized successfully")
|
||||
|
||||
return provider
|
||||
|
||||
def instrument_app(app):
|
||||
"""Instrument FastAPI app with OpenTelemetry"""
|
||||
|
||||
# Initialize OpenTelemetry
|
||||
init_opentelemetry()
|
||||
|
||||
# Instrument FastAPI
|
||||
FastAPIInstrumentor().instrument_app(app)
|
||||
|
||||
# Instrument Redis
|
||||
RedisInstrumentor().instrument()
|
||||
|
||||
logger.info("✓ FastAPI and Redis instrumented")
|
||||
```
|
||||
|
||||
**Update `app/main.py`:**
|
||||
```python
|
||||
# Add at the top after imports
|
||||
from app.opentelemetry_config import instrument_app
|
||||
|
||||
# After creating the FastAPI app
|
||||
app = FastAPI(...)
|
||||
|
||||
# Instrument with OpenTelemetry
|
||||
instrument_app(app)
|
||||
```
|
||||
|
||||
### Solution 2: Use Bootstrap Script (Alternative)
|
||||
|
||||
**File:** `app/bootstrap.py` (NEW FILE)
|
||||
```python
|
||||
#!/usr/bin/env python
|
||||
"""Bootstrap script to initialize OpenTelemetry before app starts"""
|
||||
import os
|
||||
os.environ.setdefault("OTEL_PYTHON_LOG_LEVEL", "debug")
|
||||
|
||||
from opentelemetry.instrumentation.auto_instrumentation import sitecustomize
|
||||
|
||||
# This will run before anything else
|
||||
sitecustomize.initialize()
|
||||
|
||||
print("✓ OpenTelemetry bootstrap complete")
|
||||
```
|
||||
|
||||
**Update docker-compose.yml:**
|
||||
```yaml
|
||||
command: [
|
||||
"uv", "run",
|
||||
"python", "-m", "app.bootstrap", # Run bootstrap first
|
||||
"opentelemetry-instrument",
|
||||
"--service_name", "devserver.api.fastapi",
|
||||
"uvicorn", "app.main:app",
|
||||
"--host", "0.0.0.0", "--port", "8000"
|
||||
]
|
||||
```
|
||||
|
||||
### Solution 3: Debug Mode (For Troubleshooting)
|
||||
|
||||
Add debug logging to see what's happening:
|
||||
|
||||
```yaml
|
||||
environment:
|
||||
OTEL_TRACES_EXPORTER: otlp
|
||||
OTEL_EXPORTER_OTLP_ENDPOINT: http://otel-collector:4317
|
||||
OTEL_PYTHON_LOG_LEVEL: debug # Add this
|
||||
OTEL_LOG_LEVEL: debug # And this
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 📋 Step-by-Step Fix Implementation
|
||||
|
||||
### Step 1: Create OpenTelemetry Config File
|
||||
```powershell
|
||||
cd d:\GNPL-Private\DevServer
|
||||
# Create the file with the code above
|
||||
```
|
||||
|
||||
### Step 2: Update main.py
|
||||
Add the import and instrumentation call
|
||||
|
||||
### Step 3: Rebuild Containers
|
||||
```powershell
|
||||
docker-compose build --no-cache devserver celery_worker
|
||||
```
|
||||
|
||||
### Step 4: Restart Services
|
||||
```powershell
|
||||
docker-compose up -d --force-recreate devserver celery_worker
|
||||
```
|
||||
|
||||
### Step 5: Generate Test Traffic
|
||||
```powershell
|
||||
# Make some API calls
|
||||
for ($i=1; $i -le 5; $i++) {
|
||||
Invoke-WebRequest -Uri "http://localhost:8000/api/dice/roll?session_id=test-$i"
|
||||
Start-Sleep -Milliseconds 500
|
||||
}
|
||||
```
|
||||
|
||||
### Step 6: Check Jaeger
|
||||
```powershell
|
||||
# Wait 10 seconds for traces to propagate
|
||||
Start-Sleep -Seconds 10
|
||||
|
||||
# Check services
|
||||
$response = Invoke-WebRequest -Uri "http://localhost:16686/api/services"
|
||||
($response.Content | ConvertFrom-Json).data
|
||||
```
|
||||
|
||||
**Expected:** Should see `devserver.api.fastapi` in the list
|
||||
|
||||
---
|
||||
|
||||
## 🧪 Verification Commands
|
||||
|
||||
### Check if Traces are Being Generated
|
||||
```powershell
|
||||
# Check FastAPI logs for trace IDs
|
||||
docker logs devserver_backend --tail=20 | Select-String "TraceID"
|
||||
```
|
||||
**Expected:** Should see actual hex trace IDs instead of "N/A"
|
||||
|
||||
### Check OTEL Collector is Receiving Traces
|
||||
```powershell
|
||||
docker logs devserver_otel_collector --tail=50 | Select-String "span"
|
||||
```
|
||||
**Expected:** Should see span activity
|
||||
|
||||
### Check Jaeger is Receiving from OTEL
|
||||
```powershell
|
||||
docker logs devserver_jaeger --tail=50 | Select-String "span"
|
||||
```
|
||||
**Expected:** Should see span ingestion logs
|
||||
|
||||
### Query Jaeger API
|
||||
```powershell
|
||||
# Get all services
|
||||
Invoke-WebRequest -Uri "http://localhost:16686/api/services"
|
||||
|
||||
# Get traces for a specific service
|
||||
Invoke-WebRequest -Uri "http://localhost:16686/api/traces?service=devserver.api.fastapi&limit=10"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🎯 Expected Behavior After Fix
|
||||
|
||||
### 1. Logs Should Show Real Trace IDs
|
||||
```
|
||||
2025-10-13 XX:XX:XX | DEBUG | devserver.api.fastapi | log_requests :61 |
|
||||
TraceID: 4bf92f3577b34da6a3ce929d0e0e4736 ✅ (not "N/A")
|
||||
```
|
||||
|
||||
### 2. OTEL Collector Should Show Activity
|
||||
```
|
||||
2025-10-13 XX:XX:XX | INFO | Span received from devserver.api.fastapi
|
||||
```
|
||||
|
||||
### 3. Jaeger Services List
|
||||
```json
|
||||
{
|
||||
"data": [
|
||||
"devserver.api.fastapi", ✅
|
||||
"devserver.celery.worker", ✅
|
||||
"jaeger-all-in-one"
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
### 4. Jaeger UI Should Show Traces
|
||||
- Open: http://localhost:16686
|
||||
- Select Service: `devserver.api.fastapi`
|
||||
- See: List of traces with operations like "api.dice.roll", "api.tasks.fibonacci.create"
|
||||
|
||||
---
|
||||
|
||||
## 📝 Current Status Summary
|
||||
|
||||
| Component | Status | Notes |
|
||||
|-----------|--------|-------|
|
||||
| **Batch-Set API** | ✅ FIXED | Now converts dicts/lists to JSON strings |
|
||||
| **Containers** | ✅ Running | All 11 containers up |
|
||||
| **OTEL Collector** | ✅ Configured | Pipeline ready to receive traces |
|
||||
| **Jaeger** | ✅ Ready | UI accessible, OTLP enabled |
|
||||
| **Trace Generation** | ❌ NOT WORKING | TraceID shows "N/A" |
|
||||
| **Auto-Instrumentation** | ❌ NOT WORKING | Need explicit initialization |
|
||||
|
||||
---
|
||||
|
||||
## 🚀 Next Actions
|
||||
|
||||
1. **Implement Solution 1** (Recommended):
|
||||
- Create `opentelemetry_config.py`
|
||||
- Update `main.py` to call `instrument_app(app)`
|
||||
- Rebuild and restart
|
||||
|
||||
2. **Or Use Simpler Approach**:
|
||||
- Remove `opentelemetry-instrument` wrapper
|
||||
- Use explicit instrumentation in code
|
||||
- More control and easier debugging
|
||||
|
||||
3. **Test Thoroughly**:
|
||||
- Generate API traffic
|
||||
- Check logs for real trace IDs
|
||||
- Verify Jaeger shows traces
|
||||
|
||||
---
|
||||
|
||||
**Last Updated:** October 13, 2025
|
||||
**Status:** Redis fix applied ✅ | Tracing fix documented, needs implementation ⏳
|
||||
Reference in New Issue
Block a user