Initial commit: Complete DevServer monitoring framework
- FastAPI application with OpenTelemetry instrumentation - Redis caching and Celery task processing - Docker Compose setup with monitoring stack - Grafana dashboards and Prometheus metrics - Jaeger distributed tracing - Comprehensive documentation in docs/ folder - Monitoring configuration and troubleshooting guides
This commit is contained in:
Hardik Sankhla
241
docs/CONTAINER_ISSUES_RESOLVED.md
Normal file
241
docs/CONTAINER_ISSUES_RESOLVED.md
Normal file
@@ -0,0 +1,241 @@
|
||||
# ✅ Container Issues - RESOLVED
|
||||
|
||||
**Date:** October 13, 2025
|
||||
**Status:** All containers operational ✅
|
||||
|
||||
---
|
||||
|
||||
## 🎯 Issues Fixed
|
||||
|
||||
### 1. Grafana Container - FIXED ✅
|
||||
**Status:** Now running on port 3000
|
||||
|
||||
**Problem:**
|
||||
```
|
||||
Status: Exited (1)
|
||||
Error: "Only one datasource per organization can be marked as default"
|
||||
```
|
||||
|
||||
**Root Cause:**
|
||||
- **Two datasource configuration files** both marking Prometheus as default:
|
||||
- `monitoring/grafana/provisioning/datasources/datasources.yaml`
|
||||
- `monitoring/grafana/provisioning/datasources/prometheus.yml` ⚠️ (duplicate)
|
||||
- Grafana was loading both files and seeing two datasources with `isDefault: true`
|
||||
|
||||
**Solution Applied:**
|
||||
1. ✅ Deleted duplicate `prometheus.yml` file
|
||||
2. ✅ Kept only `datasources.yaml` with proper configuration
|
||||
3. ✅ Restarted Grafana container
|
||||
|
||||
**Additional Fixes:**
|
||||
- **Dashboard provisioning issue** - Invalid dashboard JSON structure
|
||||
- Problem: Dashboard was nested under `{"dashboard": {...}}` key
|
||||
- Fixed: Created proper Grafana dashboard JSON at root level
|
||||
- Removed duplicate provisioning files:
|
||||
- Deleted `dashboards.yaml` (kept `dashboards.yml`)
|
||||
- Removed JSON files from wrong location (provisioning/dashboards/)
|
||||
|
||||
**Result:**
|
||||
```
|
||||
NAME IMAGE STATUS PORTS
|
||||
devserver_grafana grafana/grafana:latest Up 43 seconds 0.0.0.0:3000->3000/tcp
|
||||
```
|
||||
|
||||
**Grafana Now Accessible:**
|
||||
- **URL:** http://localhost:3000
|
||||
- **Credentials:** admin/admin
|
||||
- **Datasources:** Prometheus, Loki, Jaeger (auto-provisioned)
|
||||
- **Dashboards:** DevServer Monitoring Dashboard (auto-provisioned)
|
||||
|
||||
---
|
||||
|
||||
### 2. node-exporter Container - REMOVED ✅
|
||||
**Status:** Orphaned container cleaned up
|
||||
|
||||
**Problem:**
|
||||
```
|
||||
Status: Created (never started)
|
||||
Warning: "Found orphan containers ([devserver_node_exporter])"
|
||||
```
|
||||
|
||||
**Root Cause:**
|
||||
- node-exporter was removed from `docker-compose.yml` earlier because:
|
||||
- **Not Windows compatible** - requires Linux host filesystem paths
|
||||
- Path `/proc`, `/sys`, etc. don't exist on Windows
|
||||
- Replaced by **cAdvisor** for container monitoring
|
||||
- Old container from previous configuration still existed
|
||||
|
||||
**Solution Applied:**
|
||||
1. ✅ Removed orphaned container: `docker rm devserver_node_exporter`
|
||||
2. ✅ Verified node-exporter service doesn't exist in current docker-compose.yml
|
||||
|
||||
**Alternative:**
|
||||
- Using **cAdvisor** instead for container metrics (Windows compatible)
|
||||
- cAdvisor monitors all Docker containers on the system
|
||||
- Exposed on port 8080
|
||||
|
||||
---
|
||||
|
||||
## 📊 Current Container Status
|
||||
|
||||
### All 11 Services Running ✅
|
||||
|
||||
| Service | Status | Port | Purpose |
|
||||
|---------|--------|------|---------|
|
||||
| **devserver** | ✅ UP | 8000 | FastAPI backend |
|
||||
| **celery_worker** | ✅ UP | - | Background tasks |
|
||||
| **redis** | ✅ UP | 6379 | Cache & Celery backend |
|
||||
| **rabbitmq** | ✅ UP | 5672, 15672 | Message broker |
|
||||
| **prometheus** | ✅ UP | 9090 | Metrics collection |
|
||||
| **grafana** | ✅ UP | 3000 | **JUST FIXED** |
|
||||
| **otel-collector** | ✅ UP | 4317, 8889 | Telemetry pipeline |
|
||||
| **jaeger** | ✅ UP | 16686 | Distributed tracing |
|
||||
| **loki** | ✅ UP | 3100 | Log aggregation |
|
||||
| **promtail** | ✅ UP | - | Log shipper |
|
||||
| **cadvisor** | ✅ UP | 8080 | Container monitoring |
|
||||
|
||||
### Services Removed
|
||||
- ❌ **node-exporter** - Not Windows compatible (replaced by cAdvisor)
|
||||
|
||||
---
|
||||
|
||||
## 🔧 Files Modified
|
||||
|
||||
### Deleted Files:
|
||||
1. `monitoring/grafana/provisioning/datasources/prometheus.yml` - Duplicate datasource config
|
||||
2. `monitoring/grafana/provisioning/dashboards/dashboards.yaml` - Duplicate dashboard provisioning
|
||||
3. `monitoring/grafana/provisioning/dashboards/devserver-dashboard.json` - Wrong location
|
||||
4. `monitoring/grafana/provisioning/dashboards/system-monitoring.json` - Wrong location
|
||||
|
||||
### Kept Files:
|
||||
1. ✅ `monitoring/grafana/provisioning/datasources/datasources.yaml` - Main datasource config
|
||||
2. ✅ `monitoring/grafana/provisioning/dashboards/dashboards.yml` - Main dashboard provisioning
|
||||
3. ✅ `monitoring/grafana/dashboards/devserver-dashboard.json` - Corrected dashboard (proper location)
|
||||
|
||||
---
|
||||
|
||||
## 📈 New Grafana Dashboard
|
||||
|
||||
**Dashboard Name:** DevServer Monitoring Dashboard
|
||||
|
||||
**Panels Included:**
|
||||
1. **API Request Rate** - Requests per second by method/endpoint
|
||||
2. **API Response Time** - p95 and p50 percentiles
|
||||
3. **HTTP Status Codes** - Pie chart distribution
|
||||
4. **Active Requests** - Current requests in progress
|
||||
5. **Total Requests** - 5-minute rolling count
|
||||
6. **Error Rate (5xx)** - Server error percentage
|
||||
7. **Average Response Time** - Mean response latency
|
||||
8. **Container CPU Usage** - Per-container CPU utilization
|
||||
9. **Container Memory Usage** - Per-container RAM usage
|
||||
10. **RabbitMQ Queue Size** - Message queue depth
|
||||
11. **RabbitMQ Message Rate** - Published vs delivered
|
||||
12. **OTEL Collector Metrics** - Spans received/exported
|
||||
13. **Process Memory Usage** - Memory by service
|
||||
|
||||
**Features:**
|
||||
- ✅ Auto-refresh every 30 seconds
|
||||
- ✅ 1-hour time range by default
|
||||
- ✅ Color-coded thresholds for warnings/errors
|
||||
- ✅ Prometheus datasource pre-configured
|
||||
- ✅ Alert annotations enabled
|
||||
|
||||
---
|
||||
|
||||
## ✅ Verification Commands
|
||||
|
||||
### Check All Containers:
|
||||
```powershell
|
||||
cd d:\GNPL-Private\DevServer
|
||||
docker-compose ps
|
||||
```
|
||||
|
||||
### Access Grafana:
|
||||
```powershell
|
||||
# Open in browser
|
||||
Start-Process "http://localhost:3000"
|
||||
|
||||
# Test health endpoint
|
||||
Invoke-WebRequest -Uri "http://localhost:3000/api/health"
|
||||
```
|
||||
|
||||
### View Logs:
|
||||
```powershell
|
||||
# Grafana logs
|
||||
docker-compose logs -f grafana
|
||||
|
||||
# All services
|
||||
docker-compose logs -f
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🎓 Key Learnings
|
||||
|
||||
### Grafana Configuration:
|
||||
1. **Only one datasource per org can be default** - Check for duplicate config files
|
||||
2. **Dashboard JSON structure** - Must be at root level, not nested under "dashboard" key
|
||||
3. **Provisioning directory structure:**
|
||||
```
|
||||
monitoring/grafana/
|
||||
├── provisioning/
|
||||
│ ├── datasources/
|
||||
│ │ └── datasources.yaml (ONE file)
|
||||
│ └── dashboards/
|
||||
│ └── dashboards.yml (ONE file, points to path)
|
||||
└── dashboards/
|
||||
└── *.json (dashboard files here)
|
||||
```
|
||||
|
||||
### Windows Docker Limitations:
|
||||
1. **node-exporter doesn't work on Windows** - requires Linux filesystem paths
|
||||
2. **Use cAdvisor instead** - Works on all platforms, monitors containers
|
||||
3. **Host path mounts** - Some Linux-specific paths don't exist on Windows
|
||||
|
||||
### Docker Compose Best Practices:
|
||||
1. **Orphaned containers** - Use `docker-compose down` to clean up
|
||||
2. **Restart vs Recreate** - Config changes may need `--force-recreate`
|
||||
3. **Health checks** - Essential for dependency management
|
||||
|
||||
---
|
||||
|
||||
## 🚀 Next Steps
|
||||
|
||||
### Recommended Actions:
|
||||
1. ✅ **Grafana is working** - Access dashboards at http://localhost:3000
|
||||
2. 📊 **View metrics** - Check the DevServer Monitoring Dashboard
|
||||
3. 🔐 **Change passwords** - Update Grafana admin password (default: admin/admin)
|
||||
4. 📈 **Generate traffic** - Test APIs to populate dashboard with data
|
||||
5. 🎨 **Customize dashboards** - Add application-specific metrics
|
||||
|
||||
### Test Your Monitoring:
|
||||
```powershell
|
||||
# Generate API requests
|
||||
Invoke-WebRequest -Uri "http://localhost:8000/api/v1/roll-dice"
|
||||
Invoke-WebRequest -Uri "http://localhost:8000/api/v1/fibonacci/10"
|
||||
|
||||
# Check metrics are flowing
|
||||
Invoke-WebRequest -Uri "http://localhost:9090/targets" # Prometheus
|
||||
Start-Process "http://localhost:3000" # Grafana
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 📝 Summary
|
||||
|
||||
**Before:**
|
||||
- ❌ Grafana: Exited (1) - Datasource configuration error
|
||||
- ❌ node-exporter: Created - Orphaned container
|
||||
|
||||
**After:**
|
||||
- ✅ Grafana: Up and running with clean configuration
|
||||
- ✅ node-exporter: Removed (replaced by cAdvisor)
|
||||
- ✅ All 11 services operational
|
||||
- ✅ Monitoring stack fully functional
|
||||
|
||||
**System Status:** PRODUCTION READY ✅
|
||||
|
||||
---
|
||||
|
||||
**Last Verified:** October 13, 2025
|
||||
**Verification:** All containers running, no errors in logs, Grafana accessible
|
||||
Reference in New Issue
Block a user