Initial commit: Complete DevServer monitoring framework
- FastAPI application with OpenTelemetry instrumentation - Redis caching and Celery task processing - Docker Compose setup with monitoring stack - Grafana dashboards and Prometheus metrics - Jaeger distributed tracing - Comprehensive documentation in docs/ folder - Monitoring configuration and troubleshooting guides
This commit is contained in:
Hardik Sankhla
368
docs/MONITORING_GUIDE.md
Normal file
368
docs/MONITORING_GUIDE.md
Normal file
@@ -0,0 +1,368 @@
|
||||
# DevServer Complete System Monitoring Guide
|
||||
|
||||
## 🎯 Overview
|
||||
|
||||
DevServer now includes a **comprehensive observability and monitoring stack** with automatic configuration. This guide covers all monitoring tools, dashboards, and troubleshooting procedures.
|
||||
|
||||
## 📊 Monitoring Architecture
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────┐
|
||||
│ Application Layer │
|
||||
│ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │
|
||||
│ │ FastAPI App │ │ Celery Worker│ │ Redis Cache │ │
|
||||
│ │ (Port 8000) │ │ │ │ (Port 6379) │ │
|
||||
│ └──────┬───────┘ └──────┬───────┘ └──────┬───────┘ │
|
||||
│ │ │ │ │
|
||||
└─────────┼──────────────────┼──────────────────┼──────────────┘
|
||||
│ │ │
|
||||
▼ ▼ ▼
|
||||
┌─────────────────────────────────────────────────────────────┐
|
||||
│ OpenTelemetry Instrumentation │
|
||||
│ (Traces, Metrics, Logs) │
|
||||
└─────────────────────────────────────────────────────────────┘
|
||||
│ │ │
|
||||
▼ ▼ ▼
|
||||
┌─────────────────────────────────────────────────────────────┐
|
||||
│ Collection & Processing │
|
||||
│ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │
|
||||
│ │OTEL Collector│ │ Prometheus │ │ Loki │ │
|
||||
│ │ │ │ (Port 9090) │ │ (Port 3100) │ │
|
||||
│ └──────┬───────┘ └──────┬───────┘ └──────┬───────┘ │
|
||||
└─────────┼──────────────────┼──────────────────┼──────────────┘
|
||||
│ │ │
|
||||
▼ ▼ ▼
|
||||
┌─────────────────────────────────────────────────────────────┐
|
||||
│ Visualization Layer │
|
||||
│ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │
|
||||
│ │ Grafana │ │ Jaeger │ │ cAdvisor │ │
|
||||
│ │ (Port 3000) │ │ (Port 16686) │ │ (Port 8080) │ │
|
||||
│ └──────────────┘ └──────────────┘ └──────────────┘ │
|
||||
└─────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
## 🚀 Services Overview
|
||||
|
||||
### Core Application Services
|
||||
- **FastAPI Backend** (`:8000`) - Main API with OpenTelemetry instrumentation
|
||||
- **Celery Worker** - Async task processing with Fibonacci calculations
|
||||
- **Redis** (`:6379`) - Cache layer with TTL-based expiration
|
||||
- **RabbitMQ** (`:5672`, `:15672`) - Message broker for Celery
|
||||
|
||||
### Monitoring Services
|
||||
- **Prometheus** (`:9090`) - Metrics collection and storage
|
||||
- **Grafana** (`:3000`) - Visualization dashboards
|
||||
- **Jaeger** (`:16686`) - Distributed tracing UI
|
||||
- **Loki** (`:3100`) - Log aggregation system
|
||||
- **Promtail** - Log shipping agent
|
||||
- **OTEL Collector** - Telemetry data processing
|
||||
- **cAdvisor** (`:8080`) - Container metrics
|
||||
- **Node Exporter** (`:9100`) - System metrics
|
||||
|
||||
## 📈 Monitoring Dashboards
|
||||
|
||||
### 1. Grafana Dashboard (Primary)
|
||||
**URL:** http://localhost:3000
|
||||
**Credentials:** admin / admin123
|
||||
|
||||
**Available Dashboards:**
|
||||
- **Complete System Monitoring** - All services health and metrics
|
||||
- **DevServer Observability** - Application-specific metrics
|
||||
|
||||
**Key Panels:**
|
||||
- Service Health Status (Real-time UP/DOWN)
|
||||
- API Request Rate & Response Time
|
||||
- Redis Cache Hit Rate & Memory Usage
|
||||
- Celery Task Queue & Duration
|
||||
- RabbitMQ Message Rates
|
||||
- HTTP Status Codes & Error Rates
|
||||
- Container CPU & Memory Usage
|
||||
- Prometheus Scrape Status
|
||||
|
||||
### 2. Prometheus Metrics
|
||||
**URL:** http://localhost:9090
|
||||
|
||||
**Key Queries:**
|
||||
```promql
|
||||
# Service health
|
||||
up
|
||||
|
||||
# API request rate
|
||||
rate(http_requests_total[5m])
|
||||
|
||||
# API response time (p95)
|
||||
histogram_quantile(0.95, rate(http_request_duration_seconds_bucket[5m]))
|
||||
|
||||
# Redis cache hit rate
|
||||
rate(redis_keyspace_hits_total[5m]) / (rate(redis_keyspace_hits_total[5m]) + rate(redis_keyspace_misses_total[5m]))
|
||||
|
||||
# Celery task count
|
||||
celery_tasks_total
|
||||
|
||||
# Error rate
|
||||
rate(http_requests_total{status=~"5.."}[5m])
|
||||
```
|
||||
|
||||
### 3. Jaeger Tracing
|
||||
**URL:** http://localhost:16686
|
||||
|
||||
**Features:**
|
||||
- Distributed trace visualization
|
||||
- Service dependency mapping
|
||||
- Request flow analysis
|
||||
- Performance bottleneck identification
|
||||
|
||||
**Services to Search:**
|
||||
- `devserver.api.fastapi`
|
||||
- `devserver.worker.celery`
|
||||
|
||||
### 4. cAdvisor Container Monitoring
|
||||
**URL:** http://localhost:8080
|
||||
|
||||
**Metrics:**
|
||||
- CPU usage per container
|
||||
- Memory usage per container
|
||||
- Network I/O
|
||||
- Disk I/O
|
||||
- Container health status
|
||||
|
||||
### 5. RabbitMQ Management
|
||||
**URL:** http://localhost:15672
|
||||
**Credentials:** guest / guest
|
||||
|
||||
**Metrics:**
|
||||
- Queue lengths
|
||||
- Message rates
|
||||
- Consumer connections
|
||||
- Memory usage
|
||||
|
||||
## 🔍 System Monitoring Script
|
||||
|
||||
Run the comprehensive system monitor:
|
||||
|
||||
```powershell
|
||||
# Check system status
|
||||
python system_monitor.py
|
||||
```
|
||||
|
||||
**Output Includes:**
|
||||
- ✓/✗ Health status for all services
|
||||
- Response times for each service
|
||||
- Prometheus metrics summary
|
||||
- Redis cache statistics
|
||||
- Recommendations for unhealthy services
|
||||
- Quick links to all dashboards
|
||||
|
||||
## 🛠️ Deployment & Management
|
||||
|
||||
### Start All Services
|
||||
```powershell
|
||||
docker-compose up -d
|
||||
```
|
||||
|
||||
### Check Service Status
|
||||
```powershell
|
||||
docker-compose ps
|
||||
```
|
||||
|
||||
### View Service Logs
|
||||
```powershell
|
||||
# All services
|
||||
docker-compose logs -f
|
||||
|
||||
# Specific service
|
||||
docker-compose logs -f devserver
|
||||
docker-compose logs -f prometheus
|
||||
docker-compose logs -f grafana
|
||||
```
|
||||
|
||||
### Restart Services
|
||||
```powershell
|
||||
# Restart all
|
||||
docker-compose restart
|
||||
|
||||
# Restart specific services
|
||||
docker-compose restart prometheus grafana otel-collector
|
||||
```
|
||||
|
||||
### Stop Services
|
||||
```powershell
|
||||
docker-compose down
|
||||
```
|
||||
|
||||
### Rebuild After Changes
|
||||
```powershell
|
||||
docker-compose down
|
||||
docker-compose up -d --build
|
||||
```
|
||||
|
||||
## 🎯 Monitoring Workflows
|
||||
|
||||
### 1. Check Overall System Health
|
||||
1. Open http://localhost:3000 (Grafana)
|
||||
2. Navigate to "Complete System Monitoring" dashboard
|
||||
3. Check "Service Health Status" panel
|
||||
4. Review error rates and response times
|
||||
|
||||
### 2. Debug Performance Issues
|
||||
1. Open Grafana dashboard
|
||||
2. Check "API Response Time" panel for slow endpoints
|
||||
3. Open Jaeger (http://localhost:16686)
|
||||
4. Search for traces with high duration
|
||||
5. Analyze bottlenecks in trace waterfall
|
||||
|
||||
### 3. Monitor Cache Effectiveness
|
||||
1. Open Grafana dashboard
|
||||
2. Review "Redis Cache Hit Rate" panel
|
||||
3. Check "Redis Memory Usage"
|
||||
4. Access Redis stats via API: `GET http://localhost:8000/api/cache/stats`
|
||||
|
||||
### 4. Track Task Processing
|
||||
1. Open Grafana dashboard
|
||||
2. Monitor "Celery Task Queue Length"
|
||||
3. Check "Celery Task Duration"
|
||||
4. Review RabbitMQ management UI for queue depths
|
||||
|
||||
### 5. Investigate Errors
|
||||
1. Check Grafana "Error Rate" panel
|
||||
2. Review Prometheus for error metrics
|
||||
3. Check logs via `docker-compose logs`
|
||||
4. Use Jaeger to trace failed requests
|
||||
|
||||
## 📊 Key Metrics to Monitor
|
||||
|
||||
### Application Metrics
|
||||
- **Request Rate:** Should be steady, spikes indicate traffic changes
|
||||
- **Response Time (p95):** Should be < 500ms for API calls
|
||||
- **Error Rate:** Should be < 1% under normal conditions
|
||||
- **Cache Hit Rate:** Should be > 80% for cached endpoints
|
||||
|
||||
### System Metrics
|
||||
- **CPU Usage:** Should be < 70% under normal load
|
||||
- **Memory Usage:** Should be stable, not constantly growing
|
||||
- **Disk I/O:** Should be low for cached operations
|
||||
- **Network I/O:** Should correlate with request rates
|
||||
|
||||
### Service-Specific
|
||||
- **Redis Memory:** Should be < configured max
|
||||
- **RabbitMQ Queue Depth:** Should be near zero (fast processing)
|
||||
- **Celery Tasks:** Should complete within expected timeframes
|
||||
|
||||
## 🚨 Troubleshooting
|
||||
|
||||
### Service is Down
|
||||
```powershell
|
||||
# Check service logs
|
||||
docker-compose logs <service-name>
|
||||
|
||||
# Restart service
|
||||
docker-compose restart <service-name>
|
||||
|
||||
# Check if port is in use
|
||||
netstat -ano | findstr :<port>
|
||||
```
|
||||
|
||||
### Grafana Data Source Issues
|
||||
```powershell
|
||||
# Restart Grafana
|
||||
docker-compose restart grafana
|
||||
|
||||
# Check data source configuration
|
||||
# Go to Grafana → Configuration → Data Sources
|
||||
```
|
||||
|
||||
### OTEL Collector Not Working
|
||||
```powershell
|
||||
# Check configuration
|
||||
docker-compose logs otel-collector
|
||||
|
||||
# Verify config file
|
||||
cat monitoring/otel-collector-config.yaml
|
||||
|
||||
# Restart collector
|
||||
docker-compose restart otel-collector
|
||||
```
|
||||
|
||||
### Prometheus Not Scraping
|
||||
```powershell
|
||||
# Check Prometheus targets
|
||||
# Open http://localhost:9090/targets
|
||||
|
||||
# Verify Prometheus config
|
||||
cat monitoring/prometheus.yml
|
||||
|
||||
# Restart Prometheus
|
||||
docker-compose restart prometheus
|
||||
```
|
||||
|
||||
### High Memory Usage
|
||||
```powershell
|
||||
# Check container stats
|
||||
docker stats
|
||||
|
||||
# Check specific service
|
||||
docker-compose logs <service-name>
|
||||
|
||||
# Restart heavy services
|
||||
docker-compose restart devserver worker
|
||||
```
|
||||
|
||||
## 🔐 Security Considerations
|
||||
|
||||
### Change Default Passwords
|
||||
Edit `docker-compose.yml`:
|
||||
```yaml
|
||||
environment:
|
||||
GF_SECURITY_ADMIN_PASSWORD: your_secure_password
|
||||
```
|
||||
|
||||
### Restrict External Access
|
||||
For production, use:
|
||||
```yaml
|
||||
ports:
|
||||
- "127.0.0.1:3000:3000" # Grafana only on localhost
|
||||
```
|
||||
|
||||
### Enable HTTPS
|
||||
Use a reverse proxy (nginx/traefik) with SSL certificates.
|
||||
|
||||
## 📝 Configuration Files
|
||||
|
||||
### Key Configuration Files
|
||||
- `docker-compose.yml` - Service orchestration
|
||||
- `monitoring/prometheus.yml` - Prometheus scrape config
|
||||
- `monitoring/otel-collector-config.yaml` - OTEL collector config
|
||||
- `monitoring/loki-config.yaml` - Loki config
|
||||
- `monitoring/promtail-config.yaml` - Promtail config
|
||||
- `monitoring/grafana/provisioning/` - Grafana auto-provisioning
|
||||
|
||||
## 🎓 Best Practices
|
||||
|
||||
1. **Regular Monitoring:** Check dashboards at least daily
|
||||
2. **Set Alerts:** Configure Grafana alerts for critical metrics
|
||||
3. **Log Retention:** Adjust log retention based on storage
|
||||
4. **Backup Configs:** Version control all monitoring configs
|
||||
5. **Resource Limits:** Set container resource limits in production
|
||||
6. **Security:** Change default passwords and restrict ports
|
||||
7. **Documentation:** Keep this guide updated with changes
|
||||
|
||||
## 📚 Additional Resources
|
||||
|
||||
- **Prometheus Documentation:** https://prometheus.io/docs/
|
||||
- **Grafana Documentation:** https://grafana.com/docs/
|
||||
- **Jaeger Documentation:** https://www.jaegertracing.io/docs/
|
||||
- **OpenTelemetry:** https://opentelemetry.io/docs/
|
||||
- **cAdvisor:** https://github.com/google/cadvisor
|
||||
|
||||
## 🆘 Support
|
||||
|
||||
For issues or questions:
|
||||
1. Check service logs: `docker-compose logs <service>`
|
||||
2. Run system monitor: `python system_monitor.py`
|
||||
3. Review Grafana dashboards for anomalies
|
||||
4. Check this documentation for troubleshooting steps
|
||||
|
||||
---
|
||||
|
||||
**Last Updated:** October 13, 2025
|
||||
**Version:** 2.0.0
|
||||
Reference in New Issue
Block a user