2304c3ed12
- FastAPI application with OpenTelemetry instrumentation - Redis caching and Celery task processing - Docker Compose setup with monitoring stack - Grafana dashboards and Prometheus metrics - Jaeger distributed tracing - Comprehensive documentation in docs/ folder - Monitoring configuration and troubleshooting guides
369 lines
12 KiB
Markdown
369 lines
12 KiB
Markdown
# DevServer Complete System Monitoring Guide
|
|
|
|
## 🎯 Overview
|
|
|
|
DevServer now includes a **comprehensive observability and monitoring stack** with automatic configuration. This guide covers all monitoring tools, dashboards, and troubleshooting procedures.
|
|
|
|
## 📊 Monitoring Architecture
|
|
|
|
```
|
|
┌─────────────────────────────────────────────────────────────┐
|
|
│ Application Layer │
|
|
│ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │
|
|
│ │ FastAPI App │ │ Celery Worker│ │ Redis Cache │ │
|
|
│ │ (Port 8000) │ │ │ │ (Port 6379) │ │
|
|
│ └──────┬───────┘ └──────┬───────┘ └──────┬───────┘ │
|
|
│ │ │ │ │
|
|
└─────────┼──────────────────┼──────────────────┼──────────────┘
|
|
│ │ │
|
|
▼ ▼ ▼
|
|
┌─────────────────────────────────────────────────────────────┐
|
|
│ OpenTelemetry Instrumentation │
|
|
│ (Traces, Metrics, Logs) │
|
|
└─────────────────────────────────────────────────────────────┘
|
|
│ │ │
|
|
▼ ▼ ▼
|
|
┌─────────────────────────────────────────────────────────────┐
|
|
│ Collection & Processing │
|
|
│ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │
|
|
│ │OTEL Collector│ │ Prometheus │ │ Loki │ │
|
|
│ │ │ │ (Port 9090) │ │ (Port 3100) │ │
|
|
│ └──────┬───────┘ └──────┬───────┘ └──────┬───────┘ │
|
|
└─────────┼──────────────────┼──────────────────┼──────────────┘
|
|
│ │ │
|
|
▼ ▼ ▼
|
|
┌─────────────────────────────────────────────────────────────┐
|
|
│ Visualization Layer │
|
|
│ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │
|
|
│ │ Grafana │ │ Jaeger │ │ cAdvisor │ │
|
|
│ │ (Port 3000) │ │ (Port 16686) │ │ (Port 8080) │ │
|
|
│ └──────────────┘ └──────────────┘ └──────────────┘ │
|
|
└─────────────────────────────────────────────────────────────┘
|
|
```
|
|
|
|
## 🚀 Services Overview
|
|
|
|
### Core Application Services
|
|
- **FastAPI Backend** (`:8000`) - Main API with OpenTelemetry instrumentation
|
|
- **Celery Worker** - Async task processing with Fibonacci calculations
|
|
- **Redis** (`:6379`) - Cache layer with TTL-based expiration
|
|
- **RabbitMQ** (`:5672`, `:15672`) - Message broker for Celery
|
|
|
|
### Monitoring Services
|
|
- **Prometheus** (`:9090`) - Metrics collection and storage
|
|
- **Grafana** (`:3000`) - Visualization dashboards
|
|
- **Jaeger** (`:16686`) - Distributed tracing UI
|
|
- **Loki** (`:3100`) - Log aggregation system
|
|
- **Promtail** - Log shipping agent
|
|
- **OTEL Collector** - Telemetry data processing
|
|
- **cAdvisor** (`:8080`) - Container metrics
|
|
- **Node Exporter** (`:9100`) - System metrics
|
|
|
|
## 📈 Monitoring Dashboards
|
|
|
|
### 1. Grafana Dashboard (Primary)
|
|
**URL:** http://localhost:3000
|
|
**Credentials:** admin / admin123
|
|
|
|
**Available Dashboards:**
|
|
- **Complete System Monitoring** - All services health and metrics
|
|
- **DevServer Observability** - Application-specific metrics
|
|
|
|
**Key Panels:**
|
|
- Service Health Status (Real-time UP/DOWN)
|
|
- API Request Rate & Response Time
|
|
- Redis Cache Hit Rate & Memory Usage
|
|
- Celery Task Queue & Duration
|
|
- RabbitMQ Message Rates
|
|
- HTTP Status Codes & Error Rates
|
|
- Container CPU & Memory Usage
|
|
- Prometheus Scrape Status
|
|
|
|
### 2. Prometheus Metrics
|
|
**URL:** http://localhost:9090
|
|
|
|
**Key Queries:**
|
|
```promql
|
|
# Service health
|
|
up
|
|
|
|
# API request rate
|
|
rate(http_requests_total[5m])
|
|
|
|
# API response time (p95)
|
|
histogram_quantile(0.95, rate(http_request_duration_seconds_bucket[5m]))
|
|
|
|
# Redis cache hit rate
|
|
rate(redis_keyspace_hits_total[5m]) / (rate(redis_keyspace_hits_total[5m]) + rate(redis_keyspace_misses_total[5m]))
|
|
|
|
# Celery task count
|
|
celery_tasks_total
|
|
|
|
# Error rate
|
|
rate(http_requests_total{status=~"5.."}[5m])
|
|
```
|
|
|
|
### 3. Jaeger Tracing
|
|
**URL:** http://localhost:16686
|
|
|
|
**Features:**
|
|
- Distributed trace visualization
|
|
- Service dependency mapping
|
|
- Request flow analysis
|
|
- Performance bottleneck identification
|
|
|
|
**Services to Search:**
|
|
- `devserver.api.fastapi`
|
|
- `devserver.worker.celery`
|
|
|
|
### 4. cAdvisor Container Monitoring
|
|
**URL:** http://localhost:8080
|
|
|
|
**Metrics:**
|
|
- CPU usage per container
|
|
- Memory usage per container
|
|
- Network I/O
|
|
- Disk I/O
|
|
- Container health status
|
|
|
|
### 5. RabbitMQ Management
|
|
**URL:** http://localhost:15672
|
|
**Credentials:** guest / guest
|
|
|
|
**Metrics:**
|
|
- Queue lengths
|
|
- Message rates
|
|
- Consumer connections
|
|
- Memory usage
|
|
|
|
## 🔍 System Monitoring Script
|
|
|
|
Run the comprehensive system monitor:
|
|
|
|
```powershell
|
|
# Check system status
|
|
python system_monitor.py
|
|
```
|
|
|
|
**Output Includes:**
|
|
- ✓/✗ Health status for all services
|
|
- Response times for each service
|
|
- Prometheus metrics summary
|
|
- Redis cache statistics
|
|
- Recommendations for unhealthy services
|
|
- Quick links to all dashboards
|
|
|
|
## 🛠️ Deployment & Management
|
|
|
|
### Start All Services
|
|
```powershell
|
|
docker-compose up -d
|
|
```
|
|
|
|
### Check Service Status
|
|
```powershell
|
|
docker-compose ps
|
|
```
|
|
|
|
### View Service Logs
|
|
```powershell
|
|
# All services
|
|
docker-compose logs -f
|
|
|
|
# Specific service
|
|
docker-compose logs -f devserver
|
|
docker-compose logs -f prometheus
|
|
docker-compose logs -f grafana
|
|
```
|
|
|
|
### Restart Services
|
|
```powershell
|
|
# Restart all
|
|
docker-compose restart
|
|
|
|
# Restart specific services
|
|
docker-compose restart prometheus grafana otel-collector
|
|
```
|
|
|
|
### Stop Services
|
|
```powershell
|
|
docker-compose down
|
|
```
|
|
|
|
### Rebuild After Changes
|
|
```powershell
|
|
docker-compose down
|
|
docker-compose up -d --build
|
|
```
|
|
|
|
## 🎯 Monitoring Workflows
|
|
|
|
### 1. Check Overall System Health
|
|
1. Open http://localhost:3000 (Grafana)
|
|
2. Navigate to "Complete System Monitoring" dashboard
|
|
3. Check "Service Health Status" panel
|
|
4. Review error rates and response times
|
|
|
|
### 2. Debug Performance Issues
|
|
1. Open Grafana dashboard
|
|
2. Check "API Response Time" panel for slow endpoints
|
|
3. Open Jaeger (http://localhost:16686)
|
|
4. Search for traces with high duration
|
|
5. Analyze bottlenecks in trace waterfall
|
|
|
|
### 3. Monitor Cache Effectiveness
|
|
1. Open Grafana dashboard
|
|
2. Review "Redis Cache Hit Rate" panel
|
|
3. Check "Redis Memory Usage"
|
|
4. Access Redis stats via API: `GET http://localhost:8000/api/cache/stats`
|
|
|
|
### 4. Track Task Processing
|
|
1. Open Grafana dashboard
|
|
2. Monitor "Celery Task Queue Length"
|
|
3. Check "Celery Task Duration"
|
|
4. Review RabbitMQ management UI for queue depths
|
|
|
|
### 5. Investigate Errors
|
|
1. Check Grafana "Error Rate" panel
|
|
2. Review Prometheus for error metrics
|
|
3. Check logs via `docker-compose logs`
|
|
4. Use Jaeger to trace failed requests
|
|
|
|
## 📊 Key Metrics to Monitor
|
|
|
|
### Application Metrics
|
|
- **Request Rate:** Should be steady, spikes indicate traffic changes
|
|
- **Response Time (p95):** Should be < 500ms for API calls
|
|
- **Error Rate:** Should be < 1% under normal conditions
|
|
- **Cache Hit Rate:** Should be > 80% for cached endpoints
|
|
|
|
### System Metrics
|
|
- **CPU Usage:** Should be < 70% under normal load
|
|
- **Memory Usage:** Should be stable, not constantly growing
|
|
- **Disk I/O:** Should be low for cached operations
|
|
- **Network I/O:** Should correlate with request rates
|
|
|
|
### Service-Specific
|
|
- **Redis Memory:** Should be < configured max
|
|
- **RabbitMQ Queue Depth:** Should be near zero (fast processing)
|
|
- **Celery Tasks:** Should complete within expected timeframes
|
|
|
|
## 🚨 Troubleshooting
|
|
|
|
### Service is Down
|
|
```powershell
|
|
# Check service logs
|
|
docker-compose logs <service-name>
|
|
|
|
# Restart service
|
|
docker-compose restart <service-name>
|
|
|
|
# Check if port is in use
|
|
netstat -ano | findstr :<port>
|
|
```
|
|
|
|
### Grafana Data Source Issues
|
|
```powershell
|
|
# Restart Grafana
|
|
docker-compose restart grafana
|
|
|
|
# Check data source configuration
|
|
# Go to Grafana → Configuration → Data Sources
|
|
```
|
|
|
|
### OTEL Collector Not Working
|
|
```powershell
|
|
# Check configuration
|
|
docker-compose logs otel-collector
|
|
|
|
# Verify config file
|
|
cat monitoring/otel-collector-config.yaml
|
|
|
|
# Restart collector
|
|
docker-compose restart otel-collector
|
|
```
|
|
|
|
### Prometheus Not Scraping
|
|
```powershell
|
|
# Check Prometheus targets
|
|
# Open http://localhost:9090/targets
|
|
|
|
# Verify Prometheus config
|
|
cat monitoring/prometheus.yml
|
|
|
|
# Restart Prometheus
|
|
docker-compose restart prometheus
|
|
```
|
|
|
|
### High Memory Usage
|
|
```powershell
|
|
# Check container stats
|
|
docker stats
|
|
|
|
# Check specific service
|
|
docker-compose logs <service-name>
|
|
|
|
# Restart heavy services
|
|
docker-compose restart devserver worker
|
|
```
|
|
|
|
## 🔐 Security Considerations
|
|
|
|
### Change Default Passwords
|
|
Edit `docker-compose.yml`:
|
|
```yaml
|
|
environment:
|
|
GF_SECURITY_ADMIN_PASSWORD: your_secure_password
|
|
```
|
|
|
|
### Restrict External Access
|
|
For production, use:
|
|
```yaml
|
|
ports:
|
|
- "127.0.0.1:3000:3000" # Grafana only on localhost
|
|
```
|
|
|
|
### Enable HTTPS
|
|
Use a reverse proxy (nginx/traefik) with SSL certificates.
|
|
|
|
## 📝 Configuration Files
|
|
|
|
### Key Configuration Files
|
|
- `docker-compose.yml` - Service orchestration
|
|
- `monitoring/prometheus.yml` - Prometheus scrape config
|
|
- `monitoring/otel-collector-config.yaml` - OTEL collector config
|
|
- `monitoring/loki-config.yaml` - Loki config
|
|
- `monitoring/promtail-config.yaml` - Promtail config
|
|
- `monitoring/grafana/provisioning/` - Grafana auto-provisioning
|
|
|
|
## 🎓 Best Practices
|
|
|
|
1. **Regular Monitoring:** Check dashboards at least daily
|
|
2. **Set Alerts:** Configure Grafana alerts for critical metrics
|
|
3. **Log Retention:** Adjust log retention based on storage
|
|
4. **Backup Configs:** Version control all monitoring configs
|
|
5. **Resource Limits:** Set container resource limits in production
|
|
6. **Security:** Change default passwords and restrict ports
|
|
7. **Documentation:** Keep this guide updated with changes
|
|
|
|
## 📚 Additional Resources
|
|
|
|
- **Prometheus Documentation:** https://prometheus.io/docs/
|
|
- **Grafana Documentation:** https://grafana.com/docs/
|
|
- **Jaeger Documentation:** https://www.jaegertracing.io/docs/
|
|
- **OpenTelemetry:** https://opentelemetry.io/docs/
|
|
- **cAdvisor:** https://github.com/google/cadvisor
|
|
|
|
## 🆘 Support
|
|
|
|
For issues or questions:
|
|
1. Check service logs: `docker-compose logs <service>`
|
|
2. Run system monitor: `python system_monitor.py`
|
|
3. Review Grafana dashboards for anomalies
|
|
4. Check this documentation for troubleshooting steps
|
|
|
|
---
|
|
|
|
**Last Updated:** October 13, 2025
|
|
**Version:** 2.0.0
|