# DevServer Complete System Monitoring Guide

## 🎯 Overview

DevServer now includes a **comprehensive observability and monitoring stack** with automatic configuration. This guide covers all monitoring tools, dashboards, and troubleshooting procedures.

## 📊 Monitoring Architecture

```
┌─────────────────────────────────────────────────────────────┐
│                    Application Layer                         │
│  ┌──────────────┐  ┌──────────────┐  ┌──────────────┐      │
│  │ FastAPI App  │  │ Celery Worker│  │  Redis Cache │      │
│  │  (Port 8000) │  │              │  │  (Port 6379) │      │
│  └──────┬───────┘  └──────┬───────┘  └──────┬───────┘      │
│         │                  │                  │               │
└─────────┼──────────────────┼──────────────────┼──────────────┘
          │                  │                  │
          ▼                  ▼                  ▼
┌─────────────────────────────────────────────────────────────┐
│              OpenTelemetry Instrumentation                   │
│              (Traces, Metrics, Logs)                         │
└─────────────────────────────────────────────────────────────┘
          │                  │                  │
          ▼                  ▼                  ▼
┌─────────────────────────────────────────────────────────────┐
│                 Collection & Processing                      │
│  ┌──────────────┐  ┌──────────────┐  ┌──────────────┐      │
│  │OTEL Collector│  │  Prometheus  │  │    Loki      │      │
│  │              │  │  (Port 9090) │  │  (Port 3100) │      │
│  └──────┬───────┘  └──────┬───────┘  └──────┬───────┘      │
└─────────┼──────────────────┼──────────────────┼──────────────┘
          │                  │                  │
          ▼                  ▼                  ▼
┌─────────────────────────────────────────────────────────────┐
│                  Visualization Layer                         │
│  ┌──────────────┐  ┌──────────────┐  ┌──────────────┐      │
│  │   Grafana    │  │    Jaeger    │  │   cAdvisor   │      │
│  │  (Port 3000) │  │ (Port 16686) │  │  (Port 8080) │      │
│  └──────────────┘  └──────────────┘  └──────────────┘      │
└─────────────────────────────────────────────────────────────┘
```

## 🚀 Services Overview

### Core Application Services
- **FastAPI Backend** (`:8000`) - Main API with OpenTelemetry instrumentation
- **Celery Worker** - Async task processing with Fibonacci calculations
- **Redis** (`:6379`) - Cache layer with TTL-based expiration
- **RabbitMQ** (`:5672`, `:15672`) - Message broker for Celery

### Monitoring Services
- **Prometheus** (`:9090`) - Metrics collection and storage
- **Grafana** (`:3000`) - Visualization dashboards
- **Jaeger** (`:16686`) - Distributed tracing UI
- **Loki** (`:3100`) - Log aggregation system
- **Promtail** - Log shipping agent
- **OTEL Collector** - Telemetry data processing
- **cAdvisor** (`:8080`) - Container metrics
- **Node Exporter** (`:9100`) - System metrics

## 📈 Monitoring Dashboards

### 1. Grafana Dashboard (Primary)
**URL:** http://localhost:3000  
**Credentials:** admin / admin123

**Available Dashboards:**
- **Complete System Monitoring** - All services health and metrics
- **DevServer Observability** - Application-specific metrics

**Key Panels:**
- Service Health Status (Real-time UP/DOWN)
- API Request Rate & Response Time
- Redis Cache Hit Rate & Memory Usage
- Celery Task Queue & Duration
- RabbitMQ Message Rates
- HTTP Status Codes & Error Rates
- Container CPU & Memory Usage
- Prometheus Scrape Status

### 2. Prometheus Metrics
**URL:** http://localhost:9090

**Key Queries:**
```promql
# Service health
up

# API request rate
rate(http_requests_total[5m])

# API response time (p95)
histogram_quantile(0.95, rate(http_request_duration_seconds_bucket[5m]))

# Redis cache hit rate
rate(redis_keyspace_hits_total[5m]) / (rate(redis_keyspace_hits_total[5m]) + rate(redis_keyspace_misses_total[5m]))

# Celery task count
celery_tasks_total

# Error rate
rate(http_requests_total{status=~"5.."}[5m])
```

### 3. Jaeger Tracing
**URL:** http://localhost:16686

**Features:**
- Distributed trace visualization
- Service dependency mapping
- Request flow analysis
- Performance bottleneck identification

**Services to Search:**
- `devserver.api.fastapi`
- `devserver.worker.celery`

### 4. cAdvisor Container Monitoring
**URL:** http://localhost:8080

**Metrics:**
- CPU usage per container
- Memory usage per container
- Network I/O
- Disk I/O
- Container health status

### 5. RabbitMQ Management
**URL:** http://localhost:15672  
**Credentials:** guest / guest

**Metrics:**
- Queue lengths
- Message rates
- Consumer connections
- Memory usage

## 🔍 System Monitoring Script

Run the comprehensive system monitor:

```powershell
# Check system status
python system_monitor.py
```

**Output Includes:**
- ✓/✗ Health status for all services
- Response times for each service
- Prometheus metrics summary
- Redis cache statistics
- Recommendations for unhealthy services
- Quick links to all dashboards

## 🛠️ Deployment & Management

### Start All Services
```powershell
docker-compose up -d
```

### Check Service Status
```powershell
docker-compose ps
```

### View Service Logs
```powershell
# All services
docker-compose logs -f

# Specific service
docker-compose logs -f devserver
docker-compose logs -f prometheus
docker-compose logs -f grafana
```

### Restart Services
```powershell
# Restart all
docker-compose restart

# Restart specific services
docker-compose restart prometheus grafana otel-collector
```

### Stop Services
```powershell
docker-compose down
```

### Rebuild After Changes
```powershell
docker-compose down
docker-compose up -d --build
```

## 🎯 Monitoring Workflows

### 1. Check Overall System Health
1. Open http://localhost:3000 (Grafana)
2. Navigate to "Complete System Monitoring" dashboard
3. Check "Service Health Status" panel
4. Review error rates and response times

### 2. Debug Performance Issues
1. Open Grafana dashboard
2. Check "API Response Time" panel for slow endpoints
3. Open Jaeger (http://localhost:16686)
4. Search for traces with high duration
5. Analyze bottlenecks in trace waterfall

### 3. Monitor Cache Effectiveness
1. Open Grafana dashboard
2. Review "Redis Cache Hit Rate" panel
3. Check "Redis Memory Usage"
4. Access Redis stats via API: `GET http://localhost:8000/api/cache/stats`

### 4. Track Task Processing
1. Open Grafana dashboard
2. Monitor "Celery Task Queue Length"
3. Check "Celery Task Duration"
4. Review RabbitMQ management UI for queue depths

### 5. Investigate Errors
1. Check Grafana "Error Rate" panel
2. Review Prometheus for error metrics
3. Check logs via `docker-compose logs`
4. Use Jaeger to trace failed requests

## 📊 Key Metrics to Monitor

### Application Metrics
- **Request Rate:** Should be steady, spikes indicate traffic changes
- **Response Time (p95):** Should be < 500ms for API calls
- **Error Rate:** Should be < 1% under normal conditions
- **Cache Hit Rate:** Should be > 80% for cached endpoints

### System Metrics
- **CPU Usage:** Should be < 70% under normal load
- **Memory Usage:** Should be stable, not constantly growing
- **Disk I/O:** Should be low for cached operations
- **Network I/O:** Should correlate with request rates

### Service-Specific
- **Redis Memory:** Should be < configured max
- **RabbitMQ Queue Depth:** Should be near zero (fast processing)
- **Celery Tasks:** Should complete within expected timeframes

## 🚨 Troubleshooting

### Service is Down
```powershell
# Check service logs
docker-compose logs <service-name>

# Restart service
docker-compose restart <service-name>

# Check if port is in use
netstat -ano | findstr :<port>
```

### Grafana Data Source Issues
```powershell
# Restart Grafana
docker-compose restart grafana

# Check data source configuration
# Go to Grafana → Configuration → Data Sources
```

### OTEL Collector Not Working
```powershell
# Check configuration
docker-compose logs otel-collector

# Verify config file
cat monitoring/otel-collector-config.yaml

# Restart collector
docker-compose restart otel-collector
```

### Prometheus Not Scraping
```powershell
# Check Prometheus targets
# Open http://localhost:9090/targets

# Verify Prometheus config
cat monitoring/prometheus.yml

# Restart Prometheus
docker-compose restart prometheus
```

### High Memory Usage
```powershell
# Check container stats
docker stats

# Check specific service
docker-compose logs <service-name>

# Restart heavy services
docker-compose restart devserver worker
```

## 🔐 Security Considerations

### Change Default Passwords
Edit `docker-compose.yml`:
```yaml
environment:
  GF_SECURITY_ADMIN_PASSWORD: your_secure_password
```

### Restrict External Access
For production, use:
```yaml
ports:
  - "127.0.0.1:3000:3000"  # Grafana only on localhost
```

### Enable HTTPS
Use a reverse proxy (nginx/traefik) with SSL certificates.

## 📝 Configuration Files

### Key Configuration Files
- `docker-compose.yml` - Service orchestration
- `monitoring/prometheus.yml` - Prometheus scrape config
- `monitoring/otel-collector-config.yaml` - OTEL collector config
- `monitoring/loki-config.yaml` - Loki config
- `monitoring/promtail-config.yaml` - Promtail config
- `monitoring/grafana/provisioning/` - Grafana auto-provisioning

## 🎓 Best Practices

1. **Regular Monitoring:** Check dashboards at least daily
2. **Set Alerts:** Configure Grafana alerts for critical metrics
3. **Log Retention:** Adjust log retention based on storage
4. **Backup Configs:** Version control all monitoring configs
5. **Resource Limits:** Set container resource limits in production
6. **Security:** Change default passwords and restrict ports
7. **Documentation:** Keep this guide updated with changes

## 📚 Additional Resources

- **Prometheus Documentation:** https://prometheus.io/docs/
- **Grafana Documentation:** https://grafana.com/docs/
- **Jaeger Documentation:** https://www.jaegertracing.io/docs/
- **OpenTelemetry:** https://opentelemetry.io/docs/
- **cAdvisor:** https://github.com/google/cadvisor

## 🆘 Support

For issues or questions:
1. Check service logs: `docker-compose logs <service>`
2. Run system monitor: `python system_monitor.py`
3. Review Grafana dashboards for anomalies
4. Check this documentation for troubleshooting steps

---

**Last Updated:** October 13, 2025  
**Version:** 2.0.0