DevServer/docs/MONITORING_GUIDE.md

DevServer Complete System Monitoring Guide

🎯 Overview

DevServer now includes a comprehensive observability and monitoring stack with automatic configuration. This guide covers all monitoring tools, dashboards, and troubleshooting procedures.

📊 Monitoring Architecture

┌─────────────────────────────────────────────────────────────┐
│                    Application Layer                         │
│  ┌──────────────┐  ┌──────────────┐  ┌──────────────┐      │
│  │ FastAPI App  │  │ Celery Worker│  │  Redis Cache │      │
│  │  (Port 8000) │  │              │  │  (Port 6379) │      │
│  └──────┬───────┘  └──────┬───────┘  └──────┬───────┘      │
│         │                  │                  │               │
└─────────┼──────────────────┼──────────────────┼──────────────┘
          │                  │                  │
          ▼                  ▼                  ▼
┌─────────────────────────────────────────────────────────────┐
│              OpenTelemetry Instrumentation                   │
│              (Traces, Metrics, Logs)                         │
└─────────────────────────────────────────────────────────────┘
          │                  │                  │
          ▼                  ▼                  ▼
┌─────────────────────────────────────────────────────────────┐
│                 Collection & Processing                      │
│  ┌──────────────┐  ┌──────────────┐  ┌──────────────┐      │
│  │OTEL Collector│  │  Prometheus  │  │    Loki      │      │
│  │              │  │  (Port 9090) │  │  (Port 3100) │      │
│  └──────┬───────┘  └──────┬───────┘  └──────┬───────┘      │
└─────────┼──────────────────┼──────────────────┼──────────────┘
          │                  │                  │
          ▼                  ▼                  ▼
┌─────────────────────────────────────────────────────────────┐
│                  Visualization Layer                         │
│  ┌──────────────┐  ┌──────────────┐  ┌──────────────┐      │
│  │   Grafana    │  │    Jaeger    │  │   cAdvisor   │      │
│  │  (Port 3000) │  │ (Port 16686) │  │  (Port 8080) │      │
│  └──────────────┘  └──────────────┘  └──────────────┘      │
└─────────────────────────────────────────────────────────────┘

🚀 Services Overview

Core Application Services

• FastAPI Backend (:8000) - Main API with OpenTelemetry instrumentation
• Celery Worker - Async task processing with Fibonacci calculations
• Redis (:6379) - Cache layer with TTL-based expiration
• RabbitMQ (:5672, :15672) - Message broker for Celery

Monitoring Services

• Prometheus (:9090) - Metrics collection and storage
• Grafana (:3000) - Visualization dashboards
• Jaeger (:16686) - Distributed tracing UI
• Loki (:3100) - Log aggregation system
• Promtail - Log shipping agent
• OTEL Collector - Telemetry data processing
• cAdvisor (:8080) - Container metrics
• Node Exporter (:9100) - System metrics

📈 Monitoring Dashboards

1. Grafana Dashboard (Primary)

URL: http://localhost:3000
Credentials: admin / admin123

Available Dashboards:

• Complete System Monitoring - All services health and metrics
• DevServer Observability - Application-specific metrics

Key Panels:

• Service Health Status (Real-time UP/DOWN)
• API Request Rate & Response Time
• Redis Cache Hit Rate & Memory Usage
• Celery Task Queue & Duration
• RabbitMQ Message Rates
• HTTP Status Codes & Error Rates
• Container CPU & Memory Usage
• Prometheus Scrape Status

2. Prometheus Metrics

URL: http://localhost:9090

Key Queries:

# Service health
up

# API request rate
rate(http_requests_total[5m])

# API response time (p95)
histogram_quantile(0.95, rate(http_request_duration_seconds_bucket[5m]))

# Redis cache hit rate
rate(redis_keyspace_hits_total[5m]) / (rate(redis_keyspace_hits_total[5m]) + rate(redis_keyspace_misses_total[5m]))

# Celery task count
celery_tasks_total

# Error rate
rate(http_requests_total{status=~"5.."}[5m])

3. Jaeger Tracing

URL: http://localhost:16686

Features:

• Distributed trace visualization
• Service dependency mapping
• Request flow analysis
• Performance bottleneck identification

Services to Search:

• devserver.api.fastapi
• devserver.worker.celery

4. cAdvisor Container Monitoring

URL: http://localhost:8080

Metrics:

• CPU usage per container
• Memory usage per container
• Network I/O
• Disk I/O
• Container health status

5. RabbitMQ Management

URL: http://localhost:15672
Credentials: guest / guest

Metrics:

• Queue lengths
• Message rates
• Consumer connections
• Memory usage

🔍 System Monitoring Script

Run the comprehensive system monitor:

# Check system status
python system_monitor.py

Output Includes:

• ✓/✗ Health status for all services
• Response times for each service
• Prometheus metrics summary
• Redis cache statistics
• Recommendations for unhealthy services
• Quick links to all dashboards

🛠️ Deployment & Management

Start All Services

docker-compose up -d

Check Service Status

docker-compose ps

View Service Logs

# All services
docker-compose logs -f

# Specific service
docker-compose logs -f devserver
docker-compose logs -f prometheus
docker-compose logs -f grafana

Restart Services

# Restart all
docker-compose restart

# Restart specific services
docker-compose restart prometheus grafana otel-collector

Stop Services

docker-compose down

Rebuild After Changes

docker-compose down
docker-compose up -d --build

🎯 Monitoring Workflows

1. Check Overall System Health

1. Open http://localhost:3000 (Grafana)
2. Navigate to "Complete System Monitoring" dashboard
3. Check "Service Health Status" panel
4. Review error rates and response times

2. Debug Performance Issues

1. Open Grafana dashboard
2. Check "API Response Time" panel for slow endpoints
3. Open Jaeger (http://localhost:16686)
4. Search for traces with high duration
5. Analyze bottlenecks in trace waterfall

3. Monitor Cache Effectiveness

1. Open Grafana dashboard
2. Review "Redis Cache Hit Rate" panel
3. Check "Redis Memory Usage"
4. Access Redis stats via API: GET http://localhost:8000/api/cache/stats

4. Track Task Processing

1. Open Grafana dashboard
2. Monitor "Celery Task Queue Length"
3. Check "Celery Task Duration"
4. Review RabbitMQ management UI for queue depths

5. Investigate Errors

1. Check Grafana "Error Rate" panel
2. Review Prometheus for error metrics
3. Check logs via docker-compose logs
4. Use Jaeger to trace failed requests

📊 Key Metrics to Monitor

Application Metrics

• Request Rate: Should be steady, spikes indicate traffic changes
• Response Time (p95): Should be < 500ms for API calls
• Error Rate: Should be < 1% under normal conditions
• Cache Hit Rate: Should be > 80% for cached endpoints

System Metrics

• CPU Usage: Should be < 70% under normal load
• Memory Usage: Should be stable, not constantly growing
• Disk I/O: Should be low for cached operations
• Network I/O: Should correlate with request rates

Service-Specific

• Redis Memory: Should be < configured max
• RabbitMQ Queue Depth: Should be near zero (fast processing)
• Celery Tasks: Should complete within expected timeframes

🚨 Troubleshooting

Service is Down

# Check service logs
docker-compose logs <service-name>

# Restart service
docker-compose restart <service-name>

# Check if port is in use
netstat -ano | findstr :<port>

Grafana Data Source Issues

# Restart Grafana
docker-compose restart grafana

# Check data source configuration
# Go to Grafana → Configuration → Data Sources

OTEL Collector Not Working

# Check configuration
docker-compose logs otel-collector

# Verify config file
cat monitoring/otel-collector-config.yaml

# Restart collector
docker-compose restart otel-collector

Prometheus Not Scraping

# Check Prometheus targets
# Open http://localhost:9090/targets

# Verify Prometheus config
cat monitoring/prometheus.yml

# Restart Prometheus
docker-compose restart prometheus

High Memory Usage

# Check container stats
docker stats

# Check specific service
docker-compose logs <service-name>

# Restart heavy services
docker-compose restart devserver worker

🔐 Security Considerations

Change Default Passwords

Edit docker-compose.yml:

environment:
  GF_SECURITY_ADMIN_PASSWORD: your_secure_password

Restrict External Access

For production, use:

ports:
  - "127.0.0.1:3000:3000"  # Grafana only on localhost

Enable HTTPS

Use a reverse proxy (nginx/traefik) with SSL certificates.

📝 Configuration Files

Key Configuration Files

• docker-compose.yml - Service orchestration
• monitoring/prometheus.yml - Prometheus scrape config
• monitoring/otel-collector-config.yaml - OTEL collector config
• monitoring/loki-config.yaml - Loki config
• monitoring/promtail-config.yaml - Promtail config
• monitoring/grafana/provisioning/ - Grafana auto-provisioning

🎓 Best Practices

1. Regular Monitoring: Check dashboards at least daily
2. Set Alerts: Configure Grafana alerts for critical metrics
3. Log Retention: Adjust log retention based on storage
4. Backup Configs: Version control all monitoring configs
5. Resource Limits: Set container resource limits in production
6. Security: Change default passwords and restrict ports
7. Documentation: Keep this guide updated with changes

📚 Additional Resources

• Prometheus Documentation: https://prometheus.io/docs/
• Grafana Documentation: https://grafana.com/docs/
• Jaeger Documentation: https://www.jaegertracing.io/docs/
• OpenTelemetry: https://opentelemetry.io/docs/
• cAdvisor: https://github.com/google/cadvisor

🆘 Support

For issues or questions:

1. Check service logs: docker-compose logs <service>
2. Run system monitor: python system_monitor.py
3. Review Grafana dashboards for anomalies
4. Check this documentation for troubleshooting steps

---

Last Updated: October 13, 2025
Version: 2.0.0