DevServer/docs/QUICK_REFERENCE.md

🚀 DevServer Monitoring - Quick Reference Card

📊 Monitoring Dashboards

Service	URL	Credentials	Purpose
Grafana	http://localhost:3000	admin / admin123	Main dashboards & visualization
Prometheus	http://localhost:9090	None	Metrics & queries
Prometheus Targets	http://localhost:9090/targets	None	Check scrape status
Jaeger	http://localhost:16686	None	Distributed tracing
cAdvisor	http://localhost:8080	None	Container metrics
RabbitMQ	http://localhost:15672	guest / guest	Queue management
FastAPI	http://localhost:8000	None	API endpoints
FastAPI Docs	http://localhost:8000/docs	None	Interactive API docs

🎯 Active Prometheus Targets (7 services)

✅ Working (5/7)

• ✅ prometheus (self-monitoring)
• ✅ rabbitmq (message queue)
• ✅ loki (logs)
• ✅ cadvisor (containers)
• ✅ jaeger (traces)

⏳ Initializing (2/7)

• ⏳ fastapi-app (starting up - may take 30-60s)
• ⏳ otel-collector (initializing telemetry pipeline)

🔧 Common Commands

# Check all services
docker-compose ps

# View logs
docker-compose logs -f devserver
docker-compose logs -f prometheus

# Restart services
docker-compose restart prometheus grafana

# System health check
uv run python system_monitor.py

# Test API
Invoke-WebRequest -Uri "http://localhost:8000/health" -Method GET
Invoke-WebRequest -Uri "http://localhost:8000/metrics" -Method GET

# Generate test data
Invoke-WebRequest -Uri "http://localhost:8000/api/dice/roll" -Method GET
Invoke-WebRequest -Uri "http://localhost:8000/api/tasks/fibonacci?n=15" -Method POST

📈 Key Metrics to Monitor

Metric	Normal Range	Alert If
API Response Time (p95)	< 500ms	> 1000ms
Error Rate	< 1%	> 5%
Cache Hit Rate	> 80%	< 50%
CPU Usage	< 70%	> 90%
Memory Usage	Stable	Constantly growing
Queue Depth	Near 0	> 100

🎯 Grafana Dashboards

1. Complete System Monitoring

   • Service health status
   • API metrics
   • Cache performance
   • Container resources

2. DevServer Observability

   • Request rates
   • Response times
   • Error tracking

🔍 Troubleshooting Quick Fixes

Issue	Solution
Service down	docker-compose restart <service>
Slow API	Check Jaeger traces at http://localhost:16686
High errors	Check logs: docker-compose logs devserver
Metrics not updating	Restart Prometheus: docker-compose restart prometheus
Grafana not loading	docker-compose restart grafana
Target shows "down"	Wait 60s or check: docker-compose logs <service>

📦 Services Summary

Total: 11 containers
Core App: 4 services (FastAPI, Celery, Redis, RabbitMQ)
Monitoring: 7 services (Prometheus, Grafana, Jaeger, Loki, Promtail, OTEL, cAdvisor)

✨ Features

• ✅ Automatic OpenTelemetry tracing
• ✅ Redis caching with TTL
• ✅ Real-time metrics (15s intervals)
• ✅ Centralized logging
• ✅ Container monitoring
• ✅ Distributed tracing
• ✅ Auto-provisioned dashboards
• ✅ Zero manual configuration

🎊 Status: FULLY OPERATIONAL

Last Updated: October 13, 2025
All Services: ✅ RUNNING
Monitoring: ✅ ACTIVE
Dashboards: ✅ READY