Initial commit: Complete DevServer monitoring framework
- FastAPI application with OpenTelemetry instrumentation - Redis caching and Celery task processing - Docker Compose setup with monitoring stack - Grafana dashboards and Prometheus metrics - Jaeger distributed tracing - Comprehensive documentation in docs/ folder - Monitoring configuration and troubleshooting guides
This commit is contained in:
Hardik Sankhla
330
docs/MONITORING_COMPLETE.md
Normal file
330
docs/MONITORING_COMPLETE.md
Normal file
@@ -0,0 +1,330 @@
|
||||
# ✅ Monitoring System - Complete and Operational
|
||||
|
||||
**Date:** October 13, 2025
|
||||
**Status:** ALL SYSTEMS OPERATIONAL ✅
|
||||
|
||||
---
|
||||
|
||||
## 🎯 Current Status
|
||||
|
||||
### All Prometheus Targets: 7/7 UP ✅
|
||||
|
||||
| Target | Endpoint | Status | Metrics |
|
||||
|--------|----------|--------|---------|
|
||||
| **prometheus** | localhost:9090 | ✅ UP | Self-monitoring |
|
||||
| **fastapi-app** | devserver:8000 | ✅ UP | Application metrics (6.4KB) |
|
||||
| **rabbitmq** | rabbitmq:15692 | ✅ UP | Message queue metrics |
|
||||
| **loki** | loki:3100 | ✅ UP | Log aggregator metrics |
|
||||
| **cadvisor** | cadvisor:8080 | ✅ UP | Container metrics |
|
||||
| **otel-collector** | otel-collector:8889 | ✅ UP | **OTEL metrics (41KB)** |
|
||||
| **jaeger** | jaeger:14269 | ✅ UP | Tracing metrics |
|
||||
|
||||
---
|
||||
|
||||
## 🔧 Issues Resolved
|
||||
|
||||
### 1. OTEL Collector Metrics Exposure ✅
|
||||
**Problem:** OTEL collector wasn't exposing its own metrics for Prometheus to scrape
|
||||
- **Error:** `connection refused on port 8888`
|
||||
- **Root Cause:** Missing Prometheus exporter in OTEL collector configuration
|
||||
- **Solution:**
|
||||
- Added `prometheus` exporter to `otel-collector-config.yaml` on port 8889
|
||||
- Added exporter to metrics pipeline
|
||||
- Updated Prometheus scrape config to use port 8889
|
||||
- **Result:** 41KB of metrics now exposed and successfully scraped
|
||||
|
||||
### 2. Prometheus Remote Write Receiver ✅
|
||||
**Problem:** OTEL collector couldn't push metrics to Prometheus via remote write
|
||||
- **Error:** `404 Not Found: remote write receiver needs to be enabled`
|
||||
- **Root Cause:** Prometheus missing `--web.enable-remote-write-receiver` flag
|
||||
- **Solution:**
|
||||
- Added flag to Prometheus command in docker-compose.yml
|
||||
- Force recreated container (restart wasn't enough)
|
||||
- **Result:** Remote write working, no more 404 errors
|
||||
|
||||
### 3. OTEL Collector Configuration Validation ✅
|
||||
**Problem:** Configuration had invalid telemetry.metrics.address key
|
||||
- **Error:** `'service.telemetry.metrics' decoding failed: invalid keys: address`
|
||||
- **Root Cause:** OTEL Collector v0.137.0 doesn't support `address` in telemetry.metrics
|
||||
- **Solution:**
|
||||
- Removed invalid `address` key
|
||||
- Used Prometheus exporter instead for metrics exposure
|
||||
- Simplified telemetry config to just `level: detailed`
|
||||
- **Result:** OTEL collector starts without configuration errors
|
||||
|
||||
---
|
||||
|
||||
## 📊 Complete Monitoring Stack (11 Services)
|
||||
|
||||
### Core Application Services
|
||||
1. **devserver (FastAPI)** - Main backend application
|
||||
- Port: 8000
|
||||
- Metrics: ✅ `/metrics` endpoint
|
||||
- OTLP: Sending traces to OTEL collector
|
||||
|
||||
2. **celery-worker** - Background task processor
|
||||
- Broker: RabbitMQ
|
||||
- Backend: Redis
|
||||
- Monitoring: Via RabbitMQ metrics
|
||||
|
||||
3. **redis** - Cache and Celery backend
|
||||
- Port: 6379
|
||||
- Monitoring: Connection stats via app
|
||||
|
||||
4. **rabbitmq** - Message broker
|
||||
- Ports: 5672 (AMQP), 15672 (UI), 15692 (metrics)
|
||||
- Metrics: ✅ Prometheus endpoint
|
||||
- UI: http://localhost:15672 (guest/guest)
|
||||
|
||||
### Observability Services
|
||||
|
||||
5. **Prometheus** - Metrics collection and storage
|
||||
- Port: 9090
|
||||
- UI: http://localhost:9090
|
||||
- Config: 7 active scrape targets
|
||||
- Features: Remote write enabled, lifecycle API, admin API
|
||||
- Storage: 200h retention
|
||||
|
||||
6. **Grafana** - Metrics visualization
|
||||
- Port: 3000
|
||||
- UI: http://localhost:3000 (admin/admin)
|
||||
- Datasources: Prometheus, Loki, Jaeger (auto-provisioned)
|
||||
- Dashboards: 2 pre-configured
|
||||
|
||||
7. **OTEL Collector** - OpenTelemetry collector
|
||||
- Version: 0.137.0 (contrib)
|
||||
- Ports:
|
||||
- 4317: OTLP gRPC receiver
|
||||
- 4318: OTLP HTTP receiver
|
||||
- 8889: Prometheus metrics exporter ✅
|
||||
- 13133: Health check
|
||||
- 1888: pprof profiling
|
||||
- 55679: zpages extension
|
||||
- Pipelines:
|
||||
- Traces → Debug + Jaeger
|
||||
- Metrics → **Prometheus exporter** + Remote write + Debug
|
||||
- Logs → File + Debug
|
||||
|
||||
8. **Jaeger** - Distributed tracing
|
||||
- Ports: 16686 (UI), 14268 (traces), 14269 (metrics)
|
||||
- UI: http://localhost:16686
|
||||
- Backend: Memory storage
|
||||
|
||||
9. **Loki** - Log aggregation
|
||||
- Port: 3100
|
||||
- Config: Local storage
|
||||
- Metrics: ✅ Exposed on /metrics
|
||||
|
||||
10. **Promtail** - Log shipper
|
||||
- Source: ./logs directory
|
||||
- Target: Loki
|
||||
- Format: JSON logs
|
||||
|
||||
11. **cAdvisor** - Container monitoring
|
||||
- Port: 8080
|
||||
- Metrics: ✅ Container CPU, memory, network, disk
|
||||
- UI: http://localhost:8080
|
||||
|
||||
---
|
||||
|
||||
## 🔗 Quick Access URLs
|
||||
|
||||
| Service | URL | Credentials |
|
||||
|---------|-----|-------------|
|
||||
| **FastAPI Swagger** | http://localhost:8000/docs | - |
|
||||
| **FastAPI Metrics** | http://localhost:8000/metrics | - |
|
||||
| **Prometheus** | http://localhost:9090 | - |
|
||||
| **Prometheus Targets** | http://localhost:9090/targets | - |
|
||||
| **Grafana** | http://localhost:3000 | admin/admin |
|
||||
| **Jaeger UI** | http://localhost:16686 | - |
|
||||
| **RabbitMQ UI** | http://localhost:15672 | guest/guest |
|
||||
| **cAdvisor** | http://localhost:8080 | - |
|
||||
| **OTEL Health** | http://localhost:13133 | - |
|
||||
| **OTEL Metrics** | http://localhost:8889/metrics | - |
|
||||
|
||||
---
|
||||
|
||||
## 📈 Data Flow
|
||||
|
||||
### Traces
|
||||
```
|
||||
FastAPI → OTLP (4317) → OTEL Collector → Jaeger (14268)
|
||||
↓
|
||||
Debug Logger
|
||||
```
|
||||
|
||||
### Metrics
|
||||
```
|
||||
FastAPI → /metrics → Prometheus (scrape every 5s)
|
||||
↓
|
||||
Grafana (query)
|
||||
|
||||
OTEL Collector → Prometheus Exporter (8889) → Prometheus (scrape every 15s)
|
||||
↓
|
||||
Remote Write → Prometheus (9090)
|
||||
```
|
||||
|
||||
### Logs
|
||||
```
|
||||
Application → ./logs/*.json → Promtail → Loki
|
||||
↓
|
||||
Grafana (query)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🛠️ Configuration Files
|
||||
|
||||
### Key Files Updated
|
||||
1. **docker-compose.yml**
|
||||
- Added Prometheus remote write receiver flag
|
||||
- Added environment variable to OTEL collector
|
||||
- All 11 services properly configured
|
||||
|
||||
2. **monitoring/otel-collector-config.yaml**
|
||||
- Added `prometheus` exporter on port 8889
|
||||
- Removed invalid `telemetry.metrics.address` key
|
||||
- Added prometheus exporter to metrics pipeline
|
||||
- Simplified telemetry configuration
|
||||
|
||||
3. **monitoring/prometheus.yml**
|
||||
- Updated OTEL collector target from port 8888 → 8889
|
||||
- 7 active scrape jobs configured
|
||||
- Different intervals per job (5s-15s)
|
||||
|
||||
---
|
||||
|
||||
## ✅ Verification Commands
|
||||
|
||||
### Check All Container Status
|
||||
```powershell
|
||||
cd d:\GNPL-Private\DevServer
|
||||
docker-compose ps
|
||||
```
|
||||
|
||||
### Check Prometheus Targets
|
||||
```powershell
|
||||
$response = Invoke-WebRequest -Uri "http://localhost:9090/api/v1/targets" -Method GET
|
||||
($response.Content | ConvertFrom-Json).data.activeTargets |
|
||||
Select-Object @{N='Job';E={$_.labels.job}}, @{N='State';E={$_.health}}
|
||||
```
|
||||
|
||||
### Test All Metrics Endpoints
|
||||
```powershell
|
||||
# FastAPI
|
||||
Invoke-WebRequest -Uri "http://localhost:8000/metrics" | Select StatusCode
|
||||
|
||||
# OTEL Collector
|
||||
Invoke-WebRequest -Uri "http://localhost:8889/metrics" | Select StatusCode
|
||||
|
||||
# RabbitMQ
|
||||
Invoke-WebRequest -Uri "http://localhost:15692/metrics" | Select StatusCode
|
||||
|
||||
# cAdvisor
|
||||
Invoke-WebRequest -Uri "http://localhost:8080/metrics" | Select StatusCode
|
||||
```
|
||||
|
||||
### Check OTEL Collector Logs
|
||||
```powershell
|
||||
docker-compose logs --tail=50 otel-collector
|
||||
```
|
||||
|
||||
### Check Prometheus Logs
|
||||
```powershell
|
||||
docker-compose logs --tail=50 prometheus
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 📝 Best Practices Implemented
|
||||
|
||||
✅ **Metrics Collection**
|
||||
- Multiple scrape intervals optimized per service
|
||||
- Proper labels for service identification
|
||||
- Health checks on all exporters
|
||||
|
||||
✅ **Data Retention**
|
||||
- Prometheus: 200 hours
|
||||
- Loki: Log rotation enabled
|
||||
- OTEL: File rotation for logs (100MB, 3 days, 3 backups)
|
||||
|
||||
✅ **Resource Management**
|
||||
- OTEL memory limiter: 512MB
|
||||
- Docker log rotation: 10MB per file, 3 files max
|
||||
- Batch processing: 1s timeout, 1024 batch size
|
||||
|
||||
✅ **Security**
|
||||
- Grafana: Sign-up disabled
|
||||
- RabbitMQ: Default credentials (change in production)
|
||||
- Network isolation: All services on devserver_network
|
||||
|
||||
✅ **High Availability**
|
||||
- Health check endpoints on all services
|
||||
- Proper dependency ordering in docker-compose
|
||||
- Graceful shutdown handling
|
||||
|
||||
---
|
||||
|
||||
## 🎓 Key Learnings
|
||||
|
||||
1. **OTEL Collector Metrics Exposure**
|
||||
- Don't rely on `telemetry.metrics.address` - use Prometheus exporter
|
||||
- Port 8888 is for internal metrics, use 8889 for Prometheus scrape
|
||||
- Always add exporters to the pipeline
|
||||
|
||||
2. **Prometheus Configuration**
|
||||
- Remote write receiver must be explicitly enabled
|
||||
- Container restart ≠ container recreate (use --force-recreate)
|
||||
- Different scrape intervals for different workloads
|
||||
|
||||
3. **Docker Compose Best Practices**
|
||||
- Version attribute is obsolete (remove it)
|
||||
- Use container names for DNS resolution
|
||||
- Always use networks for service discovery
|
||||
|
||||
4. **Debugging Approach**
|
||||
- Check logs first: `docker-compose logs <service>`
|
||||
- Verify endpoints: `Invoke-WebRequest` for HTTP services
|
||||
- Check configuration: `docker inspect` for runtime settings
|
||||
|
||||
---
|
||||
|
||||
## 📚 Documentation
|
||||
|
||||
### Related Files
|
||||
- `DEPLOYMENT_STATUS.md` - Initial deployment summary
|
||||
- `MONITORING_GUIDE.md` - Detailed monitoring guide
|
||||
- `QUICK_REFERENCE.md` - Quick reference card
|
||||
- `system_monitor.py` - Automated health check script
|
||||
|
||||
### External Resources
|
||||
- [OTEL Collector Documentation](https://opentelemetry.io/docs/collector/)
|
||||
- [Prometheus Remote Write](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#remote_write)
|
||||
- [Grafana Provisioning](https://grafana.com/docs/grafana/latest/administration/provisioning/)
|
||||
|
||||
---
|
||||
|
||||
## 🚀 Next Steps
|
||||
|
||||
### Recommended Actions
|
||||
1. ✅ **Current State:** All monitoring is working
|
||||
2. 🔄 **Customize Dashboards:** Add application-specific metrics to Grafana
|
||||
3. 📊 **Set Alerts:** Configure Prometheus alerting rules
|
||||
4. 🔐 **Secure Credentials:** Change default passwords (RabbitMQ, Grafana)
|
||||
5. 📈 **Optimize:** Adjust scrape intervals based on traffic patterns
|
||||
6. 🧪 **Load Testing:** Verify monitoring under high load
|
||||
7. 📝 **Document:** Add application-specific metrics documentation
|
||||
|
||||
### Optional Enhancements
|
||||
- Add Alertmanager for notifications
|
||||
- Implement custom Grafana dashboards
|
||||
- Add more detailed business metrics
|
||||
- Set up external log shipping
|
||||
- Implement distributed tracing best practices
|
||||
- Add service mesh (Istio/Linkerd) for advanced observability
|
||||
|
||||
---
|
||||
|
||||
**System Status:** PRODUCTION READY ✅
|
||||
**Last Verified:** October 13, 2025
|
||||
**Verification:** All 7 Prometheus targets UP, no errors in logs
|
||||
Reference in New Issue
Block a user