Initial commit: Complete DevServer monitoring framework
- FastAPI application with OpenTelemetry instrumentation - Redis caching and Celery task processing - Docker Compose setup with monitoring stack - Grafana dashboards and Prometheus metrics - Jaeger distributed tracing - Comprehensive documentation in docs/ folder - Monitoring configuration and troubleshooting guides
This commit is contained in:
Hardik Sankhla
340
docs/DOCKER_CLEAN_REBUILD.md
Normal file
340
docs/DOCKER_CLEAN_REBUILD.md
Normal file
@@ -0,0 +1,340 @@
|
||||
# ✅ Docker System - Clean Rebuild Complete
|
||||
|
||||
**Date:** October 13, 2025
|
||||
**Operation:** Full Docker cleanup and fresh rebuild
|
||||
**Status:** ALL SERVICES OPERATIONAL ✅
|
||||
|
||||
---
|
||||
|
||||
## 🧹 Cleanup Operations Performed
|
||||
|
||||
### 1. Stopped All Containers ✅
|
||||
```bash
|
||||
docker-compose down
|
||||
```
|
||||
- Removed all 11 running containers
|
||||
- Disconnected from devserver_network
|
||||
|
||||
### 2. Removed All Volumes ✅
|
||||
```bash
|
||||
docker-compose down -v
|
||||
```
|
||||
**Volumes Deleted:**
|
||||
- `devserver_prometheus_data` - Metrics storage
|
||||
- `devserver_redis_data` - Cache data
|
||||
- `devserver_grafana_data` - Dashboards & datasources
|
||||
- `devserver_loki_data` - Log storage
|
||||
|
||||
**Result:** Fresh start with no persistent data
|
||||
|
||||
### 3. System Cleanup ✅
|
||||
```bash
|
||||
docker system prune -f
|
||||
```
|
||||
**Cleaned Up:**
|
||||
- 9 stopped containers
|
||||
- 1 unused network
|
||||
- 70+ build cache objects
|
||||
- **Total Space Reclaimed: 4.387 GB** 🎯
|
||||
|
||||
---
|
||||
|
||||
## 🚀 Rebuild and Startup
|
||||
|
||||
### Services Started (Fresh)
|
||||
```bash
|
||||
docker-compose up -d
|
||||
```
|
||||
|
||||
**All 11 containers started successfully:**
|
||||
|
||||
| Service | Status | Health | Port |
|
||||
|---------|--------|--------|------|
|
||||
| **devserver (FastAPI)** | ✅ Running | HTTP 200 | 8000 |
|
||||
| **celery_worker** | ✅ Running | Active | - |
|
||||
| **redis** | ✅ Running | Healthy | 6379 |
|
||||
| **rabbitmq** | ✅ Running | Active | 5672, 15672 |
|
||||
| **prometheus** | ✅ Running | Active | 9090 |
|
||||
| **grafana** | ✅ Running | HTTP 200 | 3000 |
|
||||
| **otel-collector** | ✅ Running | Active | 4317, 8889 |
|
||||
| **jaeger** | ✅ Running | Active | 16686 |
|
||||
| **loki** | ✅ Running | Active | 3100 |
|
||||
| **promtail** | ✅ Running | Active | - |
|
||||
| **cadvisor** | ✅ Running | Healthy | 8080 |
|
||||
|
||||
---
|
||||
|
||||
## 📊 Verification Results
|
||||
|
||||
### FastAPI Backend ✅
|
||||
```
|
||||
Status: 200 OK
|
||||
Endpoint: http://localhost:8000/health
|
||||
```
|
||||
|
||||
### Prometheus Monitoring ✅
|
||||
```
|
||||
Status: All 7/7 targets UP
|
||||
Targets:
|
||||
- cadvisor: up
|
||||
- fastapi-app: up
|
||||
- jaeger: up
|
||||
- loki: up
|
||||
- otel-collector: up
|
||||
- prometheus: up
|
||||
- rabbitmq: up
|
||||
```
|
||||
|
||||
### Grafana Dashboard ✅
|
||||
```
|
||||
Status: 200 OK
|
||||
Endpoint: http://localhost:3000/api/health
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🎯 What's Fresh
|
||||
|
||||
### New Configuration State
|
||||
1. **No Old Data** - All volumes wiped clean
|
||||
2. **Fresh Grafana** - Will need to log in again (admin/admin)
|
||||
3. **Empty Prometheus** - No historical metrics yet
|
||||
4. **Clean Loki** - No old logs
|
||||
5. **Reset Redis** - Empty cache
|
||||
|
||||
### Retained Configuration
|
||||
✅ **All configuration files preserved:**
|
||||
- `docker-compose.yml` - Service definitions
|
||||
- `monitoring/prometheus.yml` - Scrape configs
|
||||
- `monitoring/otel-collector-config.yaml` - OTEL pipeline
|
||||
- `monitoring/grafana/provisioning/` - Auto-provisioned datasources & dashboards
|
||||
- All environment variables and settings
|
||||
|
||||
---
|
||||
|
||||
## 🔗 Access URLs (Fresh Start)
|
||||
|
||||
| Service | URL | Credentials | Status |
|
||||
|---------|-----|-------------|--------|
|
||||
| **FastAPI Docs** | http://localhost:8000/docs | - | ✅ Ready |
|
||||
| **FastAPI Metrics** | http://localhost:8000/metrics | - | ✅ Ready |
|
||||
| **Prometheus** | http://localhost:9090 | - | ✅ Ready |
|
||||
| **Prometheus Targets** | http://localhost:9090/targets | - | ✅ 7/7 UP |
|
||||
| **Grafana** | http://localhost:3000 | admin/admin | ✅ Ready |
|
||||
| **Jaeger UI** | http://localhost:16686 | - | ✅ Ready |
|
||||
| **RabbitMQ UI** | http://localhost:15672 | guest/guest | ✅ Ready |
|
||||
| **cAdvisor** | http://localhost:8080 | - | ✅ Ready |
|
||||
| **OTEL Health** | http://localhost:13133 | - | ✅ Ready |
|
||||
| **OTEL Metrics** | http://localhost:8889/metrics | - | ✅ Ready |
|
||||
|
||||
---
|
||||
|
||||
## 📈 Monitoring Dashboard
|
||||
|
||||
### Grafana Setup
|
||||
**First-time login:**
|
||||
- URL: http://localhost:3000
|
||||
- Username: `admin`
|
||||
- Password: `admin`
|
||||
- You'll be prompted to change password on first login
|
||||
|
||||
**Auto-provisioned:**
|
||||
- ✅ **3 Datasources:** Prometheus, Loki, Jaeger
|
||||
- ✅ **Dashboard:** DevServer Monitoring Dashboard
|
||||
- ✅ **13 Panels:** API metrics, container stats, RabbitMQ, OTEL
|
||||
|
||||
---
|
||||
|
||||
## 🧪 Test Your Setup
|
||||
|
||||
### 1. Test FastAPI Endpoints
|
||||
```powershell
|
||||
# Health check
|
||||
Invoke-WebRequest -Uri "http://localhost:8000/health"
|
||||
|
||||
# API endpoints
|
||||
Invoke-WebRequest -Uri "http://localhost:8000/api/v1/roll-dice"
|
||||
Invoke-WebRequest -Uri "http://localhost:8000/api/v1/fibonacci/10"
|
||||
```
|
||||
|
||||
### 2. Check Metrics Collection
|
||||
```powershell
|
||||
# FastAPI metrics
|
||||
Invoke-WebRequest -Uri "http://localhost:8000/metrics"
|
||||
|
||||
# Prometheus targets (should show 7/7 UP)
|
||||
Start-Process "http://localhost:9090/targets"
|
||||
```
|
||||
|
||||
### 3. View Dashboards
|
||||
```powershell
|
||||
# Grafana
|
||||
Start-Process "http://localhost:3000"
|
||||
|
||||
# Jaeger traces
|
||||
Start-Process "http://localhost:16686"
|
||||
```
|
||||
|
||||
### 4. Monitor Containers
|
||||
```powershell
|
||||
# Container metrics
|
||||
Start-Process "http://localhost:8080"
|
||||
|
||||
# RabbitMQ queue
|
||||
Start-Process "http://localhost:15672"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🔄 Container Management Commands
|
||||
|
||||
### View Status
|
||||
```powershell
|
||||
cd d:\GNPL-Private\DevServer
|
||||
docker-compose ps
|
||||
```
|
||||
|
||||
### View Logs
|
||||
```powershell
|
||||
# All services
|
||||
docker-compose logs -f
|
||||
|
||||
# Specific service
|
||||
docker-compose logs -f devserver
|
||||
docker-compose logs -f grafana
|
||||
docker-compose logs -f prometheus
|
||||
```
|
||||
|
||||
### Restart Services
|
||||
```powershell
|
||||
# All services
|
||||
docker-compose restart
|
||||
|
||||
# Specific service
|
||||
docker-compose restart grafana
|
||||
docker-compose restart devserver
|
||||
```
|
||||
|
||||
### Stop Services
|
||||
```powershell
|
||||
# Stop without removing
|
||||
docker-compose stop
|
||||
|
||||
# Stop and remove containers
|
||||
docker-compose down
|
||||
|
||||
# Stop, remove containers and volumes
|
||||
docker-compose down -v
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 📝 System Metrics
|
||||
|
||||
### Docker Resources Used
|
||||
- **Containers:** 11 running
|
||||
- **Networks:** 1 (devserver_devserver_network)
|
||||
- **Volumes:** 4 (redis, prometheus, grafana, loki)
|
||||
- **Images:** 13 pulled images
|
||||
|
||||
### Image Sizes
|
||||
- `devserver-devserver`: 2.07GB
|
||||
- `devserver-celery_worker`: 2.07GB
|
||||
- All monitoring images: ~3GB total
|
||||
|
||||
---
|
||||
|
||||
## 🎓 What This Rebuild Did
|
||||
|
||||
### Benefits of Clean Rebuild
|
||||
1. ✅ **Removed corrupt data** - Any problematic cached data cleared
|
||||
2. ✅ **Fresh configuration load** - All configs re-read from files
|
||||
3. ✅ **Clean network state** - No orphaned connections
|
||||
4. ✅ **Reclaimed disk space** - 4.4GB freed up
|
||||
5. ✅ **Reset authentication** - Fresh credentials state
|
||||
6. ✅ **Clean metrics** - Start collecting fresh data
|
||||
|
||||
### Configuration Preserved
|
||||
- ✅ All YAML config files intact
|
||||
- ✅ Dashboard definitions preserved
|
||||
- ✅ Datasource provisioning working
|
||||
- ✅ Network configuration maintained
|
||||
- ✅ Port mappings unchanged
|
||||
|
||||
---
|
||||
|
||||
## ⚠️ Important Notes
|
||||
|
||||
### After Clean Rebuild
|
||||
1. **Grafana:** Need to log in again with admin/admin
|
||||
2. **Prometheus:** No historical metrics (starts collecting fresh)
|
||||
3. **Loki:** No old logs available
|
||||
4. **Grafana Dashboards:** Will show "No data" until metrics accumulate
|
||||
5. **RabbitMQ:** Queues are empty
|
||||
|
||||
### To Populate Data
|
||||
```powershell
|
||||
# Generate some API traffic
|
||||
for ($i=1; $i -le 10; $i++) {
|
||||
Invoke-WebRequest -Uri "http://localhost:8000/api/v1/roll-dice"
|
||||
Start-Sleep -Seconds 1
|
||||
}
|
||||
|
||||
# Check Prometheus has data (wait 15 seconds for scrape)
|
||||
Start-Sleep -Seconds 15
|
||||
Start-Process "http://localhost:9090/graph?g0.expr=http_requests_total"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## ✅ Success Checklist
|
||||
|
||||
- [x] All containers stopped
|
||||
- [x] All volumes removed
|
||||
- [x] Docker system cleaned (4.4GB freed)
|
||||
- [x] All 11 services started
|
||||
- [x] All containers healthy
|
||||
- [x] FastAPI responding (200)
|
||||
- [x] Prometheus targets UP (7/7)
|
||||
- [x] Grafana accessible (200)
|
||||
- [x] OTEL collector running
|
||||
- [x] Jaeger UI accessible
|
||||
- [x] RabbitMQ management UI up
|
||||
- [x] cAdvisor monitoring active
|
||||
|
||||
---
|
||||
|
||||
## 🚀 Next Steps
|
||||
|
||||
### Recommended Actions
|
||||
1. ✅ **System is running** - All services operational
|
||||
2. 🔐 **Change Grafana password** - Login and update from default
|
||||
3. 📊 **Generate test traffic** - Run API requests to populate dashboards
|
||||
4. 📈 **Check dashboards** - Wait 30 seconds, then view metrics in Grafana
|
||||
5. 🔍 **Monitor logs** - Use `docker-compose logs -f` to watch activity
|
||||
6. 🧪 **Test API endpoints** - Verify all functionality works
|
||||
7. 📝 **Document changes** - Note any custom configurations needed
|
||||
|
||||
### Optional Enhancements
|
||||
- Set up alerts in Prometheus
|
||||
- Create custom Grafana dashboards
|
||||
- Configure log retention policies
|
||||
- Add more API endpoints
|
||||
- Implement rate limiting
|
||||
- Set up SSL/TLS certificates
|
||||
|
||||
---
|
||||
|
||||
## 📚 Related Documentation
|
||||
|
||||
- **MONITORING_COMPLETE.md** - Complete monitoring system overview
|
||||
- **CONTAINER_ISSUES_RESOLVED.md** - Previous issues and fixes
|
||||
- **QUICK_REFERENCE.md** - Daily operations guide
|
||||
- **DEPLOYMENT_STATUS.md** - Initial deployment summary
|
||||
|
||||
---
|
||||
|
||||
**Rebuild Status:** SUCCESSFUL ✅
|
||||
**System Status:** PRODUCTION READY ✅
|
||||
**Last Verified:** October 13, 2025 at 08:46 IST
|
||||
**All Services:** 11/11 running, 7/7 Prometheus targets UP
|
||||
Reference in New Issue
Block a user