DevServer/docs/DOCKER_CLEAN_REBUILD.md

✅ Docker System - Clean Rebuild Complete

Date: October 13, 2025
Operation: Full Docker cleanup and fresh rebuild
Status: ALL SERVICES OPERATIONAL ✅

---

🧹 Cleanup Operations Performed

1. Stopped All Containers ✅

docker-compose down

• Removed all 11 running containers
• Disconnected from devserver_network

2. Removed All Volumes ✅

docker-compose down -v

Volumes Deleted:

• devserver_prometheus_data - Metrics storage
• devserver_redis_data - Cache data
• devserver_grafana_data - Dashboards & datasources
• devserver_loki_data - Log storage

Result: Fresh start with no persistent data

3. System Cleanup ✅

docker system prune -f

Cleaned Up:

• 9 stopped containers
• 1 unused network
• 70+ build cache objects
• Total Space Reclaimed: 4.387 GB 🎯

---

🚀 Rebuild and Startup

Services Started (Fresh)

docker-compose up -d

All 11 containers started successfully:

Service	Status	Health	Port
devserver (FastAPI)	✅ Running	HTTP 200	8000
celery_worker	✅ Running	Active	-
redis	✅ Running	Healthy	6379
rabbitmq	✅ Running	Active	5672, 15672
prometheus	✅ Running	Active	9090
grafana	✅ Running	HTTP 200	3000
otel-collector	✅ Running	Active	4317, 8889
jaeger	✅ Running	Active	16686
loki	✅ Running	Active	3100
promtail	✅ Running	Active	-
cadvisor	✅ Running	Healthy	8080

---

📊 Verification Results

FastAPI Backend ✅

Status: 200 OK
Endpoint: http://localhost:8000/health

Prometheus Monitoring ✅

Status: All 7/7 targets UP
Targets:
  - cadvisor: up
  - fastapi-app: up
  - jaeger: up
  - loki: up
  - otel-collector: up
  - prometheus: up
  - rabbitmq: up

Grafana Dashboard ✅

Status: 200 OK
Endpoint: http://localhost:3000/api/health

---

🎯 What's Fresh

New Configuration State

1. No Old Data - All volumes wiped clean
2. Fresh Grafana - Will need to log in again (admin/admin)
3. Empty Prometheus - No historical metrics yet
4. Clean Loki - No old logs
5. Reset Redis - Empty cache

Retained Configuration

✅ All configuration files preserved:

• docker-compose.yml - Service definitions
• monitoring/prometheus.yml - Scrape configs
• monitoring/otel-collector-config.yaml - OTEL pipeline
• monitoring/grafana/provisioning/ - Auto-provisioned datasources & dashboards
• All environment variables and settings

---

🔗 Access URLs (Fresh Start)

Service	URL	Credentials	Status
FastAPI Docs	http://localhost:8000/docs	-	✅ Ready
FastAPI Metrics	http://localhost:8000/metrics	-	✅ Ready
Prometheus	http://localhost:9090	-	✅ Ready
Prometheus Targets	http://localhost:9090/targets	-	✅ 7/7 UP
Grafana	http://localhost:3000	admin/admin	✅ Ready
Jaeger UI	http://localhost:16686	-	✅ Ready
RabbitMQ UI	http://localhost:15672	guest/guest	✅ Ready
cAdvisor	http://localhost:8080	-	✅ Ready
OTEL Health	http://localhost:13133	-	✅ Ready
OTEL Metrics	http://localhost:8889/metrics	-	✅ Ready

---

📈 Monitoring Dashboard

Grafana Setup

First-time login:

• URL: http://localhost:3000
• Username: admin
• Password: admin
• You'll be prompted to change password on first login

Auto-provisioned:

• ✅ 3 Datasources: Prometheus, Loki, Jaeger
• ✅ Dashboard: DevServer Monitoring Dashboard
• ✅ 13 Panels: API metrics, container stats, RabbitMQ, OTEL

---

🧪 Test Your Setup

1. Test FastAPI Endpoints

# Health check
Invoke-WebRequest -Uri "http://localhost:8000/health"

# API endpoints
Invoke-WebRequest -Uri "http://localhost:8000/api/v1/roll-dice"
Invoke-WebRequest -Uri "http://localhost:8000/api/v1/fibonacci/10"

2. Check Metrics Collection

# FastAPI metrics
Invoke-WebRequest -Uri "http://localhost:8000/metrics"

# Prometheus targets (should show 7/7 UP)
Start-Process "http://localhost:9090/targets"

3. View Dashboards

# Grafana
Start-Process "http://localhost:3000"

# Jaeger traces
Start-Process "http://localhost:16686"

4. Monitor Containers

# Container metrics
Start-Process "http://localhost:8080"

# RabbitMQ queue
Start-Process "http://localhost:15672"

---

🔄 Container Management Commands

View Status

cd d:\GNPL-Private\DevServer
docker-compose ps

View Logs

# All services
docker-compose logs -f

# Specific service
docker-compose logs -f devserver
docker-compose logs -f grafana
docker-compose logs -f prometheus

Restart Services

# All services
docker-compose restart

# Specific service
docker-compose restart grafana
docker-compose restart devserver

Stop Services

# Stop without removing
docker-compose stop

# Stop and remove containers
docker-compose down

# Stop, remove containers and volumes
docker-compose down -v

---

📝 System Metrics

Docker Resources Used

• Containers: 11 running
• Networks: 1 (devserver_devserver_network)
• Volumes: 4 (redis, prometheus, grafana, loki)
• Images: 13 pulled images

Image Sizes

• devserver-devserver: 2.07GB
• devserver-celery_worker: 2.07GB
• All monitoring images: ~3GB total

---

🎓 What This Rebuild Did

Benefits of Clean Rebuild

1. ✅ Removed corrupt data - Any problematic cached data cleared
2. ✅ Fresh configuration load - All configs re-read from files
3. ✅ Clean network state - No orphaned connections
4. ✅ Reclaimed disk space - 4.4GB freed up
5. ✅ Reset authentication - Fresh credentials state
6. ✅ Clean metrics - Start collecting fresh data

Configuration Preserved

• ✅ All YAML config files intact
• ✅ Dashboard definitions preserved
• ✅ Datasource provisioning working
• ✅ Network configuration maintained
• ✅ Port mappings unchanged

---

⚠️ Important Notes

After Clean Rebuild

1. Grafana: Need to log in again with admin/admin
2. Prometheus: No historical metrics (starts collecting fresh)
3. Loki: No old logs available
4. Grafana Dashboards: Will show "No data" until metrics accumulate
5. RabbitMQ: Queues are empty

To Populate Data

# Generate some API traffic
for ($i=1; $i -le 10; $i++) {
    Invoke-WebRequest -Uri "http://localhost:8000/api/v1/roll-dice"
    Start-Sleep -Seconds 1
}

# Check Prometheus has data (wait 15 seconds for scrape)
Start-Sleep -Seconds 15
Start-Process "http://localhost:9090/graph?g0.expr=http_requests_total"

---

✅ Success Checklist

• All containers stopped
• All volumes removed
• Docker system cleaned (4.4GB freed)
• All 11 services started
• All containers healthy
• FastAPI responding (200)
• Prometheus targets UP (7/7)
• Grafana accessible (200)
• OTEL collector running
• Jaeger UI accessible
• RabbitMQ management UI up
• cAdvisor monitoring active

---

🚀 Next Steps

Recommended Actions

1. ✅ System is running - All services operational
2. 🔐 Change Grafana password - Login and update from default
3. 📊 Generate test traffic - Run API requests to populate dashboards
4. 📈 Check dashboards - Wait 30 seconds, then view metrics in Grafana
5. 🔍 Monitor logs - Use docker-compose logs -f to watch activity
6. 🧪 Test API endpoints - Verify all functionality works
7. 📝 Document changes - Note any custom configurations needed

Optional Enhancements

• Set up alerts in Prometheus
• Create custom Grafana dashboards
• Configure log retention policies
• Add more API endpoints
• Implement rate limiting
• Set up SSL/TLS certificates

---

📚 Related Documentation

• MONITORING_COMPLETE.md - Complete monitoring system overview
• CONTAINER_ISSUES_RESOLVED.md - Previous issues and fixes
• QUICK_REFERENCE.md - Daily operations guide
• DEPLOYMENT_STATUS.md - Initial deployment summary

---

Rebuild Status: SUCCESSFUL ✅
System Status: PRODUCTION READY ✅
Last Verified: October 13, 2025 at 08:46 IST
All Services: 11/11 running, 7/7 Prometheus targets UP