Troubleshooting

This guide covers common issues encountered when running the eTeamups Platform and provides step-by-step resolutions for each scenario.

Service Startup Failures

When one or more services fail to start, begin by checking container status and logs.

Diagnose

Check the status of all containers:

docker compose ps

View logs for the failing service:

docker compose logs <service-name>

Common Causes

Resolution Steps

1. Verify that docker.env exists and contains all required values.

2. Check for port conflicts:

   lsof -i:27018   # MongoDB
   lsof -i:6379    # Redis
   lsof -i:9000    # Auth Service
   lsof -i:9100    # Profile Service
   lsof -i:9107    # Organisation Service
   lsof -i:9102    # Media Service
   

3. Ensure infrastructure services are healthy before starting application services:

   docker compose up -d mongodb redis
   docker compose ps   # Wait until both show "healthy"
   docker compose up -d
   

Database Connection Issues

Testing MongoDB Connectivity

Connect to MongoDB directly from the host:

mongosh "mongodb://admin:password@localhost:27018/eteamups?authSource=admin"

Check the MongoDB container logs for errors:

docker compose logs mongodb

Verify MongoDB is responding to commands:

docker exec eteamups-mongodb mongosh --eval "db.adminCommand('ping')"

Common MongoDB Issues

Redis Connection Issues

Testing Redis Connectivity

Test the Redis connection from within the container:

docker exec eteamups-redis redis-cli -a "$REDIS_PASSWORD" ping

A successful connection returns PONG.

Check Redis memory usage:

docker exec eteamups-redis redis-cli -a "$REDIS_PASSWORD" info memory

Common Redis Issues

Docker and Container Issues

Image Pull Failures

If Docker cannot pull images from GitHub Container Registry (GHCR), re-authenticate:

echo $GITHUB_TOKEN | docker login ghcr.io -u USERNAME --password-stdin

Ensure the GITHUB_TOKEN has read:packages scope.

Out of Memory

Check current resource consumption:

docker stats

If the host is running low on memory, add swap space:

sudo fallocate -l 2G /swapfile
sudo chmod 600 /swapfile
sudo mkswap /swapfile
sudo swapon /swapfile

To make the swap permanent, add an entry to /etc/fstab:

/swapfile swap swap defaults 0 0

Container Restart Loop

When a container continuously restarts:

1. Check the logs for the root cause:

   docker compose logs <service-name> --tail 50
   

2. Verify all required environment variables are set in docker.env.

3. Check resource limits – the container may be getting killed by the OOM killer. Review docker stats output.

4. Inspect the container’s exit code:

   docker inspect <container-name> --format='{{.State.ExitCode}}'
   

Volume Permission Issues

If a service reports permission denied errors when accessing mounted volumes:

1. Check file ownership inside the container:

   docker exec <container-name> ls -la /path/to/volume
   

2. Adjust ownership on the host if needed, matching the UID/GID used inside the container.

Network Issues

If services cannot communicate with each other:

1. List Docker networks:

   docker network ls
   

2. Verify all services are attached to the same network:

   docker network inspect <network-name>
   

3. Test connectivity between containers:

   docker exec <container-name> wget -qO- http://<target-service>:<port>/health
   

SSL/TLS Certificate Issues

Verifying Certificate Files

Check that the required certificate files exist:

ls -la nginx/ssl/

The directory must contain:

• fullchain.pem – The full certificate chain (server certificate plus intermediate certificates).
• privkey.pem – The private key.

Generating Self-Signed Certificates for Testing

For local development and testing environments:

./scripts/generate-ssl.sh

Common SSL Issues

Performance Issues

High Memory Usage

Monitor memory consumption across all containers:

docker stats --no-stream --format "table {{.Name}}\t{{.MemUsage}}\t{{.MemPerc}}"

Each application service is limited to 512 MB by default (see resource limits in Monitoring). If a service consistently approaches its limit, investigate for memory leaks in the application logs.

Slow API Responses

Query the apilogs collection in MongoDB to identify slow endpoints:

// Find requests with processing time greater than 1 second
db.apilogs.find({ processingTime: { $gt: 1000 } }).sort({ startTime: -1 }).limit(20).pretty()

// Average processing time by endpoint
db.apilogs.aggregate([
  { $group: { _id: "$url", avgTime: { $avg: "$processingTime" }, count: { $sum: 1 } } },
  { $sort: { avgTime: -1 } },
  { $limit: 20 }
])

MongoDB Slow Queries

Enable the MongoDB profiler to capture slow queries:

// Enable profiling for queries slower than 100ms
db.setProfilingLevel(1, { slowms: 100 })

// Review slow queries
db.system.profile.find().sort({ ts: -1 }).limit(10).pretty()

Check that indexes are properly defined for frequently queried fields.

Redis Memory Full

Check current Redis memory usage:

docker exec eteamups-redis redis-cli -a "$REDIS_PASSWORD" INFO memory

If Redis memory is approaching the maxmemory limit:

1. Review the eviction policy (default is allkeys-lru).

2. Increase maxmemory in the Redis configuration if the host has available resources.

3. Identify large keys consuming excessive memory:

   docker exec eteamups-redis redis-cli -a "$REDIS_PASSWORD" --bigkeys
   

Too Many Connections / Rate Limiting

The platform enforces rate limits at two levels:

If legitimate traffic is being rate-limited, review and adjust the limits in the application rate limiter configuration and the Nginx limit_req_zone directives.

Common Error Codes

Log Analysis for Debugging

Application Logs

View the most recent application logs for a specific service:

docker compose logs <service-name> --tail 100 -f

Nginx Logs

Use the built-in log analysis scripts:

# View logs interactively
./scripts/view-logs.sh

# Monitor logs in real time
./scripts/log-monitor.sh -r

# Analyze errors
./scripts/log-monitor.sh -e

# Analyze performance
./scripts/log-monitor.sh -p

API Request Logs in MongoDB

Query the apilogs collection for detailed request-level debugging:

docker exec eteamups-mongodb mongosh eteamups --eval "db.apilogs.find().sort({startTime:-1}).limit(10).pretty()"

For more targeted queries, connect to MongoDB directly:

// Find all failed requests (5xx) in the last hour
db.apilogs.find({
  startTime: { $gte: new Date(Date.now() - 3600000) },
  url: { $not: /health/ }
}).sort({ startTime: -1 })

// Find requests for a specific endpoint
db.apilogs.find({ url: { $regex: /\/auth\/login/ } }).sort({ startTime: -1 }).limit(10)

Useful Diagnostic Commands

A quick reference of commands for diagnosing platform issues:

# Run the full platform health check
./scripts/health-check.sh

# Check Docker resource usage (one-time snapshot)
docker stats --no-stream

# Check disk space on the host
df -h

# Check running containers and their status
docker compose ps

# Kill processes occupying specific ports
./scripts/kill-ports.sh

# View recent API logs from MongoDB
docker exec eteamups-mongodb mongosh eteamups --eval "db.apilogs.find().sort({startTime:-1}).limit(10).pretty()"

# Inspect a specific container's configuration
docker inspect <container-name>

# Check Docker disk usage
docker system df

# View Docker network configuration
docker network ls