Backup and Restore

This guide covers data backup and restoration procedures for the eTeamups Platform. The platform stores data in MongoDB (primary database), Redis (caching and session store), and the file system (media uploads). All backup and restore operations are handled through dedicated scripts.

Backup Overview

MongoDB Backup

Running a Backup

Execute the backup script from the project root:

./scripts/backup.sh

The script performs the following steps:

1. Creates a date-stamped directory under backups/ (e.g., backups/20241127/).
2. Runs mongodump inside the eteamups-mongodb container with gzip compression.
3. Copies the backup from the container to the host filesystem.
4. Removes the temporary backup directory from inside the container.
5. Reports the backup size.

What Gets Backed Up

The mongodump command authenticates using the MONGO_INITDB_ROOT_USERNAME and MONGO_INITDB_ROOT_PASSWORD from docker.env and backs up all databases, including:

• The primary application database (eteamups by default)
• The apilogs collection containing API request logs
• Any other databases present in the MongoDB instance

Backup Output

A successful backup produces the following structure:

backups/
  20241127/
    mongodb-20241127_143000/
      eteamups/
        users.bson.gz
        users.metadata.json.gz
        apilogs.bson.gz
        apilogs.metadata.json.gz
        ...
      admin/
        ...
    redis-20241127_143000.rdb
    media-20241127_143000.tar.gz
    docker-compose-20241127_143000.yml
    manifest-20241127_143000.txt

Backup Manifest

Each backup includes a manifest-{timestamp}.txt file recording:

• Backup date and timestamp
• List of backed-up components
• Docker Compose version
• List of running services at the time of backup

Configuration

Automatic Cleanup

The backup script automatically deletes backup directories older than RETENTION_DAYS (default: 30 days). This runs at the end of every backup operation.

MongoDB Restore

Running a Restore

Restore from a specific backup date:

./scripts/restore.sh 20241127

If no date is provided, the script lists available backups:

./scripts/restore.sh
# Output:
# Error: Please provide backup date
# Available backups:
# 20241125
# 20241126
# 20241127

Restore Process

The scripts/restore.sh script performs the following steps:

1. Validates the backup – Checks that the backup directory exists and displays its contents.
2. Prompts for confirmation – Requires typing yes to proceed. This is a destructive operation.
3. Stops application services – Stops auth, profile, organisation, media, message-queue, admin-portal, and nginx containers. MongoDB and Redis remain running.
4. Restores MongoDB – Copies the backup into the container, drops the existing database, and runs mongorestore with gzip decompression.
5. Restores Redis – Stops Redis, copies the RDB file into the container, and restarts Redis.
6. Restores media files – Extracts the media archive into the media_uploads Docker volume, replacing existing files.
7. Starts all services – Brings all containers back up and waits for them to become healthy (up to 2.5 minutes).

Partial Restore

To restore only specific components, you can run the relevant Docker commands manually:

Restore Only MongoDB

# Copy backup to container
docker cp backups/20241127/mongodb-20241127_143000 eteamups-mongodb:/data/restore

# Restore (without dropping existing data -- merges)
docker exec eteamups-mongodb mongorestore \
  --username="$MONGO_INITDB_ROOT_USERNAME" \
  --password="$MONGO_INITDB_ROOT_PASSWORD" \
  --authenticationDatabase=admin \
  --gzip \
  /data/restore

# Clean up
docker exec eteamups-mongodb rm -rf /data/restore

Restore Only a Specific Collection

docker exec eteamups-mongodb mongorestore \
  --username="$MONGO_INITDB_ROOT_USERNAME" \
  --password="$MONGO_INITDB_ROOT_PASSWORD" \
  --authenticationDatabase=admin \
  --gzip \
  --nsInclude="eteamups.users" \
  /data/restore

Restore Only Redis

docker compose stop redis
docker cp backups/20241127/redis-20241127_143000.rdb eteamups-redis:/data/dump.rdb
docker compose start redis

Redis Persistence

AOF (Append-Only File)

Redis is configured with AOF persistence in docker-compose.yml:

command: >
  redis-server
  --appendonly yes
  --appendfsync everysec
  --maxmemory 512mb
  --maxmemory-policy allkeys-lru
  --requirepass ${REDIS_PASSWORD:-changeme}

How AOF Works

• Every write command is appended to the AOF file (appendonly.aof) inside the Redis data volume.
• If Redis restarts, it replays the AOF file to reconstruct the dataset.
• With appendfsync everysec, you could lose at most 1 second of writes in a crash scenario.

Redis Data Volume

Redis data is stored in the redis_data Docker volume, which persists across container restarts. The volume contains:

• dump.rdb – Point-in-time RDB snapshot (created by SAVE commands during backup)
• appendonly.aof – Append-only file for continuous persistence

Manual Redis Snapshot

To create an immediate RDB snapshot:

docker exec eteamups-redis redis-cli -a "$REDIS_PASSWORD" SAVE

To copy it to the host:

docker cp eteamups-redis:/data/dump.rdb ./redis-backup.rdb

Backup Scheduling

Setting Up Automated Backups with Cron

Add a daily backup cron job:

crontab -e

Add the following line for a daily backup at 2:00 AM:

0 2 * * * cd /path/to/eteamups-platform && ./scripts/backup.sh >> backups/backup.log 2>&1

Recommended Backup Schedule

Off-Site Backup

The backup script supports two off-site methods if the corresponding environment variables are set:

rsync

Set the BACKUP_REMOTE_PATH variable:

export BACKUP_REMOTE_PATH="user@backup-server:/backups/eteamups"

The script will automatically rsync the backup after local completion.

AWS S3

Set the AWS_S3_BUCKET variable (requires the AWS CLI to be installed):

export AWS_S3_BUCKET="my-backup-bucket"

The script will sync the backup to s3://{bucket}/backups/{date}/.

Backup Verification

Testing Backup Integrity

After each backup, verify its contents:

# List the backup contents
ls -lh backups/20241127/

# Check MongoDB backup completeness
ls backups/20241127/mongodb-*/eteamups/

# Verify gzip archive integrity for media
gzip -t backups/20241127/media-*.tar.gz && echo "Archive OK" || echo "Archive corrupted"

# Check manifest
cat backups/20241127/manifest-*.txt

Restore Test Procedure

Periodically test your backups by restoring to a separate environment:

1. Spin up a temporary MongoDB instance:

   docker run -d --name mongo-test -p 27018:27017 mongo:7-jammy
   

2. Copy and restore the backup:

   docker cp backups/20241127/mongodb-20241127_143000 mongo-test:/data/restore
   docker exec mongo-test mongorestore --gzip /data/restore
   

3. Verify the data:

   docker exec mongo-test mongosh --eval "db.getSiblingDB('eteamups').getCollectionNames()"
   

4. Clean up:

   docker stop mongo-test && docker rm mongo-test
   

Disaster Recovery Procedures

Scenario 1: Single Service Failure

If one application service fails but the database is intact:

1. Check container status: docker compose ps
2. View container logs: docker compose logs auth-service
3. Restart the service: docker compose restart auth-service
4. Verify health: ./scripts/health-check.sh

No data restore is needed since the service is stateless.

Scenario 2: Database Corruption

If MongoDB data becomes corrupted:

1. Stop all application services:

   docker compose stop auth-service profile-service organisation-service media-service message-queue-service admin-portal zeswa-hub nginx
   

2. Identify the latest good backup:

   ls -la backups/
   

3. Run the restore script:

   ./scripts/restore.sh 20241127
   

4. Verify the restoration:

   ./scripts/health-check.sh
   

Scenario 3: Complete Server Failure

If the entire server must be rebuilt:

1. Provision a new server with Docker and Docker Compose installed.

2. Clone the repository and copy your docker.env configuration.

3. Copy the latest backup from off-site storage:

   mkdir -p backups/
   rsync -az backup-server:/backups/eteamups/latest/ backups/
   

4. Pull the Docker images:

   docker compose pull
   

5. Start the infrastructure services:

   docker compose up -d mongodb redis
   

6. Wait for MongoDB and Redis to become healthy, then restore data:

   ./scripts/restore.sh {backup-date}
   

7. Start all services:

   docker compose up -d
   

8. Run the health check:

   ./scripts/health-check.sh
   

Scenario 4: Redis Data Loss

Redis is used for caching and sessions. Data loss is generally recoverable:

1. Restart Redis: docker compose restart redis
2. If the AOF file is corrupted:

   docker compose stop redis
   docker exec eteamups-redis redis-check-aof --fix /data/appendonly.aof
   docker compose start redis
   

3. If a full restore is needed, use the RDB file from the latest backup (see the partial restore section above).
4. Active user sessions will be lost and users will need to re-authenticate.

Related Scripts