MongoDB & Backups Guide

Complete guide to the MongoDB database and backup system for UnderstandTech deployments.

---

Table of Contents

1. Architecture Overview

2. Connecting to MongoDB

3. Backup System

4. Restore Procedures

5. Maintenance Operations

6. Troubleshooting

---

1. Architecture Overview

Services

The MongoDB deployment uses a separation-of-concerns architecture built around two specialized containers working together. The first container runs MongoDB itself—the actual database engine that stores and serves your application data. The second container handles backup operations on a schedule, keeping your data safe without requiring manual intervention. By separating these responsibilities, we can restart or update the backup system without touching the database, and vice versa. This separation also means each container can be monitored, scaled, and configured independently.

┌─────────────────────────────────────────────────┐
│                                                 │
│  ┌────────────┐         ┌──────────────────┐   │
│  │  MongoDB   │◄────────│ MongoDB Backup   │   │
│  │  :27017    │         │  (tiredofit)     │   │
│  └─────┬──────┘         └──────────────────┘   │
│        │                         │              │
│        ▼                         ▼              │
│  ┌────────────┐         ┌──────────────────┐   │
│  │ mongodb-   │         │ mongodb-         │   │
│  │ data       │         │ backups          │   │
│  │ (volume)   │         │ (volume)         │   │
│  └────────────┘         └──────────────────┘   │
│                                                 │
└─────────────────────────────────────────────────┘

MongoDB Container (ut-mongodb)

This container runs the official MongoDB 8.2 server image, which represents the latest stable release with improvements in query performance, security, and operational features. The version choice matters because MongoDB makes backward-incompatible changes between major versions, so knowing which version you're running helps when consulting documentation or troubleshooting issues.

• Image: mongo:8.2

• Port: 127.0.0.1:27017 (localhost only, for security)

• Storage: ut-mongodb-data volume → /data/db

• Health Check: mongosh ping command every 30s

• Resources: 4GB memory limit, 2 CPU cores

Backup Container (ut-mongodb-backup)

This container runs a specialized backup tool from tiredofit that handles the mechanical work of creating database backups on a schedule. The tool connects to the MongoDB container over the Docker network, exports the database contents using MongoDB's native mongodump utility, compresses the resulting backup file with gzip, and stores it in a separate volume.

The backup schedule defaults to running daily at 02:00 UTC (2 AM), which you can configure through environment variables.

• Image: tiredofit/db-backup:latest

• Backup Storage: ut-mongodb-backups volume → /backup

• Schedule: Daily at 02:00 (configurable via BACKUP_BEGIN)

• Retention: 30 days (configurable via CLEANUP_TIME)

• Format: Gzipped MongoDB archives (.archive.gz)

Data Persistance

Two critical volumes store all persistent data for this deployment:

The ut-mongodb-data volume holds the live database files. This is where MongoDB stores your actual data: collections, documents, indexes, and all the metadata it needs to run. The volume lives at /var/lib/docker/volumes/ut-mongodb-data/_data/ on the host filesystem, though you rarely need to access it directly. MongoDB manages these files in a binary format optimized for performance, which is why you interact with the data through MongoDB's query language rather than by looking at the files themselves.

The ut-mongodb-backups volume stores the backup archives created by the backup container. Each backup is a complete snapshot of your database at a specific point in time, compressed into a single .archive.gz file. These backups live at /var/lib/docker/volumes/ut-mongodb-backups/_data/ and can be browsed directly from the host system. Unlike the live database volume, backup files are designed to be human-readable (when decompressed) and portable - you can copy them to other systems, archive them to cloud storage, or restore them to a completely different MongoDB installation.

Volume	Path	Purpose	Backup Needed
ut-mongodb-data	/data/db	Live database files	Yes (automated)
ut-mongodb-backups	/backup	Backup archives	Optional (offsite copies)

Configuration

The MongoDB stack is configured entirely through environment variables stored in your .env file.

# Database Credentials
MONGODB_USERNAME=mongoadmin
MONGODB_PASSWORD=<your-secure-password>
MONGODB_DATABASE=ut-db
MONGODB_HOST=mongodb
MONGODB_PORT=27017

# Backup Schedule
BACKUP_BEGIN=1520           # Time in HHMM format (15:20 = 3:20 PM)
BACKUP_INTERVAL=1440        # Minutes between backups (1440 = daily)
BACKUP_CLEANUP_TIME=43200   # Keep backups for 30 days (in minutes)
BACKUP_COMPRESSION=GZ
BACKUP_COMPRESSION_LEVEL=6

# Timezone
TIMEZONE=UTC

---

2. Connecting to MongoDB

MongoDB's network access is intentionally restricted for security. By default, the container only exposes port 27017 to the internal Docker network (ut-backend-network), making it accessible only to other containers in the stack. This isolation prevents unauthorized external access while allowing your application services to communicate with the database.

A. From the Host (DGX System)

By default, MongoDB is not accessible from the host system. The compose file uses expose: "27017" rather than ports, which means the port is only available within the Docker network.

To connect from the host, you need to temporarily modify the compose file:

1

Edit compose.yaml

Find the mongodb service and change:

# Current configuration (internal only)
expose:
  - "27017"

To:

# Modified configuration (localhost access)
ports:
  - "127.0.0.1:27017:27017"

The 127.0.0.1: prefix binds the port only to localhost, preventing network-wide exposure.

2

Restart MongoDB

docker compose up -d mongodb

3

Connect from host

# Using mongosh (if installed on host)
mongosh "mongodb://mongoadmin:12345678@localhost:27017/?authSource=admin"

# Or test connection via container exec
docker exec ut-mongodb mongosh \
  -u mongoadmin \
  -p 12345678 \
  --authenticationDatabase admin \
  --eval "db.adminCommand('ping')"

4

Revert when done (recommended for security)

Change the compose file back to expose: - "27017" and restart:

docker compose up -d mongodb

B. From a Remote Machine (MongoDB Compass)

Why MongoDB Compass?

MongoDB Compass is the official graphical user interface for MongoDB. It provides a visual way to explore your data, run queries, analyze performance, and manage your database without writing commands. This is especially useful when you need to:

• Browse collections and documents visually

• Debug data issues by inspecting actual database contents

• Build and test queries with a visual query builder

• Monitor database performance and index usage

• Perform administrative tasks through a user-friendly interface

Since the DGX Spark runs ARM64 architecture and Compass doesn't have an ARM-native Linux build, you'll need to connect from a separate machine (like your Mac, Windows PC, or x86 Linux workstation).

Connection Requirements

Because MongoDB isn't exposed to the network by default, you have two options for remote access:

1. SSH tunnel (recommended - secure, no compose changes needed)

2. Temporary network exposure (testing only - requires compose file changes)

We'll cover SSH tunneling first since it's the most secure approach.

Prerequisites: SSH Access Setup

Before you can create an SSH tunnel, you need SSH access to the DGX system. If you haven't already set this up, follow these steps:

On your remote machine (Mac/Windows/Linux):

1. Generate an SSH key pair (skip if you already have one):

# On Mac/Linux
ssh-keygen -t ed25519 -C "[email protected]"

# On Windows (PowerShell)
ssh-keygen -t ed25519 -C "[email protected]"

1. Press Enter to accept the default location (~/.ssh/id_ed25519), and optionally set a passphrase for added security.

2. Copy your public key to the DGX:

# Mac/Linux - using ssh-copy-id (easiest)
ssh-copy-id [email protected]

# Alternative: manually copy the key
cat ~/.ssh/id_ed25519.pub
# Then paste the output into ~/.ssh/authorized_keys on the DGX

1. Test SSH connection:

ssh [email protected]

1. You should be able to log in without being prompted for a password (unless you set a key passphrase). Type exit to close the session.

On Windows (if ssh-copy-id isn't available):

Windows 10/11 includes OpenSSH, but ssh-copy-id may not be available. Instead:

# Display your public key
type $env:USERPROFILE\.ssh\id_ed25519.pub

# Copy the output, then SSH to the DGX with password
ssh [email protected]

# On the DGX, add the key
mkdir -p ~/.ssh
chmod 700 ~/.ssh
nano ~/.ssh/authorized_keys
# Paste your public key on a new line, save and exit

# Set proper permissions
chmod 600 ~/.ssh/authorized_keys
exit

Troubleshooting SSH Access:

• "Permission denied (publickey)": Your key isn't in the DGX's authorized_keys file. Repeat step 2 above.

• "Host key verification failed": The DGX's SSH fingerprint changed (e.g., after OS reinstall). Run ssh-keygen -R understand.local and try again.

• Connection timeout: Check network connectivity. Can you ping the DGX? ping understand.local

Once SSH access is working, you can proceed with the tunnel setup.

Option 1: SSH Tunnel (Recommended - Most Secure)

An SSH tunnel creates an encrypted connection between your local machine and the DGX, forwarding network traffic through this secure channel. This method is ideal because it keeps MongoDB unexposed and encrypts all traffic.

1

Step 1: Set up the SSH tunnel

On your remote machine (Mac, Windows, Linux), open a terminal and run:

# Forward local port 27017 to DGX's MongoDB
ssh -L 27017:localhost:27017 [email protected] -N

# If port 27017 is already in use locally, use a different port:
ssh -L 27018:localhost:27017 [email protected] -N

Lags explained:

• -L 27017:localhost:27017 - Forward local port 27017 to remote port 27017

• -N - Don't open a shell, just keep the tunnel open

• Keep this terminal window open while you're using Compass

2

Connect MongoDB Compass

With the tunnel running, configure Compass with these settings:

Field	Value
Connection String	mongodb://localhost:27017 (or :27018 if you changed the port)
Username	mongoadmin
Password	(your MONGODB_PASSWORD)
Authentication Database	admin
Authentication Method	Username/Password

Or use the full connection string:

mongodb://mongoadmin:12345678@localhost:27017/?authSource=admin

3

Test the connection

Click "Connect" in Compass. You should see your databases:

• admin - MongoDB system database

• ut-db - Your application database

• config, local - MongoDB internal databases

Option 2: Temporary Direct Access (Testing Only)

Only use this for temporary testing. It exposes MongoDB to your network.

This method requires modifying the compose file to expose MongoDB's port to the network. Use this only for quick testing when you can't set up an SSH tunnel.

1

Modify the compose file to expose MongoDB

Edit compose.yaml and find the mongodb service. Change the port configuration:

# Original (internal Docker network only)
mongodb:
  expose:
    - "27017"

# Change to (network accessible - bind to all interfaces)
mongodb:
  ports:
    - "27017:27017"

# Or better yet (localhost only - requires SSH tunnel from remote machine anyway)
mongodb:
  ports:
    - "127.0.0.1:27017:27017"

Understanding the port binding options:

• expose: - "27017" - Only accessible within Docker network (default, most secure)

• ports: - "127.0.0.1:27017:27017" - Accessible from DGX host only (requires SSH tunnel from remote)

• ports: - "27017:27017" - Accessible from entire network (least secure, use with caution)

2

Restart MongoDB

docker compose up -d mongodb

Verify the port is now exposed:

docker port ut-mongodb
# Should show: 27017/tcp -> 0.0.0.0:27017 (if bound to all interfaces)
#           or 27017/tcp -> 127.0.0.1:27017 (if bound to localhost)

3

Connect from Compass

If you used ports: - "27017:27017" (all interfaces), connect directly using the DGX's hostname

mongodb://mongoadmin:[email protected]:27017/?authSource=admin

If you used ports: - "127.0.0.1:27017:27017" (localhost only), you still need an SSH tunnel as described in Option 1, but now the tunnel will work because MongoDB is listening on the host.

4

Revert the change when done

IMPORTANT: Change the compose file back to the original configuration:

mongodb:

expose:

- "27017"

Then restart MongoDB:

docker compose up -d mongodb

Verify the port is no longer exposed:

docker port ut-mongodb
# Should show nothing (port only exposed internally)

Troubleshooting SSH Tunnel Connections

"Address already in use" error

Port 27017 is already in use on your local machine (probably local MongoDB). Use a different port:

# Check what's using the port
lsof -i :27017

# Use port 27018 instead
ssh -L 27018:localhost:27017 [email protected] -N

Then connect Compass to localhost:27018.

"Connection refused" error

1. Verify MongoDB is running on the DGX:

docker ps | grep mongodb

1. Verify it's listening on localhost:

docker exec ut-mongodb mongosh --eval "db.adminCommand('ping')"

1. Verify the port mapping

docker port ut-mongodb
# Should show: 27017/tcp -> 127.0.0.1:27017

"Host key verification failed"

The DGX's SSH key changed (e.g., after reinstall). Clear the old key:

ssh-keygen -R understand.local
# Then retry the SSH tunnel

---

3. Backup System

The backup system uses the tiredofit/db-backup container, which provides automated MongoDB backups with built-in scheduling, compression, and retention management.

Backup Schedule

Backups run automatically based on your configuration:

# Default schedule (in compose.yaml environment)
DB01_BACKUP_BEGIN=1520        # Start time: 15:20 (3:20 PM)
DB01_BACKUP_INTERVAL=1440     # Interval: 1440 minutes (24 hours)
DB01_CLEANUP_TIME=43200       # Retention: 30 days (43200 minutes)

Backup timing:

• First backup runs at BACKUP_BEGIN time

• Subsequent backups run every BACKUP_INTERVAL minutes

• Old backups are automatically deleted after CLEANUP_TIME

Backup Storage

Backups are stored in the ut-mongodb-backups Docker volume:

# View backup files
docker exec ut-mongodb-backup ls -lh /backup

# Check volume size
docker volume inspect ut-mongodb-backups
docker system df -v | grep mongodb-backups

Backup filename format:

mongo_<database>_<host>_YYYYMMDD-HHMMSS.archive.gz
Example: mongo_ut-db_mongodb_20260205-152000.archive.gz

Manual Backup Operations

Trigger Immediate Backup

# Force an immediate backup (doesn't wait for scheduled time)
docker exec ut-mongodb-backup backup-now

List Available Backups

# Show all backup files with sizes and dates
docker exec ut-mongodb-backup ls -lh /backup

# Count total backups
docker exec ut-mongodb-backup ls -1 /backup/*.gz | wc -l

# Show oldest and newest
docker exec ut-mongodb-backup ls -1t /backup/*.gz | tail -1  # oldest
docker exec ut-mongodb-backup ls -1t /backup/*.gz | head -1  # newest

Copy Backups to Host

For offsite storage or additional safety, copy backups to the host filesystem:

# Copy all backups to current directory
docker cp ut-mongodb-backup:/backup ./mongodb-backups

# Copy specific backup
docker cp ut-mongodb-backup:/backup/mongo_ut-db_mongodb_20260205-152000.archive.gz ./

# Copy to a specific location
docker cp ut-mongodb-backup:/backup /home/understand-tech/offsite-backups/

Verify Backup Integrity

# Check if backup file is valid gzip
docker exec ut-mongodb-backup gzip -t /backup/mongo_ut-db_mongodb_20260205-152000.archive.gz

# List contents without extracting
docker exec ut-mongodb-backup sh -c \
  'gunzip -c /backup/mongo_ut-db_mongodb_20260205-152000.archive.gz | mongorestore --archive --dryRun'

Backup Configuration

To modify backup behavior, edit the compose.yaml file under the mongodb-backup service:

mongodb-backup:
  environment:
    # Change backup time (HHMM format, 24-hour)
    DB01_BACKUP_BEGIN: 0200              # 2:00 AM
    
    # Change backup frequency
    DB01_BACKUP_INTERVAL: 720            # Every 12 hours
    
    # Change retention period
    DB01_CLEANUP_TIME: 86400             # Keep for 60 days (in minutes)
    
    # Change compression
    DB01_COMPRESSION: ZSTD               # Use zstd instead of gzip
    DB01_COMPRESSION_LEVEL: 3            # Lower = faster, higher = smaller

After changes, restart the backup container:

docker compose up -d mongodb-backup

Backup Best Practices

1. Monitor backup logs:

docker logs ut-mongodb-backup --tail 100 -f

1. Copy backups offsite regularly:

   1. Set up a cron job to copy backups to NAS/cloud storage

   2. Keep at least 3 backup copies in different locations (3-2-1 rule)

2. Test restores periodically

   1. Verify you can actually restore from backups

   2. Test at least monthly, or before major changes

3. Check disk space:

# Check backup volume usage
docker system df -v | grep mongodb-backups

# Check host disk space
df -h /var/lib/docker/volumes

1. Document your backup schedule:

   1. Record the backup time in your runbook

   2. Note the retention period

   3. Document where offsite copies are stored

---

4. Restore Procedures

There are two methods to restore MongoDB backups: interactive CLI (easier for beginners) and manual commands (more control for advanced users).

Prerequisites for Any Restore

1. MongoDB must be running and healthy:

docker ps | grep mongodb
docker compose ps mongodb
# Status should show "Up (healthy)"

1. Identify the backup to restore:

docker exec ut-mongodb-backup ls -lh /backup

1. Stop application services (prevents data inconsistency):

docker compose stop api workers frontend

Method 1: Interactive Restore (Recommended for Most Users)

The backup container includes an interactive restore wizard that guides you through the process.

1

Start the interactive restore

docker exec -it ut-mongodb-backup restore

2

Select backup file

1) /backup/mongo_ut-db_mongodb_20260205-152000.archive.gz
2) /backup/mongo_ut-db_mongodb_20260204-152000.archive.gz
3) Custom
4) Quit
Please select a file to restore: 1

Type the number of the backup you want to restore.

3

Select database type

What Database Type are you looking to restore?
    F ) Parsed Filename Type: 'mongo'
    M ) MySQL / MariaDB
    O ) MongoDB
    P ) Postgresql
    Q ) Quit
** Enter Value (F) (Default) | (M) | (O) | (P) :

Press Enter to accept the default (F - parsed from filename).

4

Select hostname

What Hostname do you wish to restore to:
    C ) Custom Entered Hostname
    E ) Environment Variable DB01_HOST: 'mongodb'
    F ) Parsed Filename Host: 'mongodb'
    Q ) Quit
** Enter Value (C) | (E*) | (F) :

Press Enter to accept the default (E - environment variable).

5

Select database name

What Database Name do you want to restore to?
    C ) Custom Entered Database Name
    F ) Parsed Filename DB Name: 'ut-db'
    Q ) Quit
** Enter Value (C) | (F*) :

Press Enter to accept the default (F).

6

Select database user

What database user will be used for restore:
    C ) Custom Entered Database User
    E ) Environment Variable DB01_USER: 'mongoadmin'
    Q ) Quit
** Enter Value (C) | (E) :

Type e and press Enter.

7

Select database password

What Database Password will be used to restore?
    C ) Custom Entered Database Password
    E ) Environment Variable DB01_PASS
    Q ) Quit
** Enter Value (C) | (E*) :

Press Enter to accept the default (E).

8

Select database port

What Database Port do you wish to use?
    C ) Custom Entered Database Port
    D ) Default Port for Database type 'mongo': '27017'
    E ) Environment Variable DB01_PORT: '27017'
    Q ) Quit
** Enter Value (C) | (D) | (E*) :

Press Enter to accept the default (E).

9

Confirm drop existing data

Do you wish to drop any existing data before restoring?
    Y ) Yes
    N ) No
    Q ) Quit
** Enter Value (Y) | (N) | (Q) :

• Type y for a clean restore (recommended - replaces all data)

• Type n to merge with existing data (may cause conflicts)

10

Verify restore completion

2026-02-05.15:20:00 [INFO] ** [db-backup-restore] Restoring '/backup/mongo_ut-db_mongodb_20260205-152000.archive.gz' into 'mongodb'/'ut-db'
2026-02-05T15:20:01.000+0000    preparing collections to restore from
2026-02-05T15:20:01.100+0000    reading metadata for ut-db.users from archive
2026-02-05T15:20:01.200+0000    restoring ut-db.users from archive
2026-02-05T15:20:02.000+0000    finished restoring ut-db.users (150 documents, 0 failures)
...
2026-02-05T15:20:05.000+0000    86 document(s) restored successfully. 0 document(s) failed to restore.

Look for the final line showing successful document count and zero failures.

11

Restart application services

docker compose up -d

Method 2: Manual Restore (Advanced Users)

Use direct mongorestore commands for more control over the restore process.

Standard Restore (Drop and Replace)

docker exec ut-mongodb-backup mongorestore \
  --host mongodb \
  --port 27017 \
  -u mongoadmin \
  -p <your-password> \
  --authenticationDatabase admin \
  --archive=/backup/mongo_ut-db_mongodb_20260205-152000.archive.gz \
  --gzip \
  --drop

Flags explained:

• --host mongodb - Connect to MongoDB container

• --port 27017 - MongoDB port

• -u mongoadmin / -p <password> - Authentication credentials

• --authenticationDatabase admin - Auth against admin database

• --archive=<file> - Backup file to restore

• --gzip - Decompress gzipped backup

• --drop - Drop existing collections before restoring (clean restore)

Selective Restore (Specific Database)

docker exec ut-mongodb-backup mongorestore \
  --host mongodb \
  --port 27017 \
  -u mongoadmin \
  -p <your-password> \
  --authenticationDatabase admin \
  --archive=/backup/mongo_ut-db_mongodb_20260205-152000.archive.gz \
  --gzip \
  --nsInclude="ut-db.*" \
  --drop

The --nsInclude="ut-db.*" flag restores only the ut-db database, ignoring others in the backup.

Selective Restore (Specific Collection)

docker exec ut-mongodb-backup mongorestore \
  --host mongodb \
  --port 27017 \
  -u mongoadmin \
  -p <your-password> \
  --authenticationDatabase admin \
  --archive=/backup/mongo_ut-db_mongodb_20260205-152000.archive.gz \
  --gzip \
  --nsInclude="ut-db.users" \
  --drop

Restores only the users collection from the ut-db database.

Merge Restore (Without Drop)

docker exec ut-mongodb-backup mongorestore \
  --host mongodb \
  --port 27017 \
  -u mongoadmin \
  -p <your-password> \
  --authenticationDatabase admin \
  --archive=/backup/mongo_ut-db_mongodb_20260205-152000.archive.gz \
  --gzip

Removes the --drop flag to merge backup data with existing data. Warning: This can cause conflicts if documents with the same _id exist.

Dry Run (Test Without Restoring)

docker exec ut-mongodb-backup mongorestore \
  --host mongodb \
  --port 27017 \
  -u mongoadmin \
  -p <your-password> \
  --authenticationDatabase admin \
  --archive=/backup/mongo_ut-db_mongodb_20260205-152000.archive.gz \
  --gzip \
  --dryRun

Tests the restore process without actually modifying data. Use this to verify the backup is valid.

Disaster Recovery: Full System Restore

If you need to restore from scratch after a complete failure:

1

Stop all services

docker compose down

2

Remove old MongoDB data (if corrupted)

docker volume rm ut-mongodb-data

3

Start only MongoDB (it will initialize empty)

docker compose up -d mongodb

4

Wait for MongoDB to be healthy

# Watch until status shows "healthy"
watch -n 2 'docker compose ps mongodb'

5

Start backup container

docker compose up -d mongodb-backup

6

Restore using either method above

Interactive:

docker exec -it ut-mongodb-backup restore

Or manual:

docker exec ut-mongodb-backup mongorestore \
  --host mongodb \
  --port 27017 \
  -u mongoadmin \
  -p <your-password> \
  --authenticationDatabase admin \
  --archive=/backup/mongo_ut-db_mongodb_20260205-152000.archive.gz \
  --gzip \
  --drop

7

Start remaining services

docker compose up -d

8

Verify application functionality

# Check all services are healthy
docker compose ps

# Check logs for errors
docker compose logs --tail 50 api

---

5. Maintenance Operations

Viewing Database Statistics

# Connect to MongoDB
docker exec -it ut-mongodb mongosh \
  -u mongoadmin \
  -p <your-password> \
  --authenticationDatabase admin

# Inside mongosh:
use ut-db
db.stats()                    # Database statistics
db.users.stats()              # Collection statistics
db.getCollectionNames()       # List all collections

Monitoring Database Size

# Check volume size
docker system df -v | grep mongodb-data

# Check database size from within MongoDB
docker exec ut-mongodb mongosh \
  -u mongoadmin \
  -p <your-password> \
  --authenticationDatabase admin \
  --eval "db.stats(1024*1024)" \
  ut-db

Compacting Database (Reclaim Space)

After deleting large amounts of data:

docker exec ut-mongodb mongosh \
  -u mongoadmin \
  -p <your-password> \
  --authenticationDatabase admin \
  --eval "db.runCommand({compact: 'collection_name', force: true})" \
  ut-db

Replace collection_name with the actual collection name.

Changing MongoDB Password

Step 1: Update password in MongoDB

docker exec -it ut-mongodb mongosh \
  -u mongoadmin \
  -p <old-password> \
  --authenticationDatabase admin

# Inside mongosh:
use admin
db.changeUserPassword("mongoadmin", "<new-password>")
exit

Step 2: Update .env file

nano .env
# Change MONGODB_PASSWORD to new password

Step 3: Restart services that use MongoDB

docker compose up -d api workers mongodb-backup

Exporting Specific Data

For ad-hoc exports or data migration:

# Export entire database
docker exec ut-mongodb mongodump \
  -u mongoadmin \
  -p <your-password> \
  --authenticationDatabase admin \
  --db ut-db \
  --archive=/tmp/export.archive \
  --gzip

# Copy to host
docker cp ut-mongodb:/tmp/export.archive ./export.archive

# Export specific collection as JSON
docker exec ut-mongodb mongoexport \
  -u mongoadmin \
  -p <your-password> \
  --authenticationDatabase admin \
  --db ut-db \
  --collection users \
  --out=/tmp/users.json

docker cp ut-mongodb:/tmp/users.json ./users.json

---

6. Troubleshooting

Common Issues and Solutions

MongoDB Container Won't Start

Symptom: Container exits immediately or repeatedly restarts

Diagnosis:

docker logs ut-mongodb --tail 50

Common causes:

1. Incorrect credentials in environment variables:

   1. Check MONGODB_USERNAME and MONGODB_PASSWORD in .env

   2. Ensure no special characters cause shell parsing issues

2. Corrupted data volume:

# Backup volume first
docker run --rm -v ut-mongodb-data:/data -v $(pwd):/backup alpine \
  tar czf /backup/mongodb-data-emergency-backup.tar.gz -C /data .

# Remove volume
docker volume rm ut-mongodb-data

# Recreate and restore from backup
docker compose up -d mongodb
# Then restore using backup procedures above

1. Insufficient resources:

   • Check available memory: free -h

   • Check disk space: df -h

   • Reduce MongoDB memory limits in compose.yaml if needed

Health Check Failing

Symptom: Container shows "unhealthy" status

Diagnosis:

docker inspect ut-mongodb | jq '.[0].State.Health'

Common causes:

1. MongoDB still starting up:

   • First startup takes 30-60 seconds

   • Check logs: docker logs ut-mongodb

   • Wait for "Waiting for connections" message

2. Authentication issues:

# Test health check command manually
docker exec ut-mongodb mongosh \
  --eval "db.adminCommand('ping')" \
  -u mongoadmin \
  -p <your-password> \
  --authenticationDatabase admin

Backup Container Errors

Symptom: Backups not appearing or restore failing

Diagnosis:

docker logs ut-mongodb-backup --tail 100

Common causes:

1. Connection to MongoDB failed:

   • Verify MongoDB is healthy: docker compose ps mongodb

   • Check network: docker network inspect ut-backend-network

   • Test connection:

docker exec ut-mongodb-backup nc -zv mongodb 27017

1. Incorrect environment variables:

docker exec ut-mongodb-backup env | grep DB01

1. Disk space full:

docker system df
docker volume inspect ut-mongodb-backups

Restore Fails with Authentication Error

Symptom: Authentication failed during restore

Causes and solutions:

1. Wrong credentials:

   1. Verify credentials in .env file

   2. Test connection manually:

docker exec ut-mongodb mongosh \  -u mongoadmin \  -p <your-password> \  --authenticationDatabase admin \  --eval "db.adminCommand('ping')"

1. Backup container can't reach MongoDB:

   1. Check both containers are on the same network:

docker inspect ut-mongodb | grep NetworkModedocker inspect ut-mongodb-backup | grep NetworkMode

1. Use manual restore as workaround:

docker exec ut-mongodb-backup mongorestore \
  --host mongodb \
  --port 27017 \
  -u mongoadmin \
  -p <your-password> \
  --authenticationDatabase admin \
  --archive=/backup/<backup-file>.archive.gz \
  --gzip \
  --drop

Connection Timeout from Application

Symptom: API or workers can't connect to MongoDB

Diagnosis:

docker logs ut-api --tail 50 | grep -i mongo
docker logs ut-workers --tail 50 | grep -i mongo

Solutions:

1. Verify connection string format:

# Should be:
mongodb://mongoadmin:12345678@mongodb:27017/?authSource=admin

# Check environment variables
docker exec ut-api env | grep MONGODB

1. Check network connectivity:

# From API container
docker exec ut-api ping -c 2 mongodb
docker exec ut-api nc -zv mongodb 27017

1. Restart dependent services:

docker compose restart api workers

SSH Tunnel Connection Problems

See the Troubleshooting SSH Tunnel Connections section under "Connecting to MongoDB."

Getting Help

If you encounter issues not covered here:

1. Check container logs:

docker logs ut-mongodb --tail 100
docker logs ut-mongodb-backup --tail 100

1. Check system resources:

df -h                    # Disk space
free -h                  # Memory
docker stats --no-stream # Container resource usage

1. Verify network connectivity:

docker network inspect ut-backend-network
docker exec ut-api ping -c 2 mongodb

1. Review backup container documentation:

• tiredofit/docker-db-backup

• MongoDB Backup Methods

1. Contact your system administrator with:

• Error messages from logs

• Steps you've already tried

• Output from diagnostic commands above

---

Quick Reference

Common Commands

# Backups
docker exec ut-mongodb-backup backup-now                              # Trigger backup
docker exec ut-mongodb-backup ls -lh /backup                          # List backups
docker cp ut-mongodb-backup:/backup ./mongodb-backups                 # Copy to host

# Restore (Interactive)
docker exec -it ut-mongodb-backup restore                             # Start wizard

# Restore (Manual)
docker exec ut-mongodb-backup mongorestore \
  --host mongodb --port 27017 -u mongoadmin -p <password> \
  --authenticationDatabase admin \
  --archive=/backup/<file>.archive.gz --gzip --drop

# Connect via SSH Tunnel
ssh -L 27017:localhost:27017 [email protected] -N      # From remote
# Then: mongodb://mongoadmin:password@localhost:27017/?authSource=admin

# MongoDB Shell
docker exec -it ut-mongodb mongosh -u mongoadmin -p <password> \
  --authenticationDatabase admin

# Health Checks
docker compose ps mongodb                                              # Container status
docker logs ut-mongodb --tail 50                                       # Recent logs
docker exec ut-mongodb mongosh --eval "db.adminCommand('ping')"       # Test connection

# Maintenance
docker system df -v | grep mongodb                                     # Check volumes
docker volume inspect ut-mongodb-data                                  # Volume details

Important Files and Directories

Item	Location	Purpose
Compose file	compose.yaml	Service definitions
Environment	.env	Credentials and config
MongoDB data	/var/lib/docker/volumes/ut-mongodb-data/_data/	Live database
Backup archives	/var/lib/docker/volumes/ut-mongodb-backups/_data/	Backup files
Container logs	View with docker logs <container>	Troubleshooting

Default Settings

Setting	Value	Notes
MongoDB Version	8.2	See compose.yaml
Backup Time	15:20 (3:20 PM)	BACKUP_BEGIN=1520
Backup Frequency	Daily	BACKUP_INTERVAL=1440
Backup Retention	30 days	CLEANUP_TIME=43200
Compression	Gzip (level 6)	GZ format
Port	127.0.0.1:27017	Localhost only
Auth Database	admin	Required for auth

---

Security Best Practices

1. Strong passwords:

   1. Use at least 16 characters with mixed case, numbers, and symbols

   2. Change default password immediately after deployment

   3. Store credentials securely (password manager, secrets vault)

2. Network isolation:

   1. Keep MongoDB on localhost-only binding (127.0.0.1:27017)

   2. Use SSH tunnels for remote access, not direct port exposure

   3. Internal Docker networks provide isolation between services

3. Backup security:

   1. Store backup archives in encrypted storage for offsite copies

   2. Restrict access to backup volume/directory

   3. Test restore procedures regularly to ensure backups work

4. Access control:

   1. Use separate MongoDB users for different services (not just mongoadmin)

   2. Grant minimum required privileges per user

   3. Rotate credentials periodically (e.g., every 90 days)

5. Monitoring:

   1. Review MongoDB logs regularly for unusual activity

   2. Set up alerts for failed authentication attempts

   3. Monitor disk usage to prevent backup volume from filling

6. Updates:

   1. Keep MongoDB container image updated for security patches

   2. Review backup container updates from tiredofit repository

   3. Test updates in non-production environment first

---

Additional Resources

• MongoDB Official Documentation: https://www.mongodb.com/docs/manual/

• MongoDB Backup Guide: https://www.mongodb.com/docs/manual/core/backups/

• tiredofit/docker-db-backup: https://github.com/tiredofit/docker-db-backup

• MongoDB Compass Download: https://www.mongodb.com/products/compass

• UnderstandTech Main README: See README.md in repository root