UnderstandTech Platform Logging Guide

Overview

The UnderstandTech platform deployed on NVIDIA DGX systems generates logs from multiple containerized services. This guide teaches you how to manage logs for the UnderstandTech platform effectively. You'll learn two complementary approaches: viewing real-time logs for immediate troubleshooting (through Portainer's web interface or Docker commands), and automatically archiving logs to disk for long-term analysis and compliance. The archival system we'll set up captures daily snapshots of all container logs, compresses them to save space, and automatically removes old archives after a configurable retention period.

Target Audience: System administrators and DevOps engineers managing UnderstandTech deployments

Prerequisites:

• Root or sudo access to the DGX host system

• UnderstandTech stack deployed and running

• Basic familiarity with Linux command line

---

Table of Contents

1. Log Architecture

2. Real-Time Log Access

3. Automated Log Archival

4. Log Analysis and Search

---

1. Log Architecture

Container Logging Strategy

Docker uses what's called a "logging driver" to determine how these logs are captured and stored. The UnderstandTech platform uses the JSON file logging driver, which is Docker's default. This driver writes logs to JSON-formatted files located in /var/lib/docker/containers/<container-id>/, with one log file per container. Each line in these files contains the log message along with metadata like timestamps and which stream (stdout or stderr) it came from.

The Log Rotation Problem

Without any intervention, Docker would keep appending to these log files forever, eventually filling up your disk. To prevent this, the UnderstandTech stack is configured with automatic log rotation. Look at any service definition in the compose file and you'll see this configuration:

logging:
  driver: json-file
  options:
    max-size: "50m"    # Maximum size per log file
    max-file: "5"      # Keep 5 rotated files (250MB total per container)

This means Docker will automatically start a new log file when the current one reaches 50MB, keeping only the 5 most recent files. With these settings, each container can accumulate up to 250MB of logs (5 files × 50MB) before Docker starts deleting the oldest file to make room for new logs. This automatic rotation happens entirely within Docker - you don't have to do anything to enable it.

Service Breakdown

The platform consists of multiple services, each generating distinct log streams:

Service	Container Name	Log Focus	Volume
Caddy	ut-caddy	HTTP requests, TLS certificates, reverse proxy events	Low
Frontend	ut-frontend	Healthchecks	Low
API	ut-api	FastAPI requests, authentication, business logic	High
API-Customer	ut-api-customer	REST API requests, integrations	Medium
Workers	understandtech-workers-*	Background job processing, RQ task execution	Medium
Workers-Customer	understand-tech-workers-customer-*	REST API worker-specific background tasks	Medium
LLM	ut-llm	Model inference, GPU operations, Ollama events	High
MongoDB	ut-mongodb	Database operations, queries, authentication	Medium
Redis	ut-redis	Cache operations, queue management	Low

Log Retention Strategy

The UnderstandTech platform uses a two-tier approach to logging. The first tier is Docker's automatic rotation, which gives you immediate access to recent logs through docker logs commands or Portainer's web interface. These are the "active logs" that are always available for troubleshooting issues that are happening right now or recently happened.

The second tier is the automated archival system we'll set up with the ut-logs-archive script. Every day at 2 AM (configurable), this script captures the previous 24 hours of logs from all containers, compresses them with gzip, and stores them in an organized directory structure. These become your "historical logs" - a permanent record that survives container restarts and provides the long retention needed for compliance, trend analysis, or investigating issues that happened days or weeks ago.

Active Logs (Docker):

• Location: /var/lib/docker/containers/<container-id>/

• Retention: Last 250MB per container (5 files × 50MB)

• Access: Real-time via docker logs or Portainer

Archived Logs:

• Location: /var/log/understandtech/YYYY-MM/YYYY-MM-DD/

• Retention: Configurable (default: 365 days)

• Format: Gzip-compressed daily snapshots

• Access: Manual extraction and analysis

---

2. Real-Time Log Access

Using Docker CLI

View recent logs from a specific service:

# Last 100 lines
docker logs --tail 100 ut-api

# Last 10 minutes
docker logs --since 10m ut-llm

# Follow logs in real-time
docker logs -f ut-api

# Include timestamps
docker logs -t ut-mongodb
Multi-service log streaming:
# All services at once
docker compose -f /opt/understandtech/compose.yaml logs -f

# Specific services
docker compose -f /opt/understandtech/compose.yaml logs -f api llm workers

# Last 50 lines from each service
docker compose -f /opt/understandtech/compose.yaml logs --tail 50

Using Portainer Web Interface

Portainer provides a (on the dgx host) browser-based log viewer accessible at https://localhost:9443

Steps:

1. Open Portainer in Firefox

2. Navigate to Containers in the left sidebar

3. Click on the container name (e.g., ut-api)

4. Select Logs from the top menu

5. Use the interface controls:

   1. Auto-refresh: Enable to stream logs in real-time

   2. Timestamps: Toggle timestamp display

   3. Lines: Adjust number of lines shown (100-2000)

   4. Search: Filter logs with keyword search

   5. Download: Export current view as text file

Portainer provides a (on the dgx host) browser-based log viewer accessible at https://localhost:9443

Quick Diagnostic Commands

Check service health:

docker ps --format "table {{.Names}}\t{{.Status}}\t{{.Ports}}"

Follow all logs with container name prefix:

docker compose -f /opt/understandtech/compose.yaml logs -f --tail 20 | grep -E "(ut-api|ut-llm|ut-mongodb)"

Search for errors across all services:

docker compose -f /opt/understandtech/compose.yaml logs --tail 1000 | grep -i error

---

3. Automated Log Archival

The ut-logs-archive.sh Script

The ut-logs-archive.sh script provides automated daily log archival with compression and configurable retention policies. The script is designed around a simple concept: once per day, capture the last 24 hours of logs from every UnderstandTech container, compress them, and store them in an organized archive. It then cleans up any archives that have exceeded your retention period.

The script has several command modes. Running it with no arguments performs an immediate archive (useful for testing). The --install command sets up a cron job so the archival happens automatically each night. The --status command shows you statistics about your archive (how much space it's using, how many days are archived, etc.), and --cleanup lets you manually trigger the deletion of old archives if needed.

Installation

1

Download and make executable:

cd understandtech
chmod +x ut-logs-archive.sh

2

Configure script variables:

Now let's look at the configuration section at the top of the script. Open it with any text editor/VSCode and you'll see several variables you can customize:

ARCHIVE_DIR="/var/log/understandtech"  # Where archives are stored 

RETENTION_DAYS=365  # Keep archives for one year 


CONTAINER_PREFIX="ut-"  # Archive containers starting with "ut-" 

WORKER_PREFIX="understandtech-workers" # Also archive worker containers 

ARCHIVE_HOUR="02:00" # Run archival at 2 AM daily

ARCHIVE_DIR

The ARCHIVE_DIR setting determines where your compressed log archives will be saved. The default location (/var/log/understandtech) follows Linux filesystem conventions, which is good for consistency. Make sure this directory is on a filesystem with enough space - for a typical deployment with default settings, expect about 1-2GB per month of archives.

RETENTION_DAYS

The RETENTION_DAYS setting controls how long archives are kept before automatic deletion. The default of 365 days (one year) balances storage costs with the needs of troubleshooting and compliance. If you're in a regulated industry that requires longer retention, you might increase this to 2555 days (seven years) or more. The script will handle the extra storage just fine, as long as you have the disk space.

CONTAINER_PREFIX

The container prefix settings tell the script which containers to archive. By default, it captures anything starting with ut- (all your main services) plus the worker containers (which have a slightly different naming pattern). You shouldn't need to change these unless you've customized your container names.

3

Create archive directory

sudo mkdir -p /var/log/understandtech
sudo chown $(whoami):$(whoami) /var/log/understandtech

4

Test manual archive:

./ut-logs-archive.sh

Expected output:

[INFO] Starting log archive for 2026-02-03
[INFO] Archive complete: 9 archived, 0 skipped, 0 failed
[INFO] Location: /var/log/understandtech/2026-02/2026-02-03

5

Install automated daily cron job:

./ut-logs-archive.sh --install

This creates a cron entry that runs daily at 2:00 AM, exporting the previous 24 hours of logs from all ut-* containers.

Script Commands:

Command	Purpose
./ut-logs-archive.sh	Run manual archive (exports last 24h)
./ut-logs-archive.sh --install	Install daily cron job
./ut-logs-archive.sh --uninstall	Remove cron job
./ut-logs-archive.sh --status	Show archive statistics and cron status
./ut-logs-archive.sh --cleanup	Manually remove archives older than retention period
./ut-logs-archive.sh --help	Display usage information

Archive Directory Structure

When you set up automated archival, the script creates a hierarchical directory structure organized by month and day. Each day's logs live in their own directory, named with the date in YYYY-MM-DD format. This organization makes it incredibly easy to find logs from a specific time period - you just navigate to the right date folder.

Inside each day's directory, you'll find one compressed file per container. The filenames match the container names (like ut-api.log.gz or ut-workers-1.log.gz), so you can quickly identify which service's logs you're looking at. The gzip compression typically reduces log file sizes by 90% or more, meaning your one-year retention policy won't fill up your disk.

/var/log/understandtech/
├── 2026-01/
│   ├── 2026-01-15/
│   │   ├── ut-api.log.gz
│   │   ├── ut-api-customer.log.gz
│   │   ├── ut-caddy.log.gz
│   │   ├── ut-frontend.log.gz
│   │   ├── ut-llm.log.gz
│   │   ├── ut-mongodb.log.gz
│   │   ├── ut-redis.log.gz
│   │   └── ut-workers-*.log.gz
│   ├── 2026-01-16/
│   │   └── ...
│   └── ...
├── 2026-02/
│   ├── 2026-02-01/
│   │   └── ...
│   └── 2026-02-03/
│       └── ...
└── ...

Monitoring Archive Health

View archive status:

./ut-logs-archive.sh --status

Output example:

UnderstandTech Log Archive Status
══════════════════════════════════════════════════════════════

Archive location:  /var/log/understandtech
Retention period:  365 days
Container prefix:  ut-*

Archive Statistics:
  Total size:    3.2G
  Days archived: 45
  Total files:   405
  Date range:    2025-12-20 → 2026-02-03

Archives by Month:
  2025-12      12 days    108 files     780M
  2026-01      31 days    279 files     2.1G
  2026-02       3 days     27 files     240M

✓ Cron job is installed
  0 2 * * * /opt/understandtech/ut-logs.sh >> /var/log/ut-logs-archive.log 2>&1

Check cron execution log:

tail -f /var/log/ut-logs-archive.log

Automatic Cleanup

The script automatically removes archives older than RETENTION_DAYS (default: 365) during each run. Manual cleanup can be triggered by running:

./ut-logs-archive.sh --cleanup

This prompts for confirmation before deletion.

---

4. Log Analysis and Search

Accessing Archived Logs

View a specific archived log:

# Decompress and display with pagination
zcat /var/log/understandtech/2026-02/2026-02-03/ut-api.log.gz | less

# View last 100 lines
zcat /var/log/understandtech/2026-02/2026-02-03/ut-api.log.gz | tail -100

Search for specific patterns:

# Search for errors in a single service on a specific day
zgrep -i "error" /var/log/understandtech/2026-02/2026-02-03/ut-api.log.gz

# Search across all services for a specific day
zgrep -i "authentication failed" /var/log/understandtech/2026-02/2026-02-03/*.log.gz

# Search entire month for a pattern
zgrep -i "connection refused" /var/log/understandtech/2026-02/*/ut-llm.log.gz

# Case-sensitive search with line numbers
zgrep -n "Exception" /var/log/understandtech/2026-02/2026-02-03/ut-workers*.log.gz

Extract logs for offlline analysis:

# Create export directory
mkdir -p ~/log-export

# Extract all logs from a specific day
gunzip -c /var/log/understandtech/2026-02/2026-02-03/*.log.gz > ~/log-export/2026-02-03-all.log

# Extract single service
gunzip -c /var/log/understandtech/2026-02/2026-02-03/ut-api.log.gz > ~/log-export/api-2026-02-03.log

Common Search Patterns

Authentication issues

zgrep -E "(authentication|unauthorized|forbidden)" /var/log/understandtech/2026-02/*/ut-api.log.gz

Database connection problems:

GPU/LLM errors:

Worker task failures:

zgrep -i "task.*failed" /var/log/understandtech/2026-02/*/ut-workers*.log.gz

HTTP 5xx errors:

Advanced Analysis

Count errors by service:

for service in api llm mongodb workers; do
    count=$(zgrep -ci "error" /var/log/understandtech/2026-02/2026-02-03/ut-${service}*.log.gz 2>/dev/null || echo "0")
    echo "$service: $count errors"
done

Timeline of events:

# Extract timestamped errors from multiple services
zgrep -h "ERROR" /var/log/understandtech/2026-02/2026-02-03/*.log.gz | sort

Export to CSV for spreadsheet analysis:

# Extract API errors with timestamps
zgrep "ERROR" /var/log/understandtech/2026-02/*/ut-api.log.gz | \
    sed 's/\t/,/g' > ~/api-errors.csv

---