Logging

The eTeamups Platform implements a multi-layer logging strategy that captures application events, API requests, and reverse proxy traffic. This guide covers all logging components, their configuration, and tools for analysis.

Logging Architecture Overview

The platform has three distinct logging layers:

Application Logging (Winston)

Configuration

The application logger is defined in libs/utility/src/Logger.ts and is shared across all microservices. It uses the Winston logging library with three transports:

1. Console transport – Colorized, human-readable output for development and container stdout.
2. File transport (error.log) – Captures only error-level messages in JSON format.
3. File transport (combined.log) – Captures all log levels in JSON format.
4. OpenTelemetry transport – Sends log records to SigNoz via the @opentelemetry/winston-transport package.

Log Format

All file-based logs use structured JSON with the following fields:

{
  "level": "info",
  "message": "Server running on Port 9000",
  "timestamp": "2024-11-27 14:30:00",
  "service": "zeswa-platform"
}

The service field defaults to the value of the OTEL_SERVICE_NAME environment variable (or eteamups-platform-meta if unset).

Error logs include a stack field with the full stack trace.

Log Level

The log level is controlled by the LOG_LEVEL environment variable (docker.env). Valid levels in order of severity:

error > warn > info > http > verbose > debug > silly

The default is info.

Logger API

The Logger class provides static methods that can be imported from libs/utility/src/Logger.ts:

import { Logger } from "libs/utility/src/Logger";

Logger.info("User registered successfully", { userId: "abc123" });
Logger.warn("Rate limit approaching", { ip: "192.168.1.1" });
Logger.error(new Error("Database connection failed"));
Logger.debug("Query result", { count: 42 });

Trace Correlation

When OpenTelemetry instrumentation is active, every Winston log record automatically includes trace_id and span_id fields. This allows you to:

1. Search for a log message in SigNoz.
2. Click through to the associated distributed trace.
3. See the exact request context that generated the log.

API Request Logging (MongoDB)

How It Works

The BasicMiddleware class in services/common/src/BasicMiddleware.ts intercepts every incoming HTTP request and logs it to MongoDB after the response completes.

The middleware flow:

1. Pre-hook (pre()) – Records the start time when a request arrives.
2. Request capture (on()) – Extracts request metadata (URL, method, headers, body, query, params, remote address).
3. Post-hook (post()) – Fires on the response close event. Calculates processing time and writes the log entry to MongoDB.

Logged Fields

Each API log entry in the apilogs collection contains:

Caller Identification

Every request is tagged with a callerApp value identifying the source application. Detection priority:

1. x-app-source header (most reliable) — client sends admin or hub explicitly
2. User-Agent — requests from Postman (PostmanRuntime/...) are auto-tagged as postman
3. Origin / Referer — matched against ADMIN_URL and HUB_URL env vars
4. Fallback — unknown

Sending x-app-source from client apps

Admin portal — add to every request:

// Centralised fetch wrapper
export const apiClient = (path: string, options: RequestInit = {}) =>
  fetch(`${process.env.NEXT_PUBLIC_API_URL}${path}`, {
    ...options,
    headers: { 'x-app-source': 'admin', ...(options.headers ?? {}) },
  });

// Or Axios default
api.defaults.headers.common['x-app-source'] = 'admin';

Hub — same pattern with value hub:

headers: { 'x-app-source': 'hub', 'Authorization': `Bearer ${token}` }

Postman — no action needed; auto-detected from User-Agent.

Querying API Logs

Connect to MongoDB and query the apilogs collection:

// Find slow requests (> 1 second)
db.apilogs.find({ processingTime: { $gt: 1000 } }).sort({ startTime: -1 });

// Find all POST requests to auth endpoints
db.apilogs.find({ method: "POST", url: { $regex: /^\/auth/ } });

// Count requests by method in the last 24 hours
db.apilogs.aggregate([
  { $match: { startTime: { $gte: new Date(Date.now() - 86400000) } } },
  { $group: { _id: "$method", count: { $sum: 1 } } }
]);

// Find requests from a specific IP
db.apilogs.find({ remoteAddress: "192.168.1.100" }).sort({ startTime: -1 });

// Average processing time by URL
db.apilogs.aggregate([
  { $group: { _id: "$url", avgTime: { $avg: "$processingTime" }, count: { $sum: 1 } } },
  { $sort: { avgTime: -1 } },
  { $limit: 20 }
]);

// All requests from Admin portal
db.apilogs.find({ callerApp: "admin" }).sort({ startTime: -1 });

// All requests from Hub
db.apilogs.find({ callerApp: "hub" }).sort({ startTime: -1 });

// Breakdown by caller
db.apilogs.aggregate([
  { $group: { _id: "$callerApp", count: { $sum: 1 } } },
  { $sort: { count: -1 } }
]);

// Slow admin requests (> 500ms)
db.apilogs.find({ callerApp: "admin", processingTime: { $gt: 500 } })
  .sort({ processingTime: -1 });

Nginx Logging

Nginx provides the outermost logging layer, capturing all traffic that reaches the reverse proxy.

Log Formats

Four log formats are defined in nginx/nginx.conf:

1. Standard Format (main)

Basic Apache-style combined log format. Compatible with most log analysis tools.

$remote_addr - $remote_user [$time_local] "$request" $status $body_bytes_sent
"$http_referer" "$http_user_agent" "$http_x_forwarded_for"

2. Detailed Format (detailed)

Enhanced format with upstream timing information. This is the default format for access logs.

$remote_addr - $remote_user [$time_local] "$request"
$status $body_bytes_sent "$http_referer" "$http_user_agent"
"$http_x_forwarded_for" rt=$request_time uct="$upstream_connect_time"
uht="$upstream_header_time" urt="$upstream_response_time"
cs=$upstream_cache_status host="$host" proto="$scheme"

Key timing fields:

3. JSON Format (json_combined)

Structured JSON output suitable for ingestion by log aggregation tools (ELK, Loki, etc.).

{
  "time_local": "27/Nov/2024:14:30:00 +0000",
  "remote_addr": "192.168.1.1",
  "request": "GET /auth/health HTTP/1.1",
  "status": "200",
  "request_time": "0.003",
  "upstream_response_time": "0.002"
}

4. Security Format (security)

Includes the request body for security audit purposes. Used for authentication endpoints and suspicious request logging.

$remote_addr - $remote_user [$time_local] "$request" $status
"$http_user_agent" "$http_referer" "$http_x_forwarded_for" "$request_body"

Log Files

Nginx writes to multiple log files, organized by purpose and service:

General Logs

Per-Service Logs

Per-Virtual-Host Logs

Security and Error Logs

Rate Limiting

Nginx defines two rate-limiting zones:

Both zones allow burst traffic with the nodelay directive. Excess requests receive HTTP 429 Too Many Requests.

Bot Detection

Nginx uses a map directive to detect bots based on the User-Agent header. Requests matching patterns like bot, crawler, spider, scanner, curl, or wget are flagged and logged separately to bot-requests.log.

Log Rotation

Automated Rotation

Log rotation runs daily at midnight via a cron job. Set it up with:

./scripts/setup-log-rotation.sh

This adds the following cron entry:

0 0 * * * cd /path/to/project && ./scripts/rotate-logs.sh >> logs/rotation.log 2>&1

Manual Rotation

Trigger rotation manually at any time:

./scripts/rotate-logs.sh

Rotation Process

The scripts/rotate-logs.sh script performs the following steps:

1. Copy logs from the Nginx container – Logs are copied from /var/log/nginx/ inside the container to the local logs/ directory.
2. Rotate each log file – Non-empty log files are moved to a dated backup (e.g., access-2024-11-27-000000.log).
3. Compress rotated logs – Backed-up files are compressed with gzip (.log.gz).
4. Signal Nginx – Sends nginx -s reopen to the container so Nginx starts writing to fresh log files.
5. Rotate application logs – PM2 logs (if present) and any other logs in the logs/ directory are also rotated.
6. Clean up old logs – Compressed logs older than the retention period are deleted.
7. Report statistics – Outputs backup directory size, archive count, and oldest/newest archived log.

Rotation Configuration

Directory Structure After Rotation

logs/
  backup/
    nginx/
      access-2024-11-27-000000.log.gz
      auth-service-2024-11-27-000000.log.gz
      error-2024-11-27-000000.log.gz
      ...
    application/
      app-combined-2024-11-27-000000.log.gz
      app-error-2024-11-27-000000.log.gz
      ...
  rotation.log    # Log rotation execution log

Log Analysis with log-monitor.sh

The scripts/log-monitor.sh script provides a comprehensive CLI tool for analyzing Nginx logs.

Usage

./scripts/log-monitor.sh [OPTION]

Running without options shows statistics and recent errors.

Available Flags

Examples

# Monitor logs in real time
./scripts/log-monitor.sh -r

# Check for recent errors
./scripts/log-monitor.sh -e

# Analyze authentication patterns
./scripts/log-monitor.sh -a

# Review performance metrics
./scripts/log-monitor.sh -p

# Check for security threats
./scripts/log-monitor.sh -t

# See disk usage and cleanup suggestions
./scripts/log-monitor.sh -c

Common Log Queries

Nginx Access Log Queries

# Top 10 IP addresses by request count
awk '{print $1}' logs/access.log | sort | uniq -c | sort -nr | head -10

# HTTP status code distribution
awk '{print $9}' logs/access.log | sort | uniq -c | sort -nr

# Requests with response time > 1 second
grep "rt=" logs/access.log | awk '{match($0, /rt=([0-9.]+)/, rt); if (rt[1] > 1) print}'

# Top 10 slowest requests
grep "rt=" logs/access.log | awk '{match($0, /rt=([0-9.]+)/, rt); print rt[1], $0}' | sort -nr | head -10

# Requests per hour
awk '{gsub(/\[|\]/, "", $4); split($4, dt, ":"); hours[dt[2]]++} END {for (h in hours) printf "%02d:00 - %d requests\n", h, hours[h]}' logs/access.log | sort

Security Log Queries

# Recent security alerts
tail -50 logs/security-alerts.log

# Failed authentication attempts (401/403)
grep -E "401|403" logs/auth-security.log | tail -20

# IP addresses with most failed auth attempts
grep -E "401|403" logs/auth-security.log | awk '{print $1}' | sort | uniq -c | sort -nr | head -10

# Requests to suspicious file extensions
grep -E "\.(php|asp|aspx|jsp)" logs/security-alerts.log | wc -l

# Monitor security alerts in real time
tail -f logs/security-alerts.log

Error Log Queries

# Recent server errors
tail -20 logs/server-errors.log

# Count 404 errors by URL
awk '{print $7}' logs/404-errors.log | sort | uniq -c | sort -nr | head -10

# Recent upstream errors
grep "upstream" logs/error.log | tail -20

Security Log Monitoring

What Gets Logged

The Nginx security configuration automatically detects and logs:

• Suspicious file extensions – Requests for .php, .asp, .aspx, .jsp files are blocked with 403 and logged to security-alerts.log.
• Hidden file access – Requests matching /. (e.g., /.env, /.git) are denied and logged to security-alerts.log.
• Bot traffic – Requests from user agents containing bot, crawler, spider, scanner, curl, or wget are logged to bot-requests.log.
• Auth events – All authentication requests are double-logged: once to auth-service.log (detailed) and once to auth-security.log (security format with request body).
• Rate limit violations – Requests exceeding rate limits receive 429 and appear in the standard access logs.

Recommended Monitoring Routine

1. Daily – Run ./scripts/log-monitor.sh -t to check for new security threats.
2. Daily – Run ./scripts/log-monitor.sh -a to review authentication patterns and look for brute-force attempts.
3. Weekly – Run ./scripts/log-monitor.sh -b to review bot activity and decide whether to block persistent scanners.
4. On incident – Use ./scripts/log-monitor.sh -r for real-time monitoring while investigating.

Docker Container Log Access

Docker container logs (stdout/stderr) can be accessed independently from the file-based logs:

# View logs for a specific service
docker compose logs auth-service

# Follow logs in real time
docker compose logs -f auth-service

# View logs for all services
docker compose logs -f

# View last 100 lines
docker compose logs --tail 100 auth-service

# View logs since a specific time
docker compose logs --since "2024-11-27T14:00:00" auth-service

The scripts/view-logs.sh script provides a convenience wrapper for viewing service logs.

Log Viewing Scripts

# View service logs (convenience wrapper)
./scripts/view-logs.sh

# Manual log rotation
./scripts/rotate-logs.sh

# Set up automated log rotation
./scripts/setup-log-rotation.sh