Skip to main content

Production Troubleshooting & Incident Runbook

This runbook provides actionable diagnostic steps and remediation procedures for engineering on-call responders and DevOps administrators managing the HiTech Insurance production application.

1. Database Connection & Mongoose Pool Failures​

Symptoms​

  • /api/health returns status degraded or unhealthy.
  • Log entries display MongooseServerSelectionError: connection timed out or MongoNetworkTimeoutError.
  • Server Actions hang or throw 500 Internal Server Error.

Root Cause Analysis & Diagnostic Steps​

  1. Network Connectivity & IP Access List: Verify the Azure App Service outbound IP addresses are allowlisted in your MongoDB Atlas Network Access security group.
  2. Connection String Syntax: Verify DATABASE_URL matches standard MongoDB URI formatting: 
    mongodb+srv://<username>:<password>@cluster0.example.mongodb.net/hitech_prod?retryWrites=true&w=majority
  3. Mongoose Cached Connection Check: Inspect src/lib/db/mongoose.ts. Ensure that concurrent serverless functions reuse global.mongoose.conn rather than opening redundant connections that exhaust the pool.

Remediation​

  • Run mongosh from an authorized network jumpbox to confirm cluster availability: 
    mongosh "$DATABASE_URL" --eval "db.adminCommand('ping')"
  • If connections are exhausted, scale up MongoDB Atlas cluster tier or increase maxPoolSize in src/lib/db/mongoose.ts: 
    const opts = {
      bufferCommands: false,
      maxPoolSize: 20, // Adjust connection ceiling
    };

2. Lead Forwarding Outages & CRM Sync Failures​

Symptoms​

  • Leads are submitted on the frontend, but agency brokers report missing leads in the external CRM.
  • Records accumulate in the crm_failed_leads MongoDB collection.

Root Cause Analysis & Diagnostic Steps​

  1. Navigate to Admin Control Center: Log in as Super Admin and open Tab 17: CRM Retry Queue (/admin?tab=crm-retries).
  2. Review Error Telemetry: Inspect errorDetails, HTTP status code (e.g. 502 Bad Gateway, 401 Unauthorized), and failure timestamp.
  3. Check CRM Credentials: Verify HTI_CRM_API_URL, HTI_CRM_BASIC_AUTH_USERNAME, and HTI_CRM_BASIC_AUTH_PASSWORD in App Service Configuration.

Remediation​

  • Execute Bulk Resync: Within Tab 17 of /admin, click "Retry All Failed Leads". The system iterates through the retry queue, executing sendLeadToHtiCrm with 500ms jitter between attempts.
  • Inspect Quarantined Leads: Check Tab 18: Quarantined Leads (/admin?tab=quarantine) to verify whether leads were intercepted due to low Google reCAPTCHA v3 bot scores or spam heuristics. Click "Release to Pipeline" to promote valid leads.

3. Sliding-Window Rate Limiting Blocking Legitimate Traffic​

Symptoms​

  • Legitimate users or agency staff report receiving HTTP 429 Too Many Requests when testing or performing high-frequency lead entry.
  • Response header includes Retry-After: {seconds}.

Root Cause Analysis & Diagnostic Steps​

Rate limit keys are recorded in the sliding_window_rate_limits collection:

  • AUTH_STRICT: 5 requests per 15 minutes.
  • FORMS: 10 requests per 10 minutes.
  • ADMIN_WRITE: 60 requests per minute.
  • LEADS_INGESTION: 120 requests per minute.

Remediation​

To clear an accidental rate-limit lockout for an administrator or testing IP address:

// Connect via mongosh
use hitech_prod;

// Find blocked keys for target IP
db.sliding_window_rate_limits.find({ key: /192\.168\.1\.50/ });

// Clear the rate limit record immediately
db.sliding_window_rate_limits.deleteMany({ key: /192\.168\.1\.50/ });

4. Azure Email Dispatch Failures & SMTP Fallback​

Symptoms​

  • Automated quote confirmations, welcome emails, or password reset messages are delayed or missing.
  • Application logs indicate AzureEmailServiceError: Function timed out.

Root Cause Analysis & Diagnostic Steps​

The platform uses a resilient Dual-Architecture Email Dispatcher:

  1. Primary: Azure Functions Email Microservice via AZURE_EMAIL_FUNCTION_URL and AZURE_EMAIL_FUNCTION_KEY.
  2. Secondary Fallback: Direct Nodemailer SMTP relay via SMTP_HOST, SMTP_PORT, SMTP_USER, and SMTP_PASS.

Remediation​

  • If the Azure Function is experiencing cold starts or outages, the application automatically falls back to Nodemailer SMTP.
  • Verify whether the system is trapped in Test Mode: If EMAIL_TEST_MODE="true", all outgoing communications are diverted away from client recipients and forwarded exclusively to EMAIL_TEST_RECIPIENT.

5. Build-Time Next.js Zod Environment Failures​

Symptoms​

During Docker build or CI execution, npm run build fails with:

Error: ❌ Invalid environment variables:
  DATABASE_URL: Required
  NEXTAUTH_SECRET: Required

Root Cause Analysis​

src/lib/env.ts parses and validates environment variables at build time using strict Zod schemas. If required variables are missing during next build, compilation aborts.

Remediation​

Verify that Dockerfile includes the non-secret build-time placeholders in the builder stage:

ENV NEXT_PUBLIC_APP_URL=https://hitechinsurance.ai
ENV NEXT_PUBLIC_APP_ENV=production
ENV NEXT_PUBLIC_RECAPTCHA_SITE_KEY=6LfDgaQtAAAAACyjOzVYR-527YM2pOrHI3NlwIDr
ENV DATABASE_URL=mongodb://placeholder:27017/placeholder
ENV NEXTAUTH_SECRET=placeholder-build-time-secret
ENV NEXTAUTH_URL=https://build-placeholder.example.com

6. High Memory Footprint & V8 Heap Diagnostics​

Symptoms​

  • App Service container memory exceeds 80% utilization.
  • Node.js throws FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory.

Root Cause Analysis & Diagnostic Steps​

  1. Navigate to /admin/system-health to inspect real-time V8 heap metrics (heapUsed, heapTotal, external, rss).
  2. Heavy memory pressure typically occurs during Bulk Email Suite broadcasts if large CSV lists or high-resolution inline images are loaded in memory simultaneously.

Remediation​

  • The platform mitigates this by chunking recipients into BulkEmailRecipientChunk (BATCH_SIZE = 5) and enforcing 250ms asynchronous delays between chunks.
  • If memory remains elevated, trigger garbage collection or restart the App Service container via the Azure CLI: 
    az webapp restart --resource-group Hightech-Webapp --name hitech-webapp