Skip to main content

Database Architecture & Connection Pooling

The persistence layer of veridex-hitech-web-application is powered by MongoDB 6.0+ accessed through Mongoose 9.6.1. The database architecture is engineered for zero lead loss, high connection concurrency, sub-50ms query responses, and automatic document lifecycle management.

1. Mongoose Connection Architecture (src/lib/db/mongoose.ts)​

In Next.js development (with Hot Module Replacement) and containerized production environments, requests can invoke server code across multiple asynchronous worker threads. Repeatedly creating connections exhausting MongoDB cluster limits (Atlas Connection Limit Exceeded) is prevented by a cached global connection singleton:

import mongoose from "mongoose";

interface MongooseCache {
  conn: typeof mongoose | null;
  promise: Promise<typeof mongoose> | null;
}

declare global {
  var mongoose: MongooseCache | undefined;
}

let cached: MongooseCache = global.mongoose || { conn: null, promise: null };

if (!global.mongoose) {
  global.mongoose = cached;
}

export async function connectDB(): Promise<typeof mongoose> {
  const uri = process.env.DATABASE_URL;
  if (!uri) {
    throw new Error("DATABASE_URL environment variable is missing.");
  }

  if (cached.conn) {
    return cached.conn;
  }

  if (!cached.promise) {
    const opts: mongoose.ConnectOptions = {
      bufferCommands: false,
      maxPoolSize: 10,
      serverSelectionTimeoutMS: 5000,
      socketTimeoutMS: 45000,
    };

    cached.promise = mongoose.connect(uri, opts).then((instance) => {
      return instance;
    });
  }

  try {
    cached.conn = await cached.promise;
  } catch (error) {
    cached.promise = null;
    throw error;
  }

  return cached.conn;
}

Connection Pool Configuration Parameters​

  • maxPoolSize: 10: Restricts the maximum number of simultaneous socket connections per container worker, preventing cluster starvation.
  • bufferCommands: false: Disables command buffering when disconnected, returning fast fail-closed errors instead of hanging request threads.
  • serverSelectionTimeoutMS: 5000: Fails quickly within 5 seconds if primary replica set nodes are unreachable.
  • socketTimeoutMS: 45000: Drops stale, broken sockets across cloud firewalls.

2. Model Recompilation Safety (HMR Guard)​

During Next.js development, files are re-evaluated frequently on code edits. Standard Mongoose declarations (model("User", UserSchema)) throw OverwriteModelError: Cannot overwrite 'User' model once compiled.

Every model in src/lib/db/models/ implements the defensive export guard:

// Pattern: Defensive Model Compilation Guard
export const User = models.User || model<IUser>("User", UserSchema);

3. Indexing Taxonomy & Performance Optimization​

The application defines strategic compound, unique, sparse, and TTL indexes across its collections:

3.1 Time-To-Live (TTL) Automatic Eviction Indexes​

MongoDB background threads scan TTL indexes every 60 seconds and automatically purge documents past their expiration timestamps:

  • ActiveVisitor (lastHeartbeat: 1, expireAfterSeconds: 300): Automatically evicts inactive visitor sessions after 5 minutes of inactivity.
  • RateLimit (resetAt: 1, expireAfterSeconds: 0): Immediately purges expired sliding-window rate limit counters when resetAt < now.
  • VerificationToken (expires: 1, expireAfterSeconds: 0): Discards single-use password reset and email activation tokens.

3.2 Unique & Sparse Indexes​

  • User:
    • email: 1 (unique: true, lowercase: true): Enforces unique account identities.
    • referralCode: 1 (unique: true, sparse: true): Allows users without referral codes while guaranteeing uniqueness for affiliates.
  • HtiLead:
    • externalId: 1 (sparse: true): Supports idempotent upserts from Falcon CRM.
  • MetaLead:
    • leadgenId: 1 (unique: true): Guarantees that Facebook/Instagram Lead Ads webhook duplicates are ignored.
  • EventCampaign:
    • slug: 1 (unique: true): Powers vanity short URLs (/e/:slug).
  • Blog:
    • slug: 1 (unique: true): Guarantees canonical article URLs.
  • DailyVisit:
    • date: 1 (unique: true): Accelerates day-by-day traffic aggregation (YYYY-MM-DD).

3.3 Compound Query Indexes​

  • Lead: { email: 1, status: 1 } and { email: 1, createdAt: -1 }: Delivers sub-10ms lookup times for customer portal draft resumption.
  • BulkEmailRecipientChunk: { campaignId: 1, chunkIndex: 1 } (unique: true): Powers fast sequential pagination through mass recipient lists.

4. Mitigation of MongoDB 16 MB BSON Document Limits​

MongoDB enforces an absolute document size limit of 16 MB (16,777,216 bytes). In two areas of this application, data volume could potentially breach this limit without defensive architectural patterns:

4.1 Mass Email Recipient Lists (BulkEmailRecipientChunk)​

A bulk email blast to 50,000+ policyholders storing recipient objects ({ email, status, error }) inside a single BulkEmailHistory document would easily exceed 16 MB.

  • Solution: The platform partitions recipients into discrete chunk documents (BulkEmailRecipientChunk) storing up to 1,000 recipients per chunk.
  • The parent BulkEmailHistory stores only campaign metadata, summary metrics (totalRecipients, sentCount, failedCount), and sample records.

4.2 Inline Rich-Text Media Offloading (Signature Collection)​

Embedding high-resolution base64 images directly into Tiptap blog articles or email templates inflates document size by ~33%, degrading database query speeds and risking BSON limits.

  • Solution: When an image is pasted or dropped into Tiptap, uploadInlineEmailImageAction offloads the image binary into the dedicated Signature collection.
  • The editor replaces the multi-megabyte base64 string with a lightweight URL: /api/signatures/[id].
  • The public route /api/signatures/[id] streams the binary over HTTPS with immutable caching headers (Cache-Control: public, max-age=31536000, immutable).