DatomniX Architecture Documentation

Developed by ETK Technologies

System Overview

DatomniX is a comprehensive multi-tenant SaaS platform for attribution and analytics. Built on a modern microservices architecture, it provides brands with unified tracking across all marketing channels, advanced attribution models, and seamless integrations with major advertising platforms.

---

Core Architecture Principles

1. Multi-Tenancy

• Workspace Isolation: Complete data separation between tenants


• Brand-Level Configuration: Each brand has independent settings and integrations


• Role-Based Access Control: Granular permissions (Super Admin, Owner, Admin, Member)


• Scalable Design: Tenant-aware queries with optimized indexing

2. Event-Driven Processing



• Redis Streams: Real-time event pipeline with guaranteed ordering


• BullMQ Queues: Background job processing with retry logic


• Parallel Consumers: Multiple workers for high-throughput processing (30x performance boost)


• Idempotency: Duplicate event detection and prevention

3. Microservices Architecture



• API Layer: NestJS REST API with Swagger documentation


• Web Dashboard: Next.js 14 with App Router and RSC


• Shared Packages: TypeScript types and configuration


• Independent Deployment: Services can be deployed separately

---

Component Architecture

┌─────────────────────────────────────────────────────────────┐
│                  Client Websites & Apps                     │
│         (Tracking Pixel • GTM • Shopify • Direct API)       │
└────────────────────────────┬────────────────────────────────┘
                             │ HTTPS
                             ↓
┌─────────────────────────────────────────────────────────────┐
│                    DatomniX Web Dashboard                    │
│                       (Next.js 14 App)                       │
│  ┌──────────┐  ┌──────────┐  ┌──────────┐  ┌────────────┐ │
│  │   Auth   │  │Workspaces│  │  Brands  │  │ Dashboard  │ │
│  └──────────┘  └──────────┘  └──────────┘  └────────────┘ │
│  ┌──────────┐  ┌──────────┐  ┌──────────┐  ┌────────────┐ │
│  │  Events  │  │Attribution│  │Integrations│  │  Settings  │ │
│  └──────────┘  └──────────┘  └──────────┘  └────────────┘ │
└────────────────────────────────┬────────────────────────────┘
                                 │ REST API + JWT Auth
                                 ↓
┌─────────────────────────────────────────────────────────────┐
│              NestJS API Core (apps/api)                     │
│  ┌──────────────────────────────────────────────────────┐  │
│  │  📥 Event Ingestion Layer                            │  │
│  │  • Client Pixel Endpoint (/api/v1/track)            │  │
│  │  • Server-to-Server API (/api/v1/events)            │  │
│  │  • WooCommerce Webhook Handler                      │  │
│  │  • Shopify Webhook Handler                          │  │
│  └──────────────────────────────────────────────────────┘  │
│  ┌──────────────────────────────────────────────────────┐  │
│  │  ⚙️ Event Processing Pipeline                        │  │
│  │  • Normalization → Validation → Enrichment          │  │
│  │  • Redis Streams Publisher                          │  │
│  │  • 3 Parallel Consumers (30x faster)                │  │
│  │  • Batch Processing (100 events/batch)              │  │
│  └──────────────────────────────────────────────────────┘  │
│  ┌──────────────────────────────────────────────────────┐  │
│  │  🎯 Attribution Engine                               │  │
│  │  • UTM Parsing & Channel Resolution                 │  │
│  │  • Order Exclusion Rules (tag & email based)        │  │
│  │  • Subscription Classification                      │  │
│  │  • Label Generation for Path Analysis               │  │
│  │  • Last-Click Attribution                           │  │
│  │  • First-Click Attribution                          │  │
│  │  • Linear Attribution                               │  │
│  │  • Time-Decay Attribution                           │  │
│  │  • Position-Based Attribution                       │  │
│  │  • Configurable Lookback Windows (1-90 days)        │  │
│  └──────────────────────────────────────────────────────┘  │
│  ┌──────────────────────────────────────────────────────┐  │
│  │  🔗 Integration Orchestrator                         │  │
│  │  • Meta CAPI (Advanced: match keys, event filters)  │  │
│  │  • TikTok Events API                                │  │
│  │  • Google Analytics 4 (GA4)                         │  │
│  │  • Custom Webhooks                                  │  │
│  └──────────────────────────────────────────────────────┘  │
│  ┌──────────────────────────────────────────────────────┐  │
│  │  📊 Reporting & Analytics                            │  │
│  │  • Real-time Metrics                                │  │
│  │  • Revenue Attribution by Model                     │  │
│  │  • Customer Journey Paths                           │  │
│  │  • Model Comparison Analysis                        │  │
│  │  • Export Center (CSV/JSON with scheduling)         │  │
│  │  • Traffic Source Analysis                          │  │
│  └──────────────────────────────────────────────────────┘  │
└────────────────────────────────┬────────────────────────────┘
                                 │
                                 ↓
┌─────────────────────────────────────────────────────────────┐
│                    Data & Cache Layer                        │
│  ┌──────────────────────┐    ┌──────────────────────────┐  │
│  │   PostgreSQL 14+     │    │      Redis 6+            │  │
│  │  • Users & Auth      │    │  • Event Streams         │  │
│  │  • Workspaces        │    │  • Job Queues            │  │
│  │  • Brands            │    │  • Session Cache         │  │
│  │  • Events            │    │  • Real-time Counters    │  │
│  │  • Attributions      │    └──────────────────────────┘  │
│  │  • Integrations      │                                   │
│  └──────────────────────┘                                   │
└─────────────────────────────────────────────────────────────┘

---

Module Structure

Backend (apps/api/src/)

apps/api/src/
├── main.ts                         # Application bootstrap
├── app.module.ts                   # Root module
├── app.controller.ts               # Root controller
│
├── auth/                           # Authentication & Authorization
│   ├── auth.service.ts             # JWT generation & validation
│   ├── auth.controller.ts          # Login, register, refresh
│   ├── users.service.ts            # User management
│   ├── jwt.strategy.ts             # Passport JWT strategy
│   ├── jwt-auth.guard.ts           # Route protection
│   ├── guards/                     # Custom guards
│   │   ├── roles.guard.ts          # RBAC enforcement
│   │   ├── super-admin.guard.ts    # Super admin access
│   │   └── workspace-access.guard.ts # Workspace isolation
│   └── decorators/                 # Custom decorators
│       ├── current-user.decorator.ts
│       └── roles.decorator.ts
│
├── workspaces/                     # Multi-tenant workspaces
│   ├── workspaces.service.ts       # Workspace CRUD
│   ├── workspaces.controller.ts    # Workspace API
│   └── workspaces.module.ts
│
├── brands/                         # Brand management
│   ├── brands.service.ts           # Brand CRUD & configuration
│   ├── brands.controller.ts        # Brand API endpoints
│   ├── brands.module.ts
│   └── dto/
│       └── create-brand.dto.ts
│
├── ingestion/                      # Event ingestion
│   ├── events.controller.ts        # Event tracking endpoints
│   ├── poller.service.ts           # Background polling
│   ├── poller.processor.ts         # BullMQ job processor
│   └── ingestion.module.ts
│
├── events/                         # Event processing
│   ├── event-normalization.service.ts  # Event standardization
│   └── events.module.ts
│
├── redis/                          # Redis integration
│   ├── redis.service.ts            # Streams & pub/sub
│   ├── redis.controller.ts         # Queue stats API
│   └── redis.module.ts
│
├── attribution/                    # Attribution engine
│   ├── attribution-engine.service.ts    # Attribution logic
│   ├── attribution-consumer.service.ts  # Stream consumer (3 parallel workers)
│   ├── attribution.controller.ts        # Attribution API
│   ├── orders.controller.ts             # Order tracking
│   └── attribution.module.ts
│
├── reporting/                      # Analytics & reporting
│   ├── reporting.service.ts        # Aggregation queries
│   ├── reporting.controller.ts     # Reporting API
│   └── reporting.module.ts
│
├── integrations/                   # Platform integrations
│   ├── base-integration.ts         # Base class for integrations
│   ├── integration-orchestrator.service.ts # Event distribution
│   ├── integrations.controller.ts  # Integration management API
│   ├── integrations.module.ts
│   ├── meta/
│   │   └── meta-integration.ts     # Meta CAPI
│   ├── tiktok/
│   │   └── tiktok-integration.ts   # TikTok Events API
│   ├── ga4/
│   │   └── ga4-integration.ts      # Google Analytics 4
│   ├── shopify/
│   │   └── shopify-integration.ts  # Shopify webhooks
│   ├── woocommerce/
│   │   ├── woocommerce-integration.ts  # WooCommerce webhooks
│   │   └── woocommerce.controller.ts   # Webhook receiver
│   └── custom/
│       └── custom-integration.ts   # Custom HTTP webhooks
│
├── admin/                          # Super admin features
│   ├── admin.service.ts            # Admin operations
│   ├── admin.controller.ts         # Admin API
│   └── admin.module.ts
│
├── docs/                           # Documentation serving
│   ├── docs.controller.ts          # Markdown renderer
│   └── docs.module.ts
│
├── health/                         # Health checks
│   ├── health.controller.ts        # /health endpoint
│   └── health.module.ts
│
└── prisma/                         # Database client
    ├── prisma.service.ts           # Prisma singleton
    └── prisma.module.ts            # Global module

Frontend (apps/web/src/)

apps/web/src/
├── app/                            # Next.js App Router
│   ├── layout.tsx                  # Root layout
│   ├── page.tsx                    # Landing page
│   ├── globals.css                 # Global styles
│   │
│   ├── login/
│   │   └── page.tsx                # Login page
│   ├── register/
│   │   └── page.tsx                # Registration page
│   │
│   └── dashboard/                  # Protected dashboard
│       ├── layout.tsx              # Dashboard shell with nav
│       ├── page.tsx                # Overview (redirect to overview/)
│       ├── overview/
│       │   └── page.tsx            # Main dashboard
│       ├── events/
│       │   └── page.tsx            # Event stream
│       ├── attributions/
│       │   └── page.tsx            # Attribution list
│       ├── sources/
│       │   └── page.tsx            # Traffic sources
│       ├── integrations/
│       │   └── page.tsx            # Platform integrations
│       ├── pixel/
│       │   └── page.tsx            # Pixel setup
│       ├── team/
│       │   └── page.tsx            # Team management
│       └── settings/
│           └── page.tsx            # Account settings
│
├── components/                     # Reusable components
│   ├── ui/                         # shadcn/ui components
│   │   ├── button.tsx
│   │   ├── card.tsx
│   │   ├── dialog.tsx
│   │   ├── input.tsx
│   │   ├── table.tsx
│   │   └── ... (15 components)
│   │
│   ├── dashboard/                  # Dashboard-specific
│   │   ├── kpi-card.tsx
│   │   ├── area-chart-card.tsx
│   │   └── donut-chart-card.tsx
│   │
│   ├── theme-provider.tsx          # Dark mode support
│   ├── theme-toggle.tsx            # Theme switcher
│   ├── workspace-selector.tsx      # Workspace dropdown
│   ├── brand-selector.tsx          # Brand dropdown
│   ├── brand-management.tsx        # Brand CRUD UI
│   ├── team-management.tsx         # Team member UI
│   ├── integration-modal.tsx       # Integration config
│   ├── queue-indicator.tsx         # Real-time queue stats
│   └── ... (more components)
│
├── contexts/                       # React Context providers
│   ├── auth-context.tsx            # Authentication state
│   ├── brand-context.tsx           # Current brand state
│   └── settings-context.tsx        # User settings
│
├── hooks/                          # Custom React hooks
│   └── use-toast.ts                # Toast notifications
│
└── lib/                            # Utilities
    ├── api.ts                      # API client functions
    ├── utils.ts                    # Helper functions
    └── theme-tokens.ts             # Theme constants

---

Data Flow

1. Event Ingestion Flow

Client Website
    ↓ (JavaScript Pixel)
POST /api/v1/track
    ↓ (Validation)
Event Normalization Service
    ↓ (Canonical Schema)
Redis Streams Publisher
    ↓ (datomnix:events:brand_id)
Stream: "datomnix:events:brand_id"
    ↓ (Consumer Group)
3x Parallel Attribution Consumers
    ↓ (Batch: 100 events)
Attribution Engine
    ↓ (Last-Click Logic)
PostgreSQL Database
    ↓ (Query)
Reporting API
    ↓ (JSON Response)
Dashboard UI

2. Authentication Flow

User → Login Form
    ↓ (POST /auth/login)
Auth Service
    ↓ (Validate Credentials)
Generate JWT Token
    ↓ (Include: userId, role, workspaceId)
Return Token + User Data
    ↓ (Store in localStorage)
Dashboard Pages
    ↓ (Authorization: Bearer <token>)
JWT Strategy Validates
    ↓ (Attach user to request)
Role Guards Check Permissions
    ↓ (Allow/Deny)
Protected Resources

3. Attribution Processing Flow

Purchase Event Arrives
    ↓
Lookup Session by Session ID
    ↓
Extract UTM Parameters from Session
    ↓
Apply Attribution Model:
    • Last-Click: 100% to last touch
    • First-Click: 100% to first touch
    • Linear: Equal split
    • Time-Decay: Exponential weighting
    • Position-Based: 40% first, 40% last, 20% middle
    ↓
Create Attribution Record
    ↓
Store in Database with:
    • Order ID
    • Revenue
    • Source/Medium/Campaign
    • Model Type
    • Timestamp

4. Integration Orchestration Flow

Event Published to Stream
    ↓
Integration Orchestrator Receives Event
    ↓
Check Brand's Active Integrations
    ↓
For Each Integration (Parallel):
    ├── Meta CAPI
    │   └── POST to Graph API
    ├── TikTok Events API
    │   └── POST to Events API
    ├── Google Analytics 4
    │   └── POST to Measurement Protocol
    └── Custom Webhook
        └── POST to Custom URL
    ↓
Log Success/Failures
    ↓
Return Aggregated Results

---

Database Design

Multi-Tenant Schema

All tables include tenant_id (workspace) and where applicable brand_id for complete isolation.

#

Core Tables

Users & Authentication


User {
  id, email, passwordHash, firstName, lastName
  role: SUPER_ADMIN | USER
  createdAt, updatedAt
}

WorkspaceMember {
  workspaceId, userId, role: OWNER | ADMIN | MEMBER
  createdAt, updatedAt
}

Multi-Tenancy


Workspace {
  id, name, slug, ownerId
  createdAt, updatedAt
}

Brand {
  id, workspaceId, name, domain
  pixelApiKey, serverApiKey
  timezone, currency, vertical
  createdAt, updatedAt
}

Events & Sessions


Event {
  id, eventId (unique), brandId
  eventType, sessionId, userId
  source, medium, campaign, content, term
  metadata (JSONB), timestamp
  ingestedAt, createdAt
}

Session {
  id, sessionId, brandId, userId
  source, medium, campaign
  firstTouch, lastTouch
  createdAt, updatedAt
}

Attribution


Attribution {
  id, orderId, brandId, sessionId, userId
  modelType, value, currency
  source, medium, campaign
  attributedAt, createdAt
}

Integrations


Integration {
  id, brandId, platform
  config (JSONB), isActive
  lastSyncAt, createdAt, updatedAt
}

Indexing Strategy

-- Performance-critical indexes
CREATE INDEX events_brand_timestamp_idx ON events(brand_id, timestamp DESC);
CREATE INDEX events_session_idx ON events(session_id);
CREATE INDEX events_event_id_idx ON events(event_id); -- Unique
CREATE INDEX attributions_brand_date_idx ON attributions(brand_id, attributed_at DESC);
CREATE INDEX attributions_source_idx ON attributions(brand_id, source, medium, campaign);
CREATE INDEX sessions_brand_user_idx ON sessions(brand_id, user_id);

---

Performance Optimizations

1. Queue Processing (30x Faster)



• Before: Single consumer, sequential processing


• After: 3 parallel consumers, batch processing of 100 events


• Result: 30x throughput improvement

2. Database Queries



• Indexed columns for fast lookups


• Composite indexes for common query patterns


• JSONB for flexible metadata with GIN indexes

3. Caching Strategy



• Redis for session data (TTL: 30 minutes)


• API response caching for aggregated reports


• Browser-level caching for static assets

4. Frontend Optimization



• Server-side rendering with Next.js


• SWR for data fetching with automatic revalidation


• Code splitting per route


• Lazy loading for charts and heavy components

---

Security Architecture

Authentication & Authorization



• JWT Tokens: Secure, stateless authentication


• Password Hashing: bcrypt with salt rounds


• RBAC: 4 role levels (Super Admin, Owner, Admin, Member)


• Workspace Isolation: Middleware enforces data boundaries


• API Key Authentication: For tracking pixels and S2S events

Data Security



• Environment Variables: All secrets in .env (never committed)


• SQL Injection Prevention: Prisma parameterized queries


• XSS Protection: React automatic escaping


• CORS Configuration: Whitelist allowed origins


• HTTPS Only: Force SSL in production


• Rate Limiting: Prevent abuse on public endpoints

Privacy & Compliance



• GDPR Ready: User data deletion, export capabilities


• Data Retention: Configurable per workspace


• Consent Management: Cookie banner integration


• Anonymization: Option to hash user identifiers

---

Scalability Considerations

Current Architecture (Production-Ready)



• Single API instance (Railway)


• Redis Streams with parallel consumers


• PostgreSQL with proper indexing


• Next.js dashboard on Vercel

Scaling Strategy

Horizontal Scaling
1. Multiple API instances behind load balancer
2. Redis Streams consumer groups for parallel processing
3. PostgreSQL read replicas for reporting queries
4. CDN for static assets

Data Volume
1. Partition events table by brand and date
2. Archive old events to object storage (S3)
3. Materialized views for complex aggregations
4. Time-series database for high-frequency metrics

Performance
1. Query result caching in Redis
2. Debounced dashboard updates
3. Webhook request buffering
4. Background job processing for heavy operations

---

Monitoring & Observability

Health Checks



• API: GET /api/v1/health - Database + Redis connectivity


• Dashboard: Automatic health polling


• Uptime: External monitoring (Railway/Vercel)

Logging



• Structured JSON logs (NestJS Logger)


• Request/response logging


• Error tracking with stack traces


• Performance metrics (response times)

Metrics



• Event ingestion rate


• Attribution processing latency


• Queue depth and consumer lag


• API endpoint response times


• Database query performance

Alerts



• API downtime


• Database connection failures


• Redis unavailability


• Queue processing delays

---

Deployment Architecture

Development




Local Machine
├── Docker Compose
│   ├── PostgreSQL (port 5432)
│   └── Redis (port 6379)
├── API (port 3001)
└── Web (port 3000)

Production




Internet
    ↓
Vercel Edge Network
    ↓ (Next.js Dashboard)
Vercel Serverless Functions
    ↓ HTTPS
Railway Platform
    ├── NestJS API
    ├── PostgreSQL Database
    └── Redis Instance

---

Technology Stack

Backend



• Framework: NestJS 10 (Node.js + TypeScript)


• Database: PostgreSQL 14+ with Prisma ORM


• Cache/Queue: Redis 6+ (Streams + BullMQ)


• Auth: Passport JWT + bcrypt


• Validation: class-validator, class-transformer


• Documentation: Swagger/OpenAPI 3.0

Frontend



• Framework: Next.js 14 (React 18, App Router)


• Styling: Tailwind CSS + shadcn/ui


• State: React Context + SWR


• Charts: Recharts


• UI Components: Radix UI primitives


• Forms: React Hook Form + Zod

DevOps



• Containerization: Docker + Docker Compose


• CI/CD: GitHub Actions (future)


• Hosting: Railway (API) + Vercel (Web)


• Monitoring: Built-in health checks


• Database Migrations: Prisma Migrate

---

Northbeam-Level Features (v1.5)

DatomniX has been enhanced to match Northbeam-level sophistication for enterprise attribution and analytics.

Advanced Attribution Settings

#

UTM Configuration



• Custom Parameter Names: Brands can customize UTM parameter names beyond standard conventions


• Click ID Tracking: Automatic capture of platform-specific click IDs (gclid, fbclid, ttclid, msclkid)


• Internal Parameters: Support for proprietary tracking parameters that override standard UTMs


• API: GET/PUT /brands/{brandId}/settings/utm

#

Channel Resolution Engine



• Rule-Based Mapping: Priority-ordered rules for intelligent channel categorization


• Match Types: Supports equals, contains, starts_with, and regex pattern matching


• Field Selection: Match against source, medium, or campaign fields


• Default Fallback: Built-in logic for common channel patterns when no custom rules match


• API: GET/POST/PATCH/DELETE /brands/{brandId}/settings/utm/channel-rules

#

Order Exclusions



• Tag-Based Rules: Filter orders by Shopify/WooCommerce tags (exact, contains, regex)


• Email Exclusions: Block specific email addresses from attribution (test accounts, internal team)


• Automatic Filtering: Excluded orders stored but not included in attribution calculations


• Reason Tracking: Each exclusion records why the order was filtered


• API:

- GET/POST/DELETE /brands/{brandId}/settings/exclusions/order-tags
- GET/POST/DELETE /brands/{brandId}/settings/exclusions/emails

#

Subscription Classification



• Category Mapping: Tag-based classification (first_time, recurring, other)


• Flexible Tags: Support for various subscription platforms (Recharge, Bold, etc.)


• Revenue Segmentation: Separate tracking of new vs recurring subscription revenue


• Automatic Detection: Falls back to user history if no explicit tags present


• API: GET/POST/DELETE /brands/{brandId}/settings/subscription-tags

Meta Conversions API (Enhanced)

#

Advanced Configuration



• Configurable Match Keys: Select which customer data to send (email, phone, IP, user agent, FBP, FBC, external_id)


• Event Type Filtering: Choose specific events to forward (Purchase, Lead, AddToCart, etc.)


• Test Mode Support: Test event codes for validation before production


• Privacy Compliance: Automatic SHA-256 hashing of PII (email, phone)


• Deduplication: Event ID-based dedup with pixel events

#

Event Payload




{
  event_name: "Purchase",
  event_time: 1699564800,
  event_id: "evt_abc123",          // For deduplication
  event_source_url: "https://...",
  action_source: "website",
  user_data: {
    em: ["hash..."],                // SHA-256 hashed email
    ph: ["hash..."],                // SHA-256 hashed phone
    client_ip_address: "1.2.3.4",
    client_user_agent: "Mozilla...",
    fbp: "_fbp cookie value",
    fbc: "fb.1.xxx.fbclid"
  },
  custom_data: {
    currency: "USD",
    value: 99.99,
    content_ids: ["product_123"],
    contents: [{id: "product_123", quantity: 1}],
    num_items: 1
  }
}

Customer Journey Analysis

#

Path Aggregation System



• Nightly Jobs: Automated aggregation of customer paths from touchpoints


• Model-Specific: Separate aggregates per attribution model and lookback window


• Label Sequences: Paths stored as ordered arrays of touchpoint labels


• Hash-Based Dedup: Efficient storage using SHA-256 path hashes


• Metrics: Count of paths, transactions, and revenue per unique journey

#

Label Generation Engine


Labels automatically generated for each touchpoint based on context:

• excluded: - Order was filtered by exclusion rules


• unattributed:first_time / unattributed:returning - No clear source


• direct:none - Direct traffic with no referrer


• referral: - Referral from specific domain


• | - Attributed traffic with resolved channel

#

Path Report API




GET /reporting/brands/{brandId}/paths
  ?model=linear
  &lookbackDays=7
  &dateFrom=2025-11-01
  &dateTo=2025-11-30
  &minLength=2
  &maxLength=20
  &labelFilter=facebook,google
  &limit=100

Model Comparison Reporting

#

Side-by-Side Analysis



• Multi-Model: Compare any two attribution models simultaneously


• Granularity Options: Analyze by channel, campaign, or label


• Delta Calculations: Automatic calculation of revenue and transaction differences


• Percentage Changes: Visual indicators for significant model impact


• Same Window: Both models use identical lookback window for fair comparison

#

Comparison API




GET /reporting/brands/{brandId}/model-comparison
  ?model1=last_click
  &model2=position_based
  &lookbackDays=7
  &granularity=channel
  &dateFrom=2025-11-01
  &dateTo=2025-11-30

#

Response Format




[
  {
    "name": "Paid Social",
    "revenueModel1": 50000.00,
    "transactionsModel1": 250,
    "revenueModel2": 67500.00,
    "transactionsModel2": 270,
    "deltaRevenue": 17500.00,
    "deltaPercent": 35.0,
    "deltaTransactions": 20
  }
]

Export Center

#

Dataset Export



• Datasets: Orders, Attributions, Customer Paths, Raw Events


• Formats: CSV, JSON (Parquet planned)


• Destinations: Direct download, email, webhook, S3 (planned)


• Date Ranges: Flexible date filtering for each export


• Status Tracking: Pending, processing, completed, failed

#

Scheduled Exports



• Cron-Based: Flexible scheduling using cron expressions


• Recurring: Daily, weekly, monthly automatic exports


• Last Run Tracking: Monitor execution history


• Same Config: Use identical format and destination config for consistency

#

Export APIs




POST /exports
{
  "brandId": "brand_123",
  "dataset": "orders",
  "format": "csv",
  "destination": "download",
  "dateFrom": "2025-11-01",
  "dateTo": "2025-11-30"
}

GET /exports?workspaceId=ws_123&status=completed
GET /exports/{exportId}/download
GET /exports/scheduled

Database Models (New)

#

Settings Models



• BrandUtmSettings - Custom UTM parameter configuration


• BrandUtmChannelRule - Channel resolution rules with priority


• BrandAttributionSettings - Model and window configuration


• BrandOrderExclusionRule - Tag-based order filters


• BrandEmailExclusion - Email-based attribution filters


• BrandSubscriptionTagMapping - Subscription categorization

#

Attribution Models



• Order - Conversion events with subscription classification and exclusion tracking


• Touchpoint - Individual customer interactions with resolved channels and labels


• PathAggregate - Pre-computed customer journey paths with metrics

#

Export Models



• ExportJob - Manual and triggered export jobs


• ScheduledExport - Recurring export configuration

#

Future Models (Database ready)



• BrandTrackingDomain - Custom tracking domain with DNS verification


• ManualSpendSource - Manual spend tracking for offline channels


• ManualSpendEntry - Individual spend entries per date


• Annotation - Marketing annotations for timeline events

Services Architecture

#

Attribution Services




UtmService                          // UTM extraction and channel resolution
ExclusionService                    // Order exclusion logic
SubscriptionClassificationService   // Subscription categorization
LabelService                        // Touchpoint label generation
AttributionEngineService            // Core attribution calculation
AttributionConsumerService          // Event processing orchestration

#

Reporting Services




PathAggregationService              // Customer path generation and aggregation
PathsController                     // Path analysis API endpoints

#

Integration Services




MetaIntegration                     // Enhanced Meta CAPI with advanced config
IntegrationOrchestrator             // Multi-platform event distribution

Event Processing Flow (Enhanced)

1. Event Ingestion (Pixel/API/Webhook)
   ↓
2. UTM Extraction & Channel Resolution
   → Uses BrandUtmSettings and BrandUtmChannelRules
   ↓
3. Touchpoint Creation
   → Label generation via LabelService
   → Stores: source, medium, campaign, channel, clickId, label
   ↓
4. Order Detection (if purchase event)
   → Order Exclusion Check (ExclusionService)
   → Subscription Classification
   → Create Order record
   ↓
5. Attribution (if not excluded)
   → Multiple models + lookback windows
   → Links touchpoints to orders
   ↓
6. Integration Distribution
   → Meta CAPI (with advanced config)
   → TikTok, GA4, Custom webhooks
   ↓
7. Path Aggregation (nightly)
   → Generates PathAggregate records
   → Per model + window + date range

Performance Considerations

#

Caching Strategy



• Brand Settings: Redis cache with 5-minute TTL


• Channel Rules: In-memory cache per consumer


• Path Aggregates: Pre-computed for fast reporting


• Model Comparisons: Query-time aggregation with indexes

#

Database Indexes




-- UTM and Channel Resolution
CREATE INDEX idx_touchpoint_brand_user_ts ON touchpoints(brand_id, user_id, timestamp);
CREATE INDEX idx_touchpoint_session ON touchpoints(session_id);
CREATE INDEX idx_channel_rules_priority ON brand_utm_channel_rules(brand_id, priority);

-- Order Exclusions
CREATE INDEX idx_order_brand_date ON orders(brand_id, order_date);
CREATE INDEX idx_order_excluded ON orders(excluded);
CREATE INDEX idx_email_exclusion ON brand_email_exclusions(brand_id, email);

-- Path Aggregation
CREATE UNIQUE INDEX idx_path_aggregate_unique ON path_aggregates(
  brand_id, model, lookback_days, path_hash, date_from, date_to
);

-- Model Comparison
CREATE INDEX idx_attribution_comparison ON attributions(
  brand_id, model_type, lookback_days, attributed_at
);

---

Future Enhancements

Phase 1 (Planned)



• [ ] Real-time dashboard updates (WebSockets)


• [ ] Advanced funnel visualization


• [ ] Custom attribution models with ML


• [ ] White-label capabilities


• [ ] Mobile app (React Native)

Phase 2 (Roadmap)



• [ ] Data-driven attribution (ML-powered)


• [ ] Cross-device identity resolution


• [ ] Customer lifetime value (CLV) predictions


• [ ] Anomaly detection and alerts


• [ ] A/B testing framework


• [ ] View-through attribution

Phase 3 (Future)



• [ ] Real-time bidding optimization


• [ ] Predictive analytics


• [ ] AI-powered insights


• [ ] Advanced segmentation engine


• [ ] Multi-language support


• [ ] Enterprise SSO (SAML, OAuth)

---

Glossary

• Attribution: Assigning credit for conversions to marketing touchpoints


• Brand: Individual tracking property within a workspace


• Event: User action (page view, add to cart, purchase)


• Integration: Connection to external platform (Meta, TikTok, etc.)


• Multi-Touch Attribution: Credit distributed across multiple touchpoints


• RBAC: Role-Based Access Control


• Session: Group of events from same user in time window


• Stream: Redis data structure for ordered event processing


• Tenant: Workspace (top-level isolation boundary)


• UTM Parameters: Tracking parameters (source, medium, campaign)


• Workspace: Top-level container for brands and team members

---

Last Updated: November 9, 2025
Version: 1.0.0 (Production)
Developed by: ETK Technologies