TaskMaster - Background Job Engine

Overview

TaskMaster is the background job engine that powers async operations across the Evergreen ecosystem. Built as a separate service with Nitro.js as the API layer, Redis for queue storage, and BullMQ for job processing, it handles tens of thousands of jobs daily with dynamic module loading for extensibility.

Note: This is a separate service from the main Evergreen platform.

Scale

TaskMaster processes enterprise-level workloads:

Tens of thousands of jobs daily
Multiple job types running in parallel
Automatic retry with exponential backoff
Dead letter queues for failed job analysis

Architecture

Core Stack

┌─────────────────┐     ┌─────────────────┐     ┌─────────────────┐
│   Evergreen     │────▶│     Redis       │◀────│   TaskMaster    │
│   (Producer)    │     │   (Queue)       │     │   (Consumer)    │
└─────────────────┘     └─────────────────┘     └─────────────────┘
                                                        │
                                                        ▼
                                               ┌─────────────────┐
                                               │ Dynamic Modules │
                                               │  • Email        │
                                               │  • Reports      │
                                               │  • Enrichment   │
                                               │  • Integrations │
                                               └─────────────────┘

Components

Component	Technology	Role
API Layer	Nitro.js	HTTP endpoints for job submission
Queue Storage	Redis	Persistent job queue
Job Processing	BullMQ	Worker management, scheduling
Modules	TypeScript	Job handlers by type

Key Features

Dynamic Module Loading

Add new job types at runtime without restart. Modules are discovered and loaded on startup, each registering its job type and handler with the queue processor.

Hot Reload

Zero-downtime deployments:

Hot Swap: Update job handlers without restart
Graceful Drain: Finish current jobs before reload
Version Management: Track which version handled each job

Plugin Architecture

Extensible design with a standardized plugin interface. Each plugin declares its name, job types, handlers, and optional lifecycle hooks (onInit, onShutdown) for clean initialization and graceful shutdown.

Job Prioritization

Control job execution order:

Priority Levels: Critical, high, normal, low
Queue Separation: Different queues for different SLAs
Fair Scheduling: Prevent starvation

Retry & Error Handling

Robust failure recovery with configurable attempts (default 5), exponential backoff (1s, 2s, 4s, 8s, 16s), and retention policies for completed and failed jobs.