Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

Appearance settings

Include the ability to run specific modules independently #197

Copy link
Copy link
@jeroenrinzema

Description

@jeroenrinzema
Issue body actions

Problem Statement

Lunogram currently runs as a single monolithic process where all components start together: the HTTP controllers (management + client API), all NATS JetStream consumers, WASM-based action/provider execution, the cluster leader election, and the scheduler. There is no way to selectively enable or disable individual components.

This is a problem for production deployments where different components have different operational requirements. For example:

  • WASM execution (action execution via actions.execute.> / actions.validate.>, campaign sending via campaigns-send) runs third-party logic through Extism — webhooks, provider integrations (Twilio, Resend). These should be isolatable into a restricted namespace with stricter network access policies
  • NATS consumers (user events, org events, journey advancement, list recomputation, schema handlers) process internal platform logic and may need to scale independently from the API
  • HTTP controllers should be able to run without consuming from NATS streams, serving only API traffic behind a load balancer
  • Scheduler (leader election + delayed journey step reconciliation) is leader-only work that doesn't need HTTP or consumer overhead

Running everything in a single process means a noisy neighbour problem — a spike in WASM execution or journey processing can degrade API response times, and there's no way to apply different security policies to the WASM execution layer.

Proposed Solution

Add a configuration option that controls which components a node starts. By default, all components run (preserving current behaviour), but operators can selectively enable only the components they need.

The components to control are:

Module Description Key resources
http HTTP API server Management + client controllers, console UI, API docs
consumers NATS JetStream consumers for internal platform logic users-process, users-schema, users-events-process, users-events-schema, lists-recompute, journeys-advance, organizations-process, organizations-schema, organizations-users-process, organizations-users-schema, organizations-events-process, organizations-events-schema, actions-schema
wasm WASM-based execution for actions and campaigns actions.execute.> and actions.validate.> NATS core subscriptions, campaigns-send JetStream consumer, provider + action WASM registries
scheduler Cluster leader election + delayed journey step scheduler Redis-based consensus, periodic journey state reconciliation

When a component is disabled, its resources (listeners, goroutines, connections) are not initialised at all — not just idle.

Technical Implementation

  • Add a MODULES environment variable (comma-separated list, e.g. MODULES=http,consumers). When empty or unset, all components run (default behaviour)
  • Add a Modules []string field to config.Node with parsing logic and validation against known module names
  • Refactor cmd/lunogram/main.go run() to conditionally initialise components based on enabled modules:
    • Guard HTTP server startup with http module check
    • Guard NATS JetStream consumer bootstrap + serve (excluding campaigns-send and action subscriptions) with consumers module check
    • Guard WASM registry initialisation, campaigns-send consumer, and actions.execute.> / actions.validate.> subscriptions with wasm module check
    • Guard cluster consensus + scheduler with scheduler module check
  • Split consumer.Serve() so that WASM-dependent consumers (campaigns-send, action execute/validate) can be started independently from the internal platform consumers
  • WASM registries (provider + action) should only be initialised when the wasm module is enabled
  • Shared dependencies (database connections, NATS connection) should only be initialised when at least one component that needs them is enabled
  • Ensure graceful shutdown works correctly when only a subset of components is running

Additional Context

The existing NATS namespace (NATS_NAMESPACE) and Redis key prefix (REDIS_KEY_PREFIX) configuration already provide stream/subject isolation. This proposal complements that by controlling which components actually start within a process, enabling deployment topologies like:

┌─────────────────────┐   ┌──────────────────────┐   ┌─────────────────────┐
│  Namespace: default  │   │ Namespace: restricted │   │  Namespace: default  │
│  MODULES=http        │   │ MODULES=wasm          │   │  MODULES=consumers,  │
│                      │   │                       │   │          scheduler   │
│  - Management API    │   │  - Action execution   │   │                      │
│  - Client API        │   │  - Action validation  │   │  - Journey consumers │
│  - Console UI        │   │  - Campaign sending   │   │  - User/org events   │
│                      │   │  - Provider WASM      │   │  - List recomputation│
│                      │   │                       │   │  - Schema handlers   │
│  (no consumers,      │   │  (stricter network    │   │  - Leader election   │
│   no WASM, no sched) │   │   policies applied)   │   │  - Journey scheduler │
└─────────────────────┘   └──────────────────────┘   └─────────────────────┘

Acceptance Criteria

  • By default (no MODULES set), all components start — no change in behaviour for existing deployments
  • Setting MODULES=http starts only the HTTP server, no NATS consumers, WASM execution, or scheduler
  • Setting MODULES=consumers starts only the internal NATS JetStream consumers (excluding WASM-dependent ones), no HTTP server, WASM execution, or scheduler
  • Setting MODULES=wasm starts only the WASM registries, campaigns-send consumer, and action execute/validate subscriptions — no HTTP server, internal consumers, or scheduler
  • Setting MODULES=scheduler starts only the cluster leader election and delayed journey step scheduler
  • Multiple modules can be combined: MODULES=http,consumers starts both HTTP and internal consumers
  • Components that are not enabled do not allocate resources (no listeners, no goroutines, no unnecessary connections)
  • WASM registries are only loaded when the wasm module is enabled
  • Graceful shutdown works correctly for any combination of enabled modules
  • Invalid module names in MODULES cause a startup error with a clear message listing valid options
Reactions are currently unavailable

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions

    Morty Proxy This is a proxified and sanitized view of the page, visit original site.