Coordinator

Architecture of the Archipelag.io coordinator — the control plane that orchestrates Islands, jobs, and billing

Coordinator

The coordinator is the control plane for Archipelag.io. Built with Elixir and Phoenix, it handles Island registration, job dispatch, billing, karma tracking, and real-time streaming — all supervised by OTP for fault tolerance.

Supervision Tree

The coordinator starts as a standard OTP application under a :one_for_one supervisor:

Coordinator.Supervisor (:one_for_one)
│
├── CoordinatorWeb.Telemetry          # Phoenix telemetry handlers
├── Coordinator.Repo                  # Ecto/PostgreSQL connection pool
├── Coordinator.PromEx                # Prometheus metrics collection
├── DNSCluster                        # Fly.io DNS-based clustering
├── Phoenix.PubSub                    # In-memory pub/sub for real-time events
├── Oban                              # Background job queue (PostgreSQL-backed)
├── Jobs.OutputAccumulator            # Aggregates streaming output chunks
├── CoordinatorWeb.Endpoint           # Phoenix HTTP/WebSocket server
│
└── Nats.Supervisor (conditional)     # NATS message bus connection
    ├── Nats.HostHandler              # Host registration + heartbeats
    ├── Nats.JobHandler               # Job output + progress streaming
    └── Nats.StatusConsumer           # JetStream pull consumer for job status

Context Modules

The coordinator follows Phoenix’s context pattern — each domain area has a dedicated module that encapsulates its business logic and database access.

NATS Message Handlers

The coordinator communicates with Islands through NATS. Three GenServer/consumer processes handle different message types:

HostHandler

Subscribes to core NATS subjects for Island lifecycle:

JobHandler

Subscribes to Island output subjects for real-time streaming:

StatusConsumer

A JetStream pull consumer on the JOB_STATUS stream. Processes durable job status messages (started, succeeded, failed) with at-least-once delivery guarantees. Malformed messages are acknowledged to prevent infinite redelivery.

Background Workers

Oban provides reliable background job processing backed by PostgreSQL. Workers are scheduled via cron or enqueued on demand.

Database Schema

The coordinator uses PostgreSQL with Ecto. Key tables and their primary key types:

Telemetry and Metrics

The coordinator exposes Prometheus metrics via PromEx at GET /metrics (auth-protected in production). Metrics are organized into categories:

Structured logging with correlation IDs (job_id, host_id, workload_id) is available via Coordinator.StructuredLogger. In production, logs are emitted as JSON via Coordinator.JsonLogger.

LiveView and Channels

Real-time features are delivered through Phoenix LiveView and Channels:

Next Steps

{% end %}