Workflows

Chain multiple Cargos into multi-step DAG pipelines — transcribe, summarize, translate in a single API call

Workflows

Workflows chain multiple Cargos into multi-step pipelines where the output of one step becomes the input of the next. Instead of making separate API calls and wiring the data yourself, you describe the entire pipeline in one request and the coordinator handles everything.

How It Works

Consumer submits workflow:
  Step 1: transcribe (audio → text)
  Step 2: summarize (text → summary)        [depends on step 1]
  Step 3: translate (summary → french)       [depends on step 2]
  Step 4: classify (summary → tags)          [depends on step 2, parallel with step 3]

Coordinator:
  ├── Dispatch step 1 → Island A
  │
  ├── Step 1 done → dispatch step 2 → Island B
  │
  ├── Step 2 done → dispatch steps 3 + 4 in parallel
  │     ├── Step 3 → Island C
  │     └── Step 4 → Island D
  │
  └── All done → merge results → return to Consumer

Steps with independent dependencies run in parallel automatically — no configuration needed. The coordinator resolves the DAG and dispatches as soon as dependencies are satisfied.

Submitting a Workflow

POST /api/v1/workflows

{
  "name": "transcribe-and-summarize",
  "definition": {
    "steps": [
      {
        "key": "transcribe",
        "workload": "whisper-large",
        "input": {"audio_url": "https://example.com/meeting.mp3"}
      },
      {
        "key": "summarize",
        "workload": "llm-chat",
        "depends_on": ["transcribe"],
        "input_mapping": {
          "prompt": "Summarize this transcript:\n\n{{transcribe.output.text}}"
        }
      },
      {
        "key": "translate",
        "workload": "llm-chat",
        "depends_on": ["summarize"],
        "input_mapping": {
          "prompt": "Translate to French:\n\n{{summarize.output.text}}"
        }
      }
    ],
    "fail_mode": "fail_fast"
  }
}

Step Definition

Data Flow Between Steps

Use input_mapping to pass data from one step’s output to the next step’s input. Templates use double curly braces:

{{step_key.output.field_name}}

Examples:

• "{{transcribe.output.text}}" — the text field from the transcribe step’s output
• "Summarize: {{step1.output.text}}" — embedded in a string
• Multiple references work: "{{a.output.x}} and {{b.output.y}}"

Checking Progress

GET /api/v1/workflows/{id}

{
  "id": "workflow-uuid",
  "name": "transcribe-and-summarize",
  "status": "running",
  "steps": [
    {"key": "transcribe", "status": "succeeded", "output": {"text": "..."}},
    {"key": "summarize", "status": "running", "output": null},
    {"key": "translate", "status": "pending", "output": null}
  ]
}

For real-time updates, subscribe to the workflow:{id} PubSub channel:

{
  "total": 3,
  "succeeded": 1,
  "running": 1,
  "pending": 1,
  "failed": 0
}

Completed Workflow Result

When all steps finish, the workflow result includes every step’s output:

{
  "id": "workflow-uuid",
  "status": "succeeded",
  "result": {
    "steps": [
      {"key": "transcribe", "status": "succeeded", "output": {"text": "Meeting notes..."}},
      {"key": "summarize", "status": "succeeded", "output": {"text": "Summary: ..."}},
      {"key": "translate", "status": "succeeded", "output": {"text": "Résumé: ..."}}
    ],
    "succeeded": 3,
    "failed": 0,
    "total": 3
  }
}

Failure Modes

Parallel Execution

Steps with independent dependencies run in parallel automatically. In this example:

{
  "steps": [
    {"key": "a", "workload": "cargo-a"},
    {"key": "b", "workload": "cargo-b"},
    {"key": "merge", "workload": "cargo-merge", "depends_on": ["a", "b"]}
  ]
}

Steps a and b dispatch immediately and run on different Islands in parallel. Step merge waits for both to complete, then runs with their combined outputs.

Billing

Each step is billed as an individual job at the standard per-job rate. The workflow’s total cost is the sum of all step costs. Credits are deducted as each step dispatches, not upfront for the entire workflow.

Cancellation

POST /api/v1/workflows/{id}/cancel

Cancels all pending and running steps. Already-completed steps keep their results. The workflow status changes to "cancelled".

Conditional Steps

Steps can include a condition — an expression evaluated against predecessor outputs. If the condition is false, the step is skipped and downstream steps continue.

{
  "key": "escalate",
  "workload": "alert-service",
  "depends_on": ["classify"],
  "condition": "{{classify.output.sentiment}} == negative"
}

The expression language supports:

• Equality: ==, !=
• Numeric: >, <, >=, <=
• Text: contains, matches (regex)
• Logic: and, or

This enables branching workflows: classify content, then escalate only if negative, translate only if a specific language is detected, etc.

Retry on Failure

Steps can automatically retry on failure:

{
  "key": "call_external_api",
  "workload": "api-caller",
  "max_retries": 3,
  "timeout_seconds": 120
}

If the step fails or times out, it’s automatically re-dispatched (up to max_retries times) before being marked as permanently failed. The optional timeout_seconds sets a deadline — if the step hasn’t completed in time, it’s failed and retried.

Cost Estimation

Before submitting a workflow, you can estimate its cost:

POST /api/v1/workflows/estimate
{"definition": {"steps": [...]}}

Returns the total estimated credits and a per-step breakdown at base prices. Useful for validating that you have sufficient credits before running an expensive multi-step pipeline.

Workflow Templates

Save common workflows as reusable templates with variable placeholders. Templates can be public (visible to everyone) or private.

POST /api/v1/workflow-templates/transcription-pipeline/run
{
  "variables": {
    "model": "whisper-large",
    "audio": "https://example.com/meeting.mp3"
  }
}

Browse and instantiate templates from the API or the visual workflow editor at /workflows.

Visual Editor

The web-based workflow editor at app.archipelag.io/workflows provides:

• Drag-and-drop step creation from the Cargo catalog
• Visual dependency connections between steps
• Per-step configuration (conditions, retry, input mapping)
• Template library for common patterns
• JSON view toggle for advanced editing
• Real-time monitoring of running workflows

Use Cases

Scheduling

Run workflows at a specific future time or on a recurring schedule:

{
  "name": "daily-report",
  "scheduled_at": "2026-03-20T09:00:00Z",
  "cron_expression": "daily 09:00",
  "definition": { "steps": [...] }
}

Supported schedules: every 30m, every 2h, daily 09:00. The workflow stays in "scheduled" status until its time arrives, then dispatches automatically.

Step-Level Streaming

You don’t have to wait for the entire workflow to finish. Intermediate outputs are available as each step completes:

• PubSub: subscribe to workflow:{id} to receive {:step_output, %{step_key, output}} events
• API: GET /api/v1/workflows/{id}/steps/{step_key}/output returns a specific step’s result

In a transcribe → summarize → translate workflow, you can display the transcript immediately when step 1 finishes, then update with the summary, then the translation — progressive results instead of all-or-nothing.