Designing Jobs

Create powerful workflows that orchestrate AI agents

Designing Jobs

Jobs are the heart of Atlas workflows. They define how agents work together to accomplish tasks when signals are triggered. This guide will help you design effective, reliable jobs.

Job Structure

Every job has these key components:

jobs:
  my-job:
    # Identity
    name: "Human-readable name"
    description: "What this job accomplishes"
    
    # Triggers
    triggers:
      - signal: "signal-name"
        condition: # Optional conditions
    
    # Execution
    execution:
      strategy: "sequential"  # or "parallel"
      agents:
        - id: "agent-id"
          input_source: "signal"
          task: "Specific instructions"
    
    # Configuration
    config:
      timeout: "300s"
      retry_on_failure: true

Execution Strategies

Sequential Execution

Agents run one after another, passing data forward:

jobs:
  analysis-pipeline:
    name: "Data Analysis Pipeline"
    description: "Extract, analyze, and report on data"
    
    triggers:
      - signal: "analyze-data"
    
    execution:
      strategy: "sequential"
      agents:
        - id: "extractor"
          input_source: "signal"
          task: "Extract data from the provided source"
        
        - id: "analyzer"
          input_source: "previous"
          task: "Analyze the extracted data for patterns"
        
        - id: "reporter"
          input_source: "previous"
          task: "Create a comprehensive report"

Data Flow:

Signal → Extractor → Analyzer → Reporter → Output

Parallel Execution

Agents run simultaneously for speed:

jobs:
  multi-perspective-analysis:
    name: "Multi-Perspective Analysis"
    description: "Analyze from multiple angles simultaneously"
    
    triggers:
      - signal: "analyze-topic"
    
    execution:
      strategy: "parallel"
      agents:
        - id: "technical-analyst"
          input_source: "signal"
          task: "Analyze technical aspects"
        
        - id: "business-analyst"
          input_source: "signal"
          task: "Analyze business implications"
        
        - id: "risk-analyst"
          input_source: "signal"
          task: "Identify risks and concerns"

Data Flow:

         ┌→ Technical Analyst →┐
Signal → ├→ Business Analyst  →├→ Combined Output
         └→ Risk Analyst      →┘

Mixed Strategies

Combine sequential and parallel execution:

jobs:
  complex-workflow:
    execution:
      strategy: "sequential"
      agents:
        # First: gather data in parallel
        - parallel_group:
            agents:
              - id: "web-researcher"
                task: "Search web for information"
              - id: "database-query"
                task: "Query internal database"
        
        # Then: analyze combined results
        - id: "analyzer"
          input_source: "previous"
          task: "Analyze all gathered data"
        
        # Finally: generate output
        - id: "report-generator"
          input_source: "previous"
          task: "Create final report"

Input Sources

Control how agents receive data:

Signal Input

Agent receives the triggering signal’s data:

agents:
  - id: "processor"
    input_source: "signal"  # Gets original signal data

Previous Output

Agent receives output from the previous agent:

agents:
  - id: "first-agent"
    input_source: "signal"
  
  - id: "second-agent"
    input_source: "previous"  # Gets first-agent's output

Combined Sources

Merge multiple input sources:

agents:
  - id: "synthesizer"
    input_source: "combined"
    input_mapping:
      signal_data: "signal"
      analysis: "previous"
      context: "session"

Custom Context

Provide additional context:

agents:
  - id: "contextual-agent"
    input_source: "signal"
    context:
      project: "Q4 Analysis"
      priority: "high"
      include_history: true

Conditional Triggers

Simple Conditions

Trigger jobs based on signal data:

jobs:
  priority-handler:
    triggers:
      - signal: "support-ticket"
        condition:
          type: "simple"
          field: "priority"
          operator: "equals"
          value: "urgent"

JSON Logic Conditions

Complex conditions using JSON Logic:

jobs:
  smart-router:
    triggers:
      - signal: "process-request"
        condition:
          type: "json_logic"
          logic:
            "and": [
              {">=": [{"var": "amount"}, 1000]},
              {"==": [{"var": "category"}, "enterprise"]},
              {"in": [{"var": "region"}, ["US", "EU", "UK"]]}
            ]

Multiple Triggers

Jobs can respond to multiple signals:

jobs:
  universal-processor:
    triggers:
      # High priority path
      - signal: "urgent-request"
        
      # Normal path with conditions
      - signal: "normal-request"
        condition:
          type: "json_logic"
          logic:
            {">=": [{"var": "size"}, 100]}
      
      # Scheduled path
      - signal: "scheduled-process"

Advanced Job Configuration

Timeouts and Retries

Control job execution limits:

jobs:
  reliable-job:
    config:
      # Overall job timeout
      timeout: "600s"  # 10 minutes
      
      # Retry configuration
      retry_on_failure: true
      max_retries: 3
      retry_delay: "30s"
      retry_backoff: "exponential"  # or "linear"
      
      # Per-agent timeouts
      agent_timeouts:
        researcher: "120s"
        analyzer: "180s"
        reporter: "60s"

Error Handling

Define how jobs handle failures:

jobs:
  fault-tolerant-job:
    config:
      # Continue on agent failure
      continue_on_error: true
      
      # Fallback behavior
      on_error: "use_partial_results"  # or "fail_fast"
      
      # Error notifications
      error_handlers:
        - type: "webhook"
          url: "${ERROR_WEBHOOK_URL}"
        - type: "log"
          level: "error"

Supervision Levels

Control how much LLM supervision jobs receive:

jobs:
  # Minimal supervision for simple tasks
  simple-job:
    config:
      supervision:
        level: "minimal"
  
  # Standard supervision (default)
  normal-job:
    config:
      supervision:
        level: "standard"
  
  # Detailed supervision for complex tasks
  complex-job:
    config:
      supervision:
        level: "detailed"
        require_plan_approval: true
        allow_plan_revision: true

Memory Configuration

Control memory access for jobs:

jobs:
  memory-aware-job:
    config:
      memory:
        # Enable memory access
        enabled: true
        
        # Memory scopes to access
        scopes: ["workspace", "session"]
        
        # Context window
        max_memories: 10
        relevance_threshold: 0.7
        
        # Memory creation
        create_memories: true
        memory_tags: ["analysis", "q4-2024"]

Job Patterns

1. Pipeline Pattern

Sequential data processing:

jobs:
  etl-pipeline:
    name: "ETL Pipeline"
    description: "Extract, transform, and load data"
    
    execution:
      strategy: "sequential"
      agents:
        - id: "extractor"
          input_source: "signal"
          task: "Extract data from source: {source}"
          output_schema:
            type: "array"
            items:
              type: "object"
        
        - id: "transformer"
          input_source: "previous"
          task: "Transform data to target format"
          context:
            target_format: "normalized_v2"
        
        - id: "validator"
          input_source: "previous"
          task: "Validate transformed data"
        
        - id: "loader"
          input_source: "previous"
          task: "Load data into destination: {destination}"

2. Map-Reduce Pattern

Process items in parallel, then combine:

jobs:
  map-reduce-analyzer:
    execution:
      strategy: "map_reduce"
      
      # Map phase - parallel processing
      map:
        agent: "item-processor"
        task: "Process individual item"
        batch_size: 10
        max_parallel: 5
      
      # Reduce phase - combine results
      reduce:
        agent: "aggregator"
        task: "Combine and summarize all results"

3. Decision Tree Pattern

Conditional execution paths:

jobs:
  decision-flow:
    execution:
      strategy: "conditional"
      
      # Initial assessment
      agents:
        - id: "classifier"
          input_source: "signal"
          task: "Classify request type"
          output_var: "request_type"
      
      # Conditional branches
      branches:
        - condition:
            type: "equals"
            var: "request_type"
            value: "technical"
          agents:
            - id: "tech-specialist"
              task: "Handle technical request"
        
        - condition:
            type: "equals"
            var: "request_type"
            value: "business"
          agents:
            - id: "business-analyst"
              task: "Handle business request"
        
        # Default branch
        - condition:
            type: "default"
          agents:
            - id: "general-handler"
              task: "Handle general request"

4. Feedback Loop Pattern

Iterative improvement:

jobs:
  iterative-improver:
    execution:
      strategy: "iterative"
      max_iterations: 3
      
      agents:
        - id: "generator"
          task: "Generate initial solution"
        
        - id: "critic"
          input_source: "previous"
          task: "Critique and score the solution"
          output_schema:
            type: "object"
            properties:
              score: { type: "number" }
              feedback: { type: "string" }
        
        - id: "improver"
          input_source: "combined"
          task: "Improve based on feedback"
          
      # Continue until score > 0.8
      completion_condition:
        type: "json_logic"
        logic:
          ">": [{"var": "score"}, 0.8]

5. Ensemble Pattern

Multiple agents vote on outcome:

jobs:
  ensemble-decision:
    execution:
      strategy: "ensemble"
      
      # Multiple agents analyze independently
      agents:
        - id: "conservative-analyst"
          weight: 0.3
          task: "Provide conservative analysis"
        
        - id: "moderate-analyst"
          weight: 0.4
          task: "Provide balanced analysis"
        
        - id: "aggressive-analyst"
          weight: 0.3
          task: "Provide aggressive analysis"
      
      # Combine results
      aggregation:
        method: "weighted_average"  # or "majority_vote"
        resolver: "synthesizer"
        task: "Synthesize all perspectives"

Complex Job Examples

Customer Support Workflow

jobs:
  customer-support-flow:
    name: "Intelligent Customer Support"
    description: "Route and handle customer inquiries"
    
    triggers:
      - signal: "customer-inquiry"
    
    execution:
      strategy: "sequential"
      
      agents:
        # Understand the inquiry
        - id: "intent-classifier"
          input_source: "signal"
          task: "Classify customer intent and urgency"
          output_schema:
            type: "object"
            properties:
              intent: { type: "string" }
              urgency: { type: "string", enum: ["low", "medium", "high"] }
              category: { type: "string" }
        
        # Route based on classification
        - id: "router"
          input_source: "combined"
          task: "Determine best handling approach"
          context:
            available_agents: ["tech-support", "billing", "general"]
        
        # Handle with appropriate specialist
        - id: "dynamic"  # Dynamically selected based on router output
          input_source: "combined"
          task: "Resolve customer inquiry"
        
        # Quality check
        - id: "quality-checker"
          input_source: "previous"
          task: "Verify solution quality and completeness"
    
    config:
      supervision:
        level: "detailed"
      memory:
        enabled: true
        create_memories: true
        memory_tags: ["support", "customer-interaction"]

Research and Report Generation

jobs:
  research-report:
    name: "Comprehensive Research Report"
    description: "Deep research with multi-source synthesis"
    
    triggers:
      - signal: "research-request"
    
    execution:
      strategy: "phased"
      
      phases:
        # Phase 1: Parallel research
        research:
          strategy: "parallel"
          agents:
            - id: "web-researcher"
              task: "Search web for recent information"
              tools: ["web-search"]
            
            - id: "academic-researcher"
              task: "Search academic sources"
              tools: ["scholar-search"]
            
            - id: "data-analyst"
              task: "Analyze relevant datasets"
              tools: ["data-tools"]
        
        # Phase 2: Synthesis
        synthesis:
          strategy: "sequential"
          agents:
            - id: "synthesizer"
              input_source: "phase:research"
              task: "Combine all research findings"
            
            - id: "fact-checker"
              input_source: "previous"
              task: "Verify facts and check sources"
        
        # Phase 3: Report generation
        reporting:
          strategy: "sequential"
          agents:
            - id: "outline-creator"
              input_source: "phase:synthesis"
              task: "Create report outline"
            
            - id: "writer"
              input_source: "combined"
              task: "Write comprehensive report"
            
            - id: "editor"
              input_source: "previous"
              task: "Edit and polish report"
    
    config:
      timeout: "1800s"  # 30 minutes
      memory:
        enabled: true
        scopes: ["workspace", "session"]

Testing Jobs

Unit Testing

Test individual jobs:

# Dry run
atlas signal trigger test-signal --dry-run

# With test data
atlas signal trigger test-signal --data '{"test": true}'

# Verbose output
atlas signal trigger test-signal --verbose

Integration Testing

Test complete workflows:

# test-job.yml
jobs:
  test-integration:
    triggers:
      - signal: "test"
    
    execution:
      agents:
        - id: "test-agent"
          task: "Echo input: {input}"
    
    config:
      test_mode: true
      mock_external_calls: true

Performance Testing

Monitor job performance:

config:
  metrics:
    enabled: true
    track:
      - execution_time
      - token_usage
      - memory_usage
      - agent_performance

Best Practices

1. Clear Task Instructions

Be specific about what each agent should do:

# Good
task: "Analyze the CSV file for sales trends in Q4 2024, focusing on product categories"

# Too vague
task: "Analyze the data"

2. Appropriate Timeouts

Set realistic timeouts:

config:
  # Quick task
  timeout: "60s"
  
  # Complex analysis
  timeout: "600s"
  
  # Long-running process
  timeout: "3600s"

3. Error Recovery

Plan for failures:

config:
  retry_on_failure: true
  max_retries: 3
  fallback_strategy: "use_cached_results"

4. Output Validation

Validate agent outputs:

agents:
  - id: "generator"
    output_schema:
      type: "object"
      required: ["summary", "details"]
      properties:
        summary: { type: "string", maxLength: 500 }
        details: { type: "array" }

5. Memory Efficiency

Use memory wisely:

config:
  memory:
    max_memories: 5  # Don't overload context
    relevance_threshold: 0.8  # Only highly relevant

Troubleshooting

Job Not Triggering

• Verify signal exists and is configured correctly
• Check trigger conditions
• Review logs for errors

Agent Failures

• Check agent configuration
• Verify required tools are available
• Review task instructions for clarity

Performance Issues

• Monitor token usage
• Check for unnecessary parallel execution
• Review memory configuration

Data Flow Problems

• Verify input_source configuration
• Check output schemas
• Use logging to trace data flow

Next Steps

• Add Tools to Your Workspace
• See Configuration Examples
• Try Multi-Agent Tutorial
• Learn Advanced Patterns