1. What is AWS Batch?

AWS Batch is a fully managed service that plans, schedules, and executes batch computing workloads across AWS compute resources. It dynamically provisions the optimal quantity and type of compute resources based on the volume and requirements of your batch jobs.

Core Concept AWS Batch runs batch jobs (large-scale, non-interactive computing tasks) efficiently. You define your job, Batch handles everything: provisioning compute (EC2 or Fargate), scheduling, scaling, retry on failure, and cleanup. Think: managed job scheduler for compute-heavy workloads.

2. Batch vs Lambda

3. Key Concepts


Job

  1. A unit of work submitted to AWS Batch (e.g., a shell script, a Python program, a Docker container)
  2. Defined as a Docker container image with a command, resource requirements, and environment variables
  3. Can depend on other jobs (job dependencies)


Job Definition

  1. A blueprint for your job (like an ECS Task Definition)
  2. Specifies: Docker image, vCPUs, memory, environment variables, IAM role, retry strategy, timeout
  3. Versioned — each update creates a new revision


Job Queue

  1. Jobs are submitted to a queue and wait to be scheduled
  2. Multiple queues with different priorities (e.g., high-priority, low-priority)
  3. Each queue is associated with one or more Compute Environments
  4. Jobs are dispatched in order (FIFO within priority)


Compute Environment

  1. The pool of compute resources that processes jobs
  2. Two types: Managed (AWS handles provisioning) or Unmanaged (you manage EC2 instances)

4. Compute Environments


5. Batch Architecture Flow

AWS Batch Flow:

1. Create Job Definition (Docker image, CPU, memory)
2. Create Compute Environment (EC2/Fargate, Spot/On-Demand)
3. Create Job Queue (priority, linked to Compute Env)
4. Submit Job to Queue
5. Batch Scheduler picks up the job
6. Scheduler provisions compute (if needed)
7. Job runs as a Docker container
8. Job completes (or retries on failure)
9. Compute scales down when idle


6. Spot Integration

  1. Batch is one of the best services for leveraging Spot Instances
  2. Managed Compute Environments can use Spot with a configurable max price
  3. Batch automatically handles Spot interruptions by requeuing jobs
  4. Use allocation strategy: SPOT_CAPACITY_OPTIMIZED (recommended) or SPOT_PRICE_CAPACITY_OPTIMIZED
  5. Combine with On-Demand in the same Compute Environment for high-priority jobs

7. Job Dependencies

  1. A job can depend on the completion of one or more other jobs
  2. Creates a Directed Acyclic Graph (DAG) of job execution
  3. Example: Job C depends on Job A AND Job B completing successfully
  4. Sequential: Job A → Job B → Job C
  5. Fan-out: Job A → [Job B, Job C, Job D] (all run after A completes)

8. Array Jobs

  1. Submit a single job definition that runs as multiple child jobs (e.g., 1,000 copies)
  2. Each child gets an index number (0–N) available as an environment variable
  3. Use for: processing chunks of a large dataset, rendering frames, parameterized simulations
  4. Much simpler than submitting 1,000 individual jobs

9. Multi-Node Parallel Jobs

  1. Distribute a single job across multiple EC2 instances
  2. Used for: tightly coupled HPC workloads that need inter-node communication
  3. Supports Elastic Fabric Adapter (EFA) for high-performance networking
  4. Not supported on Fargate — EC2 only

10. Common Use Cases

  1. Financial modeling and risk analysis
  2. Media transcoding and rendering (VFX, animation)
  3. Genomics and bioinformatics processing
  4. Machine learning training data preprocessing
  5. Log analysis and ETL workloads
  6. Image/video processing at scale
  7. Scientific simulations
Exam Tip Batch questions: "Run thousands of computing jobs" = AWS Batch. "Job runs longer than 15 minutes" = Batch (not Lambda). "Docker-based batch processing" = Batch. "Cost-effective batch with Spot" = Batch Managed Compute with Spot. "HPC with inter-node communication" = Batch Multi-Node Parallel. Batch is NOT serverless on EC2 (it provisions instances). Batch on Fargate IS serverless. Lambda = short event-driven. Batch = long heavy compute.