step

aws_s3

Persist event data to Amazon S3 or S3-compatible object storage.

Overview

Persist event data to Amazon S3 or S3-compatible object storage. This step writes each event as a JSON file to S3 (or compatible services like MinIO, LocalStack, DigitalOcean Spaces). Each file is stored with a unique UUID-based key under a configurable prefix, making it easy to organize data by date, type, or workflow. The step is perfect for data archiving, creating audit trails, backing up processed events, or building data lakes. Supports custom endpoints for S3-compatible services and flexible authentication options. Events pass through unchanged after successful write.

Quick Start

steps:
- type: aws_s3
  bucket: my-bucket

Configuration

Parameter	Type	Required	Description
`bucket`	`string`	Yes	Target S3 bucket name.
`prefix`	`string`	No	Optional object key prefix (omit the leading slash). Defaults to an empty string. Default: `""`
`aws_access_key_id`	`string`	No	Explicit AWS access key ID. When omitted, boto3 uses its default credential chain.
`aws_secret_access_key`	`string`	No	Explicit AWS secret access key used with 'aws_access_key_id'.
`aws_session_token`	`string`	No	Session token for temporary credentials (for example STS AssumeRole).
`region_name`	`string`	No	AWS region for S3 client. Default: `"us-east-1"`
`endpoint_url`	`string`	No	Custom endpoint URL for S3-compatible services such as LocalStack or MinIO.
`use_ssl`	`boolean`	No	Whether to use HTTPS when talking to the endpoint (default True). Default: `true`
`verify`	`boolean`	No	Whether to verify SSL certificates when 'use_ssl' is True (default True). Default: `true`

Examples

Archive events to S3

Save each event to S3 for long-term storage

type: aws_s3
bucket: company-data-archive
prefix: events/processed/
aws_access_key_id: ${env:aws_access_key}
aws_secret_access_key: ${env:aws_secret_key}
region_name: us-east-1

Dated folder organization

Organize events by year/month for easier management

type: aws_s3
bucket: event-logs
prefix: logs/2024/11/
aws_access_key_id: ${env:aws_access_key}
aws_secret_access_key: ${env:aws_secret_key}
region_name: us-west-2

MinIO local storage

Use MinIO for local S3-compatible development

type: aws_s3
bucket: dev-events
prefix: test-data/
aws_access_key_id: minioadmin
aws_secret_access_key: minioadmin
region_name: us-east-1
endpoint_url: http://localhost:9000
use_ssl: false
verify: false

Multi-region backup

Store events in a different region for disaster recovery

type: aws_s3
bucket: backup-events-eu
prefix: backups/
aws_access_key_id: ${env:aws_access_key}
aws_secret_access_key: ${env:aws_secret_key}
region_name: eu-west-1

Advanced Options

These options are available on all steps for error handling and retry logic:

Parameter	Type	Default	Description
`retries`	`integer`	`0`	Number of retry attempts (0-10)
`backoff_seconds`	`number`	`0`	Backoff (seconds) applied between retry attempts
`retry_propagate`	`boolean`	`false`	If True, raise last exception after exhausting retries; otherwise swallow.

← Back to All Steps