Object storage systems often feel like a black box, but S3 notifications with EventBridge and SNS turn it into a real-time event stream, letting you react to changes as they happen.
Let’s see it in action. Imagine an S3 bucket, my-important-data-bucket, where we want to trigger an action whenever a new CSV file lands.
First, we need to set up the S3 event notification. This tells S3 what to send and where to send it. We’ll configure it to send s3:ObjectCreated:* events for objects ending in .csv to an EventBridge event bus.
{
"Id": "NotifyNewCSV",
"EventBusName": "default",
"SourceArn": "arn:aws:s3:::my-important-data-bucket",
"EventSource": "aws:s3",
"Events": [
"s3:ObjectCreated:*"
],
"Destination": {
"Arn": "arn:aws:events:us-east-1:123456789012:event-bus/default",
"Events": [
"s3:ObjectCreated:Put",
"s3:ObjectCreated:Post"
],
"StagingBucket": {
"Bucket": "my-staging-bucket-for-s3-events",
"KeyPrefix": "s3-event-staging/"
}
},
"Filter": {
"S3Key": {
"Rules": [
{
"Name": "suffix",
"Value": ".csv"
}
]
}
}
}
This configuration, when applied to my-important-data-bucket, means S3 will package up details about newly created .csv objects and send them to the default EventBridge event bus. The StagingBucket is a good practice for handling potential delivery issues or retries by S3 before the event is fully processed by EventBridge.
Now, on the default EventBridge bus, we create a rule that matches these incoming S3 events. This rule will then route the matched events to one or more targets. Let’s say we want to send these events to an SNS topic, csv-ingestion-topic, for further processing.
{
"Name": "RouteCSVCreatedToSNS",
"Description": "Routes S3 ObjectCreated events for .csv files to the CSV ingestion SNS topic",
"EventBusName": "default",
"EventPattern": {
"source": ["aws:s3"],
"detail-type": ["Object Created"],
"detail": {
"bucket": {
"name": ["my-important-data-bucket"]
},
"object": {
"key": [{
"suffix": ".csv"
}]
}
}
},
"State": "ENABLED",
"Targets": [
{
"Arn": "arn:aws:sns:us-east-1:123456789012:csv-ingestion-topic",
"Id": "SNS-Topic-Target",
"RoleArn": "arn:aws:iam::123456789012:role/EventBridgeInvokeSNSRole"
}
]
}
This EventBridge rule listens for events originating from aws:s3 with a detail-type of "Object Created", specifically looking for events where the bucket name is my-important-data-bucket and the object key ends with .csv. When a match is found, it invokes the specified SNS topic (csv-ingestion-topic) using the provided IAM role.
The result? Every time a CSV file is uploaded to my-important-data-bucket, an event payload containing the bucket name, object key, and other metadata is sent to csv-ingestion-topic. Subscribers to this topic (like Lambda functions or SQS queues) can then pick up this message and initiate downstream processing, such as data validation, transformation, or loading into a data warehouse.
The true power here is decoupling. S3 doesn’t need to know how the CSV files are processed, and the processing logic doesn’t need to poll S3. They communicate asynchronously through EventBridge and SNS, making the system more resilient and scalable.
What most people miss is how EventBridge’s detail-type for S3 events is normalized. S3 itself can emit various specific "Object Created" events (like s3:ObjectCreated:Put, s3:ObjectCreated:Post, s3:ObjectCreated:Copy, s3:ObjectCreated:CompleteMultipartUpload). However, when S3 sends these to EventBridge, they are consistently mapped to the generic detail-type: "Object Created". This means your EventBridge rule can simply match on "Object Created" and then use the detail.object.key or other fields for finer-grained filtering, rather than needing separate rules for each S3 creation event type.
From here, you’ll likely want to explore how to set up a Lambda function to subscribe to the SNS topic and perform specific actions based on the S3 event data.