S3 Server Access Logging is surprisingly absent from most default AWS configurations, leaving a huge blind spot for understanding who’s touching your buckets and how.
Let’s see it in action. Imagine a simple scenario: you’ve got a public S3 bucket for serving static assets, and you want to know if anyone’s actually hitting it.
# First, enable logging for your source bucket
aws s3api put-bucket-logging \
--bucket my-public-assets \
--bucket-logging-status '{
"LoggingEnabled": {
"TargetBucket": "my-s3-access-logs",
"TargetPrefix": "assets-log/"
}
}'
# Now, let's simulate some access to the public bucket
# (This would typically be a user's browser or an application)
aws s3api get-object \
--bucket my-public-assets \
--key index.html \
--request-payer requester # If your bucket is requester pays
A few minutes later, you’ll see log files appearing in your my-s3-access-logs bucket, under the assets-log/ prefix.
The core problem S3 Server Access Logging solves is providing an audit trail for requests made to your S3 buckets. Without it, you’re flying blind on:
- Who is accessing your data: IP addresses, user agents, and sometimes even authenticated AWS principals.
- What operations are being performed:
GET,PUT,DELETE,HEAD, etc. - When the access occurred: Timestamps with millisecond precision.
- Where the requests originated from: Geographic location based on IP.
- How much data was transferred: Bytes sent and received.
This information is crucial for security analysis, operational troubleshooting, and even billing.
How it Works Internally:
When you enable server access logging, S3 starts writing log records for every request made to the source bucket. These records are batched and delivered to a specified target bucket. The target bucket can be the same as the source bucket, but it’s best practice to use a separate bucket for security and organization. You can also specify a prefix within the target bucket to organize logs by date, application, or any other criteria.
The log files are delivered in Apache common log format (or a slightly modified version). They are delivered approximately hourly, though there can be delays. Each log file contains records for requests made during a specific time window.
Key Configuration Levers:
TargetBucket: The name of the S3 bucket where log files will be stored. This bucket must exist and have appropriate permissions for S3 to write to it.TargetPrefix(Optional): A prefix that S3 will add to log file names. This is incredibly useful for organizing logs, for example,my-logs/assets/ormy-logs/application-a/.- Permissions on the Target Bucket: The target bucket needs a bucket policy that grants
s3:PutObjectpermission to the S3 service principal (logging.s3.amazonaws.com). This is automatically handled if you use the AWS CLI or SDK to enable logging, but it’s worth verifying. - Permissions on the Source Bucket: The source bucket doesn’t need special permissions for logging itself, but the principal that enables logging does.
Example Bucket Policy for Target Bucket:
If your target bucket is named my-s3-access-logs and your source bucket is my-public-assets, the policy on my-s3-access-logs should look something like this:
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "GrantS3ServerAccessLogging",
"Effect": "Allow",
"Principal": {
"Service": "logging.s3.amazonaws.com"
},
"Action": "s3:PutObject",
"Resource": "arn:aws:s3:::my-s3-access-logs/*",
"Condition": {
"StringEquals": {
"s3:x-amz-acl": "bucket-owner-full-control"
}
}
},
{
"Sid": "GrantS3ServerAccessLoggingAcl",
"Effect": "Allow",
"Principal": {
"Service": "logging.s3.amazonaws.com"
},
"Action": "s3:PutObjectAcl",
"Resource": "arn:aws:s3:::my-s3-access-logs/*",
"Condition": {
"StringEquals": {
"s3:x-amz-acl": "bucket-owner-full-control"
}
}
}
]
}
The StringEquals condition on s3:x-amz-acl is important for ensuring the logs are owned by the bucket owner, not the logging service.
Analyzing the Logs:
Once logs start accumulating, you can analyze them using various tools:
- AWS Athena: Query logs directly in S3 using SQL. You’ll need to define a table schema that matches the log format.
- AWS CloudWatch Logs: Ingest logs into CloudWatch Logs for real-time monitoring and alarms.
- Custom Scripts: Download logs and parse them with Python, Perl, or other scripting languages.
- Third-Party Tools: Many security and log management solutions integrate with S3 logs.
When you enable logging for a bucket, S3 doesn’t retroactively log past requests; it only starts logging from the moment the configuration is applied.
This system is foundational for understanding your S3 traffic, and the next logical step is often integrating these logs into a more comprehensive security information and event management (SIEM) system.