S3 access logs and CloudTrail are both AWS services that record activity, but they serve fundamentally different purposes, and picking the wrong one for auditing can lead to missing critical security events or drowning in irrelevant data.

Let’s see them in action. Imagine you’re tracking who’s been looking at your sensitive S3 buckets.

First, S3 access logs. You enable them on a bucket.

aws s3api put-bucket-logging --bucket my-sensitive-bucket --bucket-logging-status '{
    "LoggingEnabled": true,
    "TargetBucket": "my-s3-access-logs-bucket",
    "TargetPrefix": "my-sensitive-bucket-logs/"
}'

Now, every GET, PUT, DELETE request to my-sensitive-bucket will land in my-s3-access-logs-bucket under the my-sensitive-bucket-logs/ prefix. These logs are raw, focused only on S3 object-level operations. A typical entry looks like this:

<bucket-owner-aws-account-id> [YYYY-MM-DDThh:mm:ss.sssZ] <remote-ip> <requester-id> <request-id> <operation> <key> <request-uri> <http-status> <error-code> <bytes-sent> <object-size> <total-time> <turn-around-time> <referrer> <user-agent> <version-id>

For example: 123456789012 [2023-10-27T10:30:00.000Z] 192.0.2.1 - - GET /my-sensitive-bucket/confidential.pdf HTTP/1.1 ... 200 - 1024 1024 123 45 - "Mozilla/5.0 ..." -

This tells you that confidential.pdf was downloaded, when, and from where. It’s granular for S3, but that’s it. It won’t tell you if someone tried to create a new IAM user or modify a security group.

Now, CloudTrail. You enable it for your entire AWS account.

aws cloudtrail create-trail --name my-account-trail --s3-bucket-name my-cloudtrail-logs-bucket --is-multi-region-trail

CloudTrail logs API calls made to any AWS service. It’s the system’s command history. A CloudTrail event looks like this JSON:

{
    "eventVersion": "1.08",
    "userIdentity": {
        "type": "IAMUser",
        "principalId": "AIDAJQILXXXXXXEXAMPLE",
        "arn": "arn:aws:iam::123456789012:user/Alice",
        "accountId": "123456789012",
        "accessKeyId": "AKIAIOSFXXNEXAMPLE",
        "userName": "Alice"
    },
    "eventTime": "2023-10-27T10:35:15Z",
    "eventSource": "s3.amazonaws.com",
    "eventName": "GetObject",
    "awsRegion": "us-east-1",
    "sourceIPAddress": "192.0.2.1",
    "userAgent": "aws-cli/2.13.17 Python/3.11.5 Darwin/23.0.0",
    "requestParameters": {
        "bucketName": "my-sensitive-bucket",
        "key": "confidential.pdf"
    },
    "responseElements": null,
    "requestID": "EXAMPLE-REQUEST-ID",
    "eventID": "EXAMPLE-EVENT-ID",
    "readOnly": true,
    "eventType": "AwsApiCall",
    "managementEvent": true,
    "recipientAccountId": "123456789012"
}

Notice the eventSource and eventName. This CloudTrail event records Alice, using the AWS CLI, performing a GetObject on my-sensitive-bucket/confidential.pdf. This is the same event as the S3 access log entry, but CloudTrail provides context: who made the call (Alice, an IAM user), how they made it (AWS CLI), and that it was an API call within the AWS control plane.

The fundamental difference: S3 access logs are a data access log for S3 only. CloudTrail is an API call log for all AWS services.

When to use S3 access logs:

  • Object-level forensics: You need to know exactly which object was accessed, by whom (if authenticated), from where, and the HTTP status. This is for detailed analysis of S3 data access patterns, like tracking downloads of specific files.
  • Third-party integrations: Some tools or compliance requirements might specifically ask for S3 access logs in their native format.

When to use CloudTrail:

  • Security auditing: This is its primary purpose. You want to know who changed what IAM policy, who launched an EC2 instance, who modified security group rules, or who deleted a bucket. CloudTrail captures these administrative actions.
  • Compliance: Many regulations require logging API activity. CloudTrail is the standard for this.
  • Operational troubleshooting: If an S3 bucket was accidentally deleted, CloudTrail will show you the DeleteBucket API call, who made it, and when. S3 access logs won’t show the DeleteBucket operation itself, only subsequent access attempts to a non-existent bucket.
  • Cross-service correlation: CloudTrail events can link actions across different services. For example, an IAM user creating a new role (iam:CreateRole) and then that role being used to launch an EC2 instance (ec2:RunInstances).

Common Pitfalls & How to Fix:

  1. Missing Admin Activity: You enabled S3 access logs but are looking for who deleted a security group.

    • Diagnosis: Check if CloudTrail is enabled for your account and configured to log management events.
    • Fix: Ensure CloudTrail is enabled. Go to the CloudTrail console, select your trail, and verify "Log management events" is set to "All".
    • Why it works: S3 access logs only record GET/PUT/DELETE object operations. CloudTrail records API calls like DeleteSecurityGroup.
  2. No Object-Level Detail for S3: You need to know which specific S3 object was accessed, but CloudTrail only shows GetObject for a bucket.

    • Diagnosis: Review your CloudTrail configuration. By default, CloudTrail logs management events (like ListBucket, GetObject, PutObject) but doesn’t always log data events for S3, which include specific object details.
    • Fix: For the relevant trail, add S3 data event selectors. In the AWS CLI:
      aws cloudtrail put-event-selectors --trail-name my-account-trail \
      --event-selectors '[
          {
              "ReadWriteType": "All",
              "IncludeManagementEvents": true,
              "DataResources": [
                  {
                      "Type": "AWS::S3::Object",
                      "Values": [
                          "arn:aws:s3:::my-sensitive-bucket/"
                      ]
                  }
              ]
          }
      ]'
      
      You can specify specific buckets or all buckets.
    • Why it works: CloudTrail’s default configuration prioritizes management events. Explicitly configuring data event selectors tells CloudTrail to also log detailed S3 object operations. Note that enabling S3 data events can incur additional costs.
  3. Logs Not Reaching Destination: S3 access logs or CloudTrail logs are not appearing in their target buckets.

    • Diagnosis:
      • S3 Access Logs: Check the TargetBucket and TargetPrefix in your put-bucket-logging configuration. Ensure the target bucket exists and the bucket owner has write permissions to it.
      • CloudTrail: Check the s3-bucket-name in your create-trail or update-trail command. Ensure the bucket exists and CloudTrail has permission to write to it (CloudTrail creates a bucket policy for you if it creates the bucket). Also, check if the trail is enabled.
    • Fix: Correct the TargetBucket/s3-bucket-name if it’s wrong. Manually verify bucket ownership and permissions. Ensure the trail is enabled: aws cloudtrail update-trail --name my-account-trail --include-management-events --is-enabled.
    • Why it works: The services need a valid, accessible destination to store their logs. Incorrect configuration or permissions prevent log delivery.
  4. Incomplete Audit Trail (Multi-Region): You’re only seeing logs from one region, but your resources are spread out.

    • Diagnosis: For CloudTrail, check if the trail is configured as a multi-region trail. For S3 access logs, they are bucket-specific and thus region-specific unless you configure logging for buckets in every region.
    • Fix: For CloudTrail, create or update your trail to be multi-region: aws cloudtrail update-trail --name my-account-trail --is-multi-region-trail. For S3, you must configure logging on each bucket individually, potentially consolidating logs into a central bucket using S3 replication or a separate log ingestion process.
    • Why it works: By default, CloudTrail trails are regional. Multi-region trails ensure API activity from all regions is captured in one place. S3 access logs are a bucket property, requiring explicit configuration per bucket.
  5. Cost Concerns with Data Events: Enabling S3 data events for all buckets is generating excessive logs and costs.

    • Diagnosis: Review the volume of logs in your CloudTrail bucket, specifically for S3 data events.
    • Fix: Refine your put-event-selectors to include only specific, high-risk buckets or prefixes. Consider a hybrid approach: use CloudTrail for administrative actions and common S3 data access, and enable S3 server access logs (which are often cheaper and simpler for raw object access) only for the most critical buckets where object-level detail is paramount.
    • Why it works: S3 data events in CloudTrail are more expensive because they capture every object interaction. Limiting their scope or using S3 access logs for less critical scenarios reduces log volume and cost.
  6. Confusing Log Formats: You’re trying to parse S3 access logs as if they were CloudTrail JSON, or vice-versa.

    • Diagnosis: Look at the file format. S3 access logs are text-based, often space-delimited or with specific delimiters. CloudTrail logs are JSON.
    • Fix: Use appropriate parsing tools. For S3 access logs, awk, sed, or custom scripts are common. For CloudTrail, use JSON parsers like jq or AWS SDKs.
    • Why it works: Different log formats require different processing tools. Misinterpreting the format leads to parsing errors and incorrect analysis.

When you’ve correctly configured both S3 access logs (for object-level detail on specific buckets) and CloudTrail (for account-wide API activity, including S3 management events and S3 data events for critical buckets), you’ll have a robust auditing and security monitoring strategy.

The next thing you’ll likely grapple with is how to efficiently query and analyze these vast log datasets, which often leads to exploring services like Amazon Athena or CloudWatch Logs.

Want structured learning?

Take the full S3 course →