Articles

S3 Recursive Delete: Remove All Objects from Large Buckets (2026)

April 18, 2026 7 min read

Deleting all objects from a large S3 bucket recursively can be a surprisingly tricky operation if you’re not careful, often leading to timeouts, throttling, or even accidental data loss if not handled correctly.

Here’s how you can effectively remove all objects from a large S3 bucket, covering the common pitfalls and providing robust solutions.

The Problem with Simple Deletes

A naive approach might be to just run aws s3 rm s3://your-bucket-name --recursive. This works fine for small buckets. However, for buckets with millions or billions of objects, this command executes a massive number of individual DeleteObject API calls. S3 has rate limits, and even though they are high, a very large number of objects will eventually hit them. You’ll see SlowDown errors, and the operation will stall or take an unfeasibly long time. Furthermore, if the command is interrupted, you’re left with a partially deleted bucket, and restarting it might mean re-deleting already deleted objects or facing the same throttling issues.

The Efficient Solution: Batch Deletion with Lifecycle Policies

The most robust and scalable way to delete all objects from a large bucket is to leverage S3’s batch operations, specifically through lifecycle policies or the S3 Batch Operations API. For a complete wipe, a lifecycle policy is often the simplest.

Method 1: Using S3 Lifecycle Policies

This method doesn’t involve running commands directly to delete objects. Instead, you configure a rule that tells S3 to expire objects after a certain period. To delete all objects immediately, you set the expiration to 0 days.

Diagnosis: If you’re considering this, you likely have a bucket you want to clear out, and the aws s3 rm --recursive command is proving too slow or error-prone.

Fix:

  1. Navigate to your S3 bucket in the AWS Management Console.
  2. Go to the "Management" tab.
  3. Under "Lifecycle rules," click "Create lifecycle rule."
  4. Give your rule a name (e.g., DeleteAllObjects).
  5. Choose "Apply to all objects in the bucket." (If you want to be more specific, you can filter by prefix, but for a full wipe, this is the option).
  6. Under "Lifecycle rule actions," select "Expire current versions of objects."
  7. In the "Days after object creation" field, enter 0.
  8. Review and create the rule.

Why it works: Setting the expiration to 0 days tells S3 to mark objects for deletion as soon as the lifecycle policy runs. S3 then handles the deletion of these objects in the background, distributed across its infrastructure, without you needing to manage API calls or worry about throttling. It’s an asynchronous, managed process.

Next Error: If you try to access an object that has been marked for deletion by a lifecycle rule but not yet physically removed by S3’s internal cleanup, you might get a 404 Not Found error.

Method 2: S3 Batch Operations (for more control or specific scenarios)

S3 Batch Operations allows you to perform bulk operations on S3 objects. You create a manifest of objects and then define an operation (like delete) to be performed on them. This is more complex than lifecycle policies but offers greater control.

Diagnosis: You need to delete objects based on a specific criteria (e.g., objects created before a certain date, or objects matching a complex prefix pattern) or want more visibility into the deletion progress than lifecycle policies offer.

Fix:

  1. Create a CSV file listing the objects to delete. Each line should be bucket_name,key,version_id (version_id is optional for non-versioned buckets).

    • Example for a non-versioned bucket:
      your-bucket-name,object1.txt,
your-bucket-name,folder/object2.jpg,
    • Example for a versioned bucket:
      your-bucket-name,object1.txt,1234567890abcdef
your-bucket-name,folder/object2.jpg,fedcba0987654321
    • You can generate this CSV using aws s3api list-objects-v2 --bucket your-bucket-name --query "Contents[].{Bucket:BucketName,Key:Key}" --output text and then scripting to format it, or using aws s3 ls s3://your-bucket-name/ --recursive --summarize --human-readable | awk '{print "your-bucket-name," $4 ","}' (for non-versioned) and then manually adding version IDs if needed. For very large lists, you’ll need to paginate list-objects-v2 and use a script.

  2. Upload this CSV file to an S3 bucket (it can be the same bucket or a different one). Let’s say you upload it as s3://your-manifest-bucket/delete-manifest.csv.

  3. Create an S3 Batch Operations job:

    aws s3control create-job \
    --account-id <your-aws-account-id> \
    --operation '{"S3DeleteObject": {}}' \
    --report '{"Bucket": "s3://your-report-bucket/", "Enabled": true, "Format": "Report_CSV_20180522", "Scope": "AllTasks"}' \
    --manifest '{"Spec": {"Format": "S3BatchOperations_CSV_20180522", "Fields": ["Bucket", "Key", "VersionId"]}, "Location": {"ObjectArn": "arn:aws:s3:::your-manifest-bucket/delete-manifest.csv", "ETag": "..."}}' \
    --priority 2 \
    --role-arn arn:aws:iam::<your-aws-account-id>:role/<your-s3-batch-operations-iam-role> \
    --description "Delete objects from large bucket"
    • Replace <your-aws-account-id> with your actual AWS account ID.
    • Replace your-manifest-bucket and delete-manifest.csv with your manifest file’s location.
    • Replace your-report-bucket with a bucket where you want job completion reports.
    • Replace arn:aws:iam::<your-aws-account-id>:role/<your-s3-batch-operations-iam-role> with an IAM role that has permissions for S3 Batch Operations (e.g., s3:GetObject, s3:DeleteObject, s3:PutObject for reports).
    • The ETag for the manifest location needs to be retrieved from the object’s metadata. You can get it using aws s3api head-object --bucket your-manifest-bucket --key delete-manifest.csv --query ETag --output text.

  4. Monitor the job using aws s3control list-jobs --account-id <your-aws-account-id> and aws s3control describe-job --account-id <your-aws-account-id> --job-id <the-job-id-from-list-jobs>.

Why it works: S3 Batch Operations is designed for massive scale. It breaks down the manifest into smaller tasks, distributes them across S3’s infrastructure, and handles retries and throttling automatically. You get detailed reports on what succeeded and failed.

Next Error: If the IAM role for S3 Batch Operations is missing permissions, the job will fail immediately with an access denied error, and you’ll see it in the job status.

Alternative: aws s3api delete-objects (for programmatic control, less common for full wipe)

While not as scalable as Batch Operations or lifecycle policies for billions of objects, aws s3api delete-objects is useful if you’re writing a script and want to delete objects in batches of up to 1000.

Diagnosis: You’re writing custom code or a script to delete objects and want to manage the batching yourself, rather than relying on higher-level tools. This is generally for smaller, controlled deletions or specific object selection logic.

Fix:

  1. List objects and group them into batches of 1000.
    # Example: Get first 1000 objects for deletion
OBJECTS_TO_DELETE=$(aws s3api list-objects-v2 --bucket your-bucket-name --query 'Contents[0:1000].{Key:Key}' --output json)

# Format for delete-objects API
DELETE_REQUEST=$(echo "$OBJECTS_TO_DELETE" | jq -c '[.[] | {Key: .Key}]')

# Execute the delete command
aws s3api delete-objects --bucket your-bucket-name --delete "$DELETE_REQUEST"
  2. Repeat this process for subsequent batches, handling pagination from list-objects-v2 and potential NextToken.

Why it works: The delete-objects API call is a single API call that can delete up to 1000 objects. This is far more efficient than calling delete-object for each individual object. You control the batching and retries in your script.

Next Error: If you don’t handle the NextToken correctly in list-objects-v2 or if your script fails mid-batch, you’ll end up with a partially deleted bucket and need to re-run or refine your script.

Key Takeaways for Large Buckets

  • Avoid aws s3 rm --recursive for massive buckets. It’s a recipe for throttling and incomplete operations.
  • Lifecycle policies are the simplest, most hands-off way for a complete bucket wipe.
  • S3 Batch Operations is powerful for controlled, large-scale deletions based on specific criteria, offering visibility and robustness.
  • Always use IAM roles for S3 Batch Operations jobs.
  • Monitor your jobs and reports meticulously, especially when using Batch Operations.

The next challenge you’ll likely encounter after successfully clearing a large bucket is managing the cost implications of the deletion process itself (e.g., costs associated with S3 Batch Operations requests or lifecycle rule evaluations) and ensuring you have appropriate backup and recovery strategies in place before performing such destructive operations.

Want structured learning?

Take the full S3 course →