RDS log rotation is surprisingly essential for debugging performance issues because logs can grow to consume a significant portion of your database instance’s storage.
Let’s see this in action with a PostgreSQL RDS instance. Imagine we’re tracking slow queries. We’d first enable log_min_duration_statement in our RDS parameter group.
-- Connect to your RDS instance and run this:
ALTER SYSTEM SET log_min_duration_statement = '500ms';
SELECT pg_reload_conf();
This tells PostgreSQL to log any query taking longer than 500 milliseconds. Now, if we look at our RDS logs (accessible via the AWS console or CLI), we might see entries like this:
2023-10-27 10:30:00 UTC:[12345]:[45]:[user@db_name]:[remote_ip]:[session_id]:[LOG]:duration: 1234.567 ms statement: SELECT * FROM large_table WHERE some_column = 'some_value';
Over time, these logs accumulate. RDS handles rotation automatically, but understanding how and where these logs go is key. By default, RDS rotates logs daily. When a log file reaches its rotation point (either by size or time), it’s compressed and moved to a historical archive.
The primary mechanism for managing these logs is through the RDS console or the AWS CLI. You can view recent logs directly, or export historical logs to Amazon S3 for longer-term storage and analysis.
To export logs to S3, you’d typically configure an export task. Here’s a simplified CLI example:
aws rds create-export-task \
--export-task-identifier my-log-export \
--source-arn arn:aws:rds:us-east-1:123456789012:log:my-rds-instance-name \
--s3-bucket-name my-rds-log-bucket \
--kms-key-id arn:aws:kms:us-east-1:123456789012:key/your-kms-key-id \
--export-from 1698393600000 \
--export-to 1698480000000
This command initiates an export of logs from a specific RDS instance (source-arn) to a designated S3 bucket (s3-bucket-name), using a KMS key for encryption. The export-from and export-to parameters specify a Unix timestamp range for the logs you want to retrieve.
The mental model here is that RDS acts as a managed service that abstracts away the complexities of log management. It ensures logs are consistently available for a defined period, then provides mechanisms to archive them further. The "rotation" itself is a background process managed by RDS, creating new log files and compressing old ones, typically based on time intervals (e.g., daily) or file size limits, whichever comes first. These archived logs are then accessible for download or export.
What many people don’t realize is that the log files themselves are not directly accessible on the instance’s filesystem like you might expect with a self-hosted database. Instead, RDS provides APIs and console interfaces to interact with these logs, effectively treating them as a managed resource rather than raw files. This abstraction simplifies operations but requires understanding the specific AWS tooling for access.
The next logical step after exporting logs is to analyze them for actionable insights using tools like Amazon CloudWatch Logs Insights or Athena.