Getting the big picture with S3 Storage Lens

S3 Storage Lens is an organization-wide dashboard for your storage. It aggregates usage and activity across all your accounts, regions and buckets into a single place. So instead of clicking through buckets one by one to answer questions such as "Which buckets hold most of my data?" you get a unified view with filters to precisely help you with that.

The good thing is that every account gets a default dashboard with free metrics, no setup required. The free tier gives you several usage metrics with 14 days of retention, which is already enough to spot some offenders. For example:

• Buckets with a huge amount of noncurrent object versions (versioning enabled, lifecycle rules forgotten);
• Incomplete multipart upload bytes sitting there for months;
• Buckets that are 100% Standard storage even though nothing suggests they need to be.

If you want to go deeper, there are the advanced metrics and recommendations: activity metrics (requests, bytes downloaded, etc.), advanced cost optimization and data protection metrics, prefix-level aggregation, CloudWatch publishing and 15 months of retention. These are paid though, and we'll get back to that in a bit.

Understanding access patterns with S3 Storage Class Analysis

Storage Lens gives you the big picture while Storage Class Analysis tells you your storage's access patterns. You enable it per bucket (optionally filtered by prefix or tags) and it starts watching your objects' access patterns: how much data is stored, how much is retrieved, and how that changes through time.

After observing for at least 30 days, it gives you a recommendation on when objects become "infrequently accessed", which is your cue to create a lifecycle rule transitioning them to Standard-IA. You can also export the results daily as CSV to a bucket and slice them however you like (QuickSight, a spreadsheet, whatever makes you happy).

It's worth noting that the analysis only makes recommendations for transitions from Standard to Standard-IA, and it's also a paid feature, charged per million objects monitored. If your access patterns are unpredictable and you just want to stop thinking about the problem, S3 Intelligent-Tiering might be a simpler answer than analyzing anything at all.

Don't let these tools become your new costs

Both Storage Lens' advanced metrics and Storage Class Analysis charge you per million objects monitored, per month. Turn them on across your accounts indiscriminately, where you might have billions of small objects (such as build outputs), and congratulations, you've just blown your AWS bill.

The way to avoid that is to do the heavy lifting with the free tier first:

1. Start with the free Storage Lens metrics. They offer pretty decent data for you to build a shortlist of offenders (try using the Pareto principle here);
2. Enable the paid tools only on the shortlist and estimate the costs before turning them on (you may do that with AWS Pricing Calculator). Turn on Storage Class Analysis for specific buckets, and use prefix or tag filters to narrow it down further instead of monitoring every single object. Use the same logic for Storage Lens advanced metrics: scope the dashboard to the accounts or buckets you're actually studying;
3. Turn them off when you're done. Once you have gathered enough information to define what you need to do and your lifecycle rules are in place, disable the analysis. It's a diagnostic tool, not an Amazon Prime subscription. The same goes for Storage Lens.

Wrap-up

Use free Storage Lens metrics (advanced ones, if needed) to find the suspects, Storage Class Analysis on the few buckets that justify it, then apply lifecycle rules to act on what you learned and switch the paid monitoring off. That's it, see you!