Feature Request for S3 to Support SageMaker

While studying SageMaker before taking a certification test, I encountered a problem, took notes while looking at possible solutions, and then used an AI tool to summarize my notes and effectively describe what was needed. Here is what I would suggest:

Feature Request: S3 Lifecycle with "Active Version Preservation"

1. Introduction & Problem Statement:

Amazon S3 is the backbone for numerous AWS services and customer data lakes, offering versatile storage tiers for cost optimization. For services like Amazon SageMaker Feature Store, which uses S3 for its Offline Store, S3's versioning and lifecycle policies are critical for managing large volumes of historical feature data. However, current S3 Lifecycle policies can inadvertently lead to operational challenges and increased costs when dealing with long-lived, versioned data where the current version of an object (or feature record) might be chronologically old but still actively required.

The core issue arises when a standard lifecycle policy transitions data to colder tiers (e.g., S3 Glacier Flexible Retrieval, S3 Glacier Deep Archive) based purely on object age or version age. If a feature record's current and only version is older than the transition threshold, it gets archived. This makes it inaccessible for immediate use by SageMaker Feature Store queries (e.g., for generating training datasets or batch inference), leading to incomplete results or costly and time-consuming data restores. This problem is particularly acute for features that are updated infrequently; their latest state might be old, but it is vital.

2. Proposed New S3 Feature: "Lifecycle with Active Version Preservation"

I propose a new S3 Lifecycle policy rule option, "Active Version Preservation."

• Short Description: This enhancement to S3 Lifecycle policies would allow users to define rules for transitioning non-current object versions to colder storage tiers (e.g., S3 Standard-IA, S3 Glacier Flexible Retrieval) while explicitly ensuring that the current (latest) version of an object always remains in a readily accessible storage class (e.g., S3 Standard or S3 Standard-IA), irrespective of the current version's individual age.

How it would work: When configuring a lifecycle rule for versioned buckets, users could specify:

1. Actions for non-current versions (e.g., "Transition non-current versions to S3 Glacier Flexible Retrieval 90 days after they become non-current").

2. A new option: "Preserve current version in S3 Standard" (or a user-selectable active tier).

With this option enabled, S3 would manage the lifecycle of non-current versions as defined. Still, it would not transition any object's single, current version to the specified colder archive tiers, even if that current version's creation date would otherwise make it eligible under a separate age-based rule for current versions. If a new version of the object is uploaded, the previously current version becomes non-current and subject to the non-current version transition rules.

3. Solving the SageMaker Feature Store Offline Store Challenge

SageMaker Feature Store's Offline Store holds historical feature data in S3, typically organized by feature group, record identifier, and event time. This data is versioned, allowing for point-in-time correct queries crucial for model training and batch predictions. The Offline Store can grow very large, making S3 lifecycle policies essential for cost management.

The Challenge Today: Consider a critical feature within SageMaker Feature Store that is updated infrequently. For example, a customer's demographic segment might be recorded and only updated if it changes.

• Record_XYZ, Feature_A, Value_1, EventTime_T1 (current version) If Value_1 remains the current value for Feature_A for an extended period (e.g., 1 year), a standard S3 lifecycle policy like "Transition objects older than 180 days to S3 Glacier Flexible Retrieval" would move this Value_1 to Glacier. Subsequently, if SageMaker Feature Store attempts to generate a training dataset that includes Record_XYZ and requires the current value of Feature_A, it will encounter an issue:

• The data for Value_1 is in Glacier and not directly queryable by Athena (which backs the Offline Store queries).

• This leads to missing feature values in the training dataset or necessitates a manual, potentially slow, and costly restoration from Glacier.

• This scenario is common when a large number of records depend on a feature that only has one historical value (which is also its current value) that is past the age to move to Glacier, or has a current value, but previous essential versions needed for point-in-time correctness have been aggressively archived.

How "Active Version Preservation" Solves This: With the proposed S3 "Active Version Preservation" feature:

1. The SageMaker Feature Store Offline S3 bucket could have a lifecycle policy:

o "Transition non-current versions to S3 Glacier Flexible Retrieval 60 days after becoming non-current."

o "Enable Active Version Preservation in S3 Standard."

2. In the example above, Value_1 (the current version of Feature_A for Record_XYZ) would remain in S3 Standard indefinitely, regardless of age, as long as it is the latest version.

3. If Feature_A for Record_XYZ is updated to Value_2 at EventTime_T2, then Value_1 becomes a non-current version. Only then would it become a candidate for transition to S3 Glacier Flexible Retrieval after 60 days (from EventTime_T2). Value_2 (the new current version) would then be preserved in S3 Standard.

This ensures that SageMaker Feature Store can always access the latest state of any feature directly from S3 Standard, dramatically improving the reliability and performance of generating training/inference datasets, especially those involving features that change infrequently.

4. Emphasizing the Need and Benefits

The lack of a mechanism to protect the current version of an object from age-based archival in S3 is a significant pain point for applications like SageMaker Feature Store that rely on S3 as a versioned, historical data store.

• Critical for ML Operations: Consistent access to the latest feature values is paramount for reliable model training, retraining, and batch inference. Data being unexpectedly archived can break automated MLOps pipelines.

• Reduced Operational Overhead: Current workarounds involve either not using colder tiers (increasing costs), setting excessively long retention periods in hotter tiers, or building complex custom solutions to identify and re-ingest/cache critical "latest" values to prevent them from being archived – all of which are suboptimal.

• Cost Optimization without Compromise: This feature would allow aggressive archiving of truly historical, non-current versions to save costs while safeguarding the accessibility of essential current data.

• Simplified Data Management: It provides a straightforward, declarative way to manage the lifecycle of versioned data, aligning better with common access patterns where the latest version is most frequently needed.

• Improved Query Performance & Reliability: Ensures that queries against the SageMaker Offline Store for current feature values (or features whose historical value is still the current active value) do not fail or suffer from retrieval delays due to data being in archival tiers.

The "Active Version Preservation" feature would be a valuable addition to S3's lifecycle management capabilities. It directly addresses a key challenge faced by users of services like SageMaker Feature Store and other systems managing versioned datasets with long-lived current states in S3. I strongly recommend its consideration and implementation.

I had one doubt about this approach. I thought that Many records may have the same metric and a timestamp on a particular day. When that day arrives for the records to transition to Glacier, if the records are the last with a specific value for a feature, when they roll off of S3 standard, that value is no longer represented for the metric. If there are records with later dates and a different value for a feature, we can assume that those values hold true retroactively to the roll-off date. So, any new records inserted could pick up the feature value from the earliest record that has not yet been rolled off. The same would hold true if a new feature is created with a date later than the roll-off date and added to a record. Its value could be applied retroactively back to the roll-off date and for future records until one arrives with a different value.

However, this assumes or imputes values to interpret or fill in gaps that can arise from data sparsity or when historical (non-current) data is archived. The proposed S3 "Active Version Preservation" feature addresses the problem of keeping the actual, most current data readily accessible without requiring an explicit data processing step or a business rule applied during data consumption.