Amazon introduced a feature to process data as it is being retrieved from S3. This allows use cases such as on-the-fly format conversion or PII redaction.
Amazon announce it launched S3 Object Lambda last Thursday, enabling developers to add code to the S3 storage data retrieval pipeline. This allows the data to be processed before it reaches the application that called it, enabling use cases such as personally identifiable information (PII) masking and compressing or decompressing files as they are downloaded.
This function is particularly helpful in situations where multiple applications need different versions of the same data. In this case, customized version of the data, such as a “clean” version that’s been stripped of PII (e.g. for GDPR compliance) or an “enhanced” version containing metadata from other services or databases needs to be created.
Amazon S3 Object Lambda saves users from needing to generate those extra copies, which take up storage space. Additionally, the Lambda function is performed along a standard S3 GET request, so there is no coding change necessary at the application level. Other ways Amazon S3 Object Lambda can modify data as it’s being retrieved include resizing images, converting data formats and implementing custom access rules.
S3 Object Lambda is available now in all AWS regions except for Asia Pacific (Osaka), AWS GovCloud (US-East), AWS GovCloud (US-West), China (Beijing) and China (Ningxia). Customers are charged for the compute required to execute Lambda changes and for the data called by applications.