AWS实战：S3 Cloud Watch Event 触发Lambda

最新推荐文章于 2024-09-07 09:52:41 发布

JessicaWind

最新推荐文章于 2024-09-07 09:52:41 发布

阅读量1.5k

点赞数

分类专栏： AWS实战文章标签： aws

本文链接：https://blog.youkuaiyun.com/meiyubaihe/article/details/123118137

版权

AWS实战专栏收录该内容

13 篇文章

订阅专栏

架构

既然是S3 Cloud Watch Event 触发Lambda，首先就需要三个AWS的service:

S3
Event Bridge
Lambda

S3有event产生时向Event Bridge发送event，Event Bridge通过event rule的配置过滤event，将符合规则的event发送给lambda进行处理。

S3如何向Event Bridge发送Event

S3默认情况下是不会主动向Event Bridge发送event的，需要enable Amazon EventBridge的配置，才会将event发送给Event Bridge，可以通过一下三种方式进行配置

AWS Web Console：bucket页面->Properties->Event notifications ->Amazon EventBridge -> on

AWS CLI

打开

aws s3api put-bucket-notification-configuration --bucket DOC-EXAMPLE-BUCKET1 --notification-configuration '{ "EventBridgeConfiguration": {} }'

关闭

aws s3api put-bucket-notification-configuration --bucket DOC-EXAMPLE-BUCKET1 --notification-configuration '{}'

Serverless

SourceEventBucket:
  Type: AWS::S3::Bucket
  Properties:
    BucketName: !Sub com.jessica.${self:provider.stage}-source-event-bucket
    NotificationConfiguration:
      EventBridgeConfiguration:
        EventBridgeEnabled: true

enable Amazon EventBridge之后，默认情况下，S3会将所有的event发送给Amazon EventBridge，S3的event类型如下：

Event type	Description
Object Created	An object was created. The reason field in the event message structure indicates which S3 API was used to create the object: PutObject, POST Object, CopyObject, or CompleteMultipartUpload.
Object Deleted (DeleteObject) Object Deleted (Lifecycle expiration)	An object was deleted. When an object is deleted using an S3 API call, the reason field is set to DeleteObject. When an object is deleted by an S3 Lifecycle expiration rule, the reason field is set to Lifecycle Expiration. For more information, see Expiring objects. When an unversioned object is deleted, or a versioned object is permanently deleted, the deletion-type field is set to Permanently Deleted. When a delete marker is created for a versioned object, the deletion-type field is set to Delete Marker Created. For more information, see Deleting object versions from a versioning-enabled bucket.
Object Restore Initiated	An object restore was initiated from S3 Glacier or S3 Glacier Deep Archive storage class or from S3 Intelligent-Tiering Archive Access or Deep Archive Access tier. For more information, see Working with archived objects.
Object Restore Completed	An object restore was completed.
Object Restore Expired	The temporary copy of an object restored from S3 Glacier or S3 Glacier Deep Archive expired and was deleted.
Object Storage Class Changed	An object was transitioned to a different storage class. For more information, see Transitioning objects using Amazon S3 Lifecycle.
Object Access Tier Changed	An object was transitioned to the S3 Intelligent-Tiering Archive Access tier or Deep Archive Access tier. For more information, see Amazon S3 Intelligent-Tiering.
Object ACL Updated	An object's access control list (ACL) was set using PutObjectACL. An event is not generated when a request results in no change to an object’s ACL. For more information, see Access control list (ACL) overview.
Object Tags Added	A set of tags was added to an object using PutObjectTagging. For more information, see Categorizing your storage using tags.
Object Tags Deleted	All tags were removed from an object using DeleteObjectTagging. For more information, see Categorizing your storage using tags.

S3向EventBridge发送event不需要额外的权限配置

Event Bridge如何对Event进行过滤

Event Bridget通过在event rule中指定event pattern来对event进行过滤

这是一个S3 put object的event示例

{
  "version": "0",
  "id": "17793124-05d4-b198-2fde-7ededc63b103",
  "detail-type": "Object Created",
  "source": "aws.s3",
  "account": "123456789012",
  "time": "2021-11-12T00:00:00Z",
  "region": "ca-central-1",
  "resources": ["arn:aws:s3:::example-bucket"],
  "detail": {
    "version": "0",
    "bucket": {
      "name": "example-bucket"
    },
    "object": {
      "key": "example-key",
      "size": 5,
      "etag": "b1946ac92492d2347c6235b4d2611184",
      "version-id": "IYV3p45BT0ac8hjHg1houSdS1a.Mro8e",
      "sequencer": "00617F08299329D189"
    },
    "request-id": "N4N7GDK58NMKJ12R",
    "requester": "123456789012",
    "source-ip-address": "1.2.3.4",
    "reason": "PutObject"
  }
}

以下为event pattern示例

过滤所有s3event
```
{
  "source": ["aws.s3"]
}
```
过滤指定账号下所有s3event

{
  "source": ["aws.s3"],
  "account": ["123456789012"],
}

过滤指定账号下某个bucket所有s3event

{
  "source": ["aws.s3"],
  "account": ["123456789012"],
  "resources": ["arn:aws:s3:::example-bucket"]
}

过滤指定账号下某个bucket所有s3 put object event

{
  "source": ["aws.s3"],
  "account": ["123456789012"],
  "detail-type": ["Object Created"],
  "resources": ["arn:aws:s3:::example-bucket"]
}

过滤指定账号下某个bucket指定prefix的所有s3 put object event

{
  "source": ["aws.s3"],
  "account": ["123456789012"],
  "detail-type": ["Object Created"],
  "resources": ["arn:aws:s3:::example-bucket"],
  "detail": {
    "object" : {
      "key":[{"prefix":"example"}]
    }
  }
}

event pattern 除了支持精确匹配外还支持一些简单的规则匹配，比如prefix，支持列表如下

Comparison	Example	Rule syntax	Supported by Pipes
Null	UserID is null	"UserID": [ null ]	Yes
Empty	LastName is empty	"LastName": [""]	Yes
Equals	Name is "Alice"	"Name": [ "Alice" ]	Yes
Equals (ignore case)	Name is "Alice"	"Name": [ { "equals-ignore-case": "alice" } ]	No
And	Location is "New York" and Day is "Monday"	"Location": [ "New York" ], "Day": ["Monday"]	Yes
Or	PaymentType is "Credit" or "Debit"	"PaymentType": [ "Credit", "Debit"]	Yes
Or (multiple fields)	Location is "New York", or Day is "Monday".	"$or": [ { "Location": [ "New York" ] }, { "Day": [ "Monday" ] } ]	No
Not	Weather is anything but "Raining"	"Weather": [ { "anything-but": [ "Raining" ] } ]	Yes
Numeric (equals)	Price is 100	"Price": [ { "numeric": [ "=", 100 ] } ]	Yes
Numeric (range)	Price is more than 10, and less than or equal to 20	"Price": [ { "numeric": [ ">", 10, "<=", 20 ] } ]	Yes
Exists	ProductName exists	"ProductName": [ { "exists": true } ]	Yes
Does not exist	ProductName does not exist	"ProductName": [ { "exists": false } ]	Yes
Begins with	Region is in the US	"Region": [ {"prefix": "us-" } ]	Yes
Ends with	FileName ends with a .png extension.	"FileName": [ { "suffix": ".png" } ]	No

Event Bridge如何触发Lambda调用

Event Bridge想要触发lambda，则Event Bridge Service(events.amazonaws.com)必须要有调用lambda的权限，可以通过创建lambda permission来实现

    LambdaInvokePermission:
      Type: AWS::Lambda::Permission
      Properties:
        FunctionName: !GetAtt S3EventHandlerLambdaFunction.Arn
        Action: lambda:InvokeFunction
        Principal: events.amazonaws.com
        SourceArn: !GetAtt S3EventRule.Arn

Serverless部署资源

以下为本文用到的所有资源的serverless部署文件，有两个版本，serverless.yml和serverless-manual.yml

serverless.yml为利用serverless framework的一些规则自动创建event rule和lambda permission，比较适合新手入门，不清楚有哪些资源需要创建的时候使用，serverless framework会将所有需要的资源创建自动加入cloudformation文件，缺点是用户无法自定义资源名称

service: event-bridge-trigger-lambda

custom:
  bucketNamePrefix: "bucketName"

provider:
  name: aws
  runtime: java8
  memorySize: 512
  timeout: 900
  deploymentBucket:
    name: com.${self:custom.bucketNamePrefix}.deploy-bucket
    serverSideEncryption: AES256
  stackName: ${self:service}
  region: ${opt:region, 'ap-southeast-1'}
  stage: ${opt:stage, 'develop'}
  versionFunctions: false

resources:
  Resources:
    SourceEventBucket:
      Type: AWS::S3::Bucket
      Properties:
        BucketName: com.jessica.${self:provider.stage}-source-event-bucket
        NotificationConfiguration:
          EventBridgeConfiguration:
            EventBridgeEnabled: true
    LambdaRole:
      Type: AWS::IAM::Role
      Properties:
        RoleName: ${self:provider.stage}_LambdaRole
        AssumeRolePolicyDocument:
          Version: "2012-10-17"
          Statement:
            - Effect: Allow
              Principal:
                Service:
                  - lambda.amazonaws.com
              Action: sts:AssumeRole
        ManagedPolicyArns:
          - arn:aws:iam::aws:policy/AmazonS3FullAccess
          - arn:aws:iam::aws:policy/CloudWatchFullAccess
    S3EventRule:
      Type: AWS::Events::Rule
      Properties:
        Description: event rule for s3 event
        EventPattern:
          source:
            - aws.s3
          detail-type:
            - Object Created
            - Object Deleted
          detail:
            bucket:
              name:
                - com.jessica.${self:provider.stage}-source-event-bucket
        Name: s3-event
        State: ENABLED
        Targets:
          - Arn: !GetAtt S3EventHandlerLambdaFunction.Arn
            Id: s3-event-handler
    LambdaInvokePermission:
      Type: AWS::Lambda::Permission
      Properties:
        FunctionName: !GetAtt S3EventHandlerLambdaFunction.Arn
        Action: lambda:InvokeFunction
        Principal: events.amazonaws.com
        SourceArn: !GetAtt S3EventRule.Arn
package:
  artifact: target/event-bridge-trigger-lambda.jar

functions:
  S3EventHandler:
    name: ${self:provider.stage}-S3EventHandler
    handler: com.jessica.aws.lambda.S3EventHandler
    role: !GetAtt LambdaRole.Arn
    maximumRetryAttempts: 0
    description: s3 event handler

serverless-manual.yml则在serverless文件中指定需要创建的event rule和lambda permission，自己进行创建

service: event-bridge-trigger-lambda

custom:
  bucketNamePrefix: "bucketName"

provider:
  name: aws
  runtime: java8
  memorySize: 512
  timeout: 900
  deploymentBucket:
    name: com.${self:custom.bucketNamePrefix}.deploy-bucket
    serverSideEncryption: AES256
  stackName: ${self:service}
  region: ${opt:region, 'ap-southeast-1'}
  stage: ${opt:stage, 'develop'}
  versionFunctions: false

resources:
  Resources:
    SourceEventBucket:
      Type: AWS::S3::Bucket
      Properties:
        BucketName: !Sub com.jessica.${self:provider.stage}-source-event-bucket
        NotificationConfiguration:
          EventBridgeConfiguration:
            EventBridgeEnabled: true
    LambdaRole:
      Type: AWS::IAM::Role
      Properties:
        RoleName: !Sub "${self:provider.stage}_LambdaRole"
        AssumeRolePolicyDocument:
          Version: "2012-10-17"
          Statement:
            - Effect: Allow
              Principal:
                Service:
                  - lambda.amazonaws.com
              Action: sts:AssumeRole
        ManagedPolicyArns:
          - arn:aws:iam::aws:policy/AmazonS3FullAccess
          - arn:aws:iam::aws:policy/CloudWatchFullAccess
package:
  artifact: target/event-bridge-trigger-lambda.jar

functions:
  S3PutObjectHandler:
    name: ${self:provider.stage}-S3EventHandler
    handler: com.jessica.aws.lambda.S3EventHandler
    role: !GetAtt LambdaRole.Arn
    maximumRetryAttempts: 0
    description: s3 event handler
    events:
      - eventBridge:
          pattern:
            source:
              - aws.s3
            detail-type:
              - Object Created
              - Object Deleted
            detail:
              bucket:
                name:
                  - !Sub com.jessica.${self:provider.stage}-source-event-bucket
              object:
                key:
                  - prefix: "event/"

Event的数据结构

对于不同的源，event包含的内容是不一样的，对于S3来说，event结构如下：EventBridge event message structure - Amazon Simple Storage Service
- version — Currently 0 (zero) for all events.
- id — A Version 4 UUID generated for every event.
- detail-type — The type of event that's being sent. See Using EventBridge for a list of event types.
- source — Identifies the service that generated the event.
- account — The 12-digit AWS account ID of the bucket owner.
- time — The time the event occurred.
- region — Identifies the AWS Region of the bucket.
- resources — A JSON array that contains the Amazon Resource Name (ARN) of the bucket.
- detail — A JSON object that contains information about the event.
  - 根据event类型的不同，detail可能包含一下字段
  - version — Currently 0 (zero) for all events.
  - bucket — Information about the Amazon S3 bucket involved in the event.
  - object — Information about the Amazon S3 object involved in the event.
  - request-id — Request ID in S3 response.
  - requester — AWS account ID or AWS service principal of requester.
  - source-ip-address — Source IP address of S3 request. Only present for events triggered by an S3 request.
  - reason — For Object Created events, the S3 API used to create the object: PutObject, POST Object, CopyObject, or CompleteMultipartUpload. For Object Deleted events, this is set to DeleteObject when an object is deleted by an S3 API call, or Lifecycle Expiration when an object is deleted by an S3 Lifecycle expiration rule. For more information, see Expiring objects.
  - deletion-type — For Object Deleted events, when an unversioned object is deleted, or a versioned object is permanently deleted, this is set to Permanently Deleted. When a delete marker is created for a versioned object, this is set to Delete Marker Created. For more information, see Deleting object versions from a versioning-enabled bucket.
  - restore-expiry-time — For Object Restore Completed events, the time when the temporary copy of the object will be deleted from S3. For more information, see Working with archived objects.
  - source-storage-class — For Object Restore Initiated and Object Restore Completed events, the storage class of the object being restored. For more information, see Working with archived objects.
  - destination-storage-class — For Object Storage Class Changed events, the new storage class of the object. For more information, see Transitioning objects using Amazon S3 Lifecycle.
  - destination-access-tier — For Object Access Tier Changed events, the new access tier of the object
Object created event示例

{
  "version": "0",
  "id": "17793124-05d4-b198-2fde-7ededc63b103",
  "detail-type": "Object Created",
  "source": "aws.s3",
  "account": "111122223333",
  "time": "2021-11-12T00:00:00Z",
  "region": "ca-central-1",
  "resources": [
    "arn:aws:s3:::DOC-EXAMPLE-BUCKET1"
  ],
  "detail": {
    "version": "0",
    "bucket": {
      "name": "DOC-EXAMPLE-BUCKET1"
    },
    "object": {
      "key": "example-key",
      "size": 5,
      "etag": "b1946ac92492d2347c6235b4d2611184",
      "version-id": "IYV3p45BT0ac8hjHg1houSdS1a.Mro8e",
      "sequencer": "617f08299329d189"
    },
    "request-id": "N4N7GDK58NMKJ12R",
    "requester": "123456789012",
    "source-ip-address": "1.2.3.4",
    "reason": "PutObject"
  }
}

Object deleted (using DeleteObject)

{
  "version": "0",
  "id": "2ee9cc15-d022-99ea-1fb8-1b1bac4850f9",
  "detail-type": "Object Deleted",
  "source": "aws.s3",
  "account": "111122223333",
  "time": "2021-11-12T00:00:00Z",
  "region": "ca-central-1",
  "resources": [
    "arn:aws:s3:::DOC-EXAMPLE-BUCKET1"
  ],
  "detail": {
    "version": "0",
    "bucket": {
      "name": "DOC-EXAMPLE-BUCKET1"
    },
    "object": {
      "key": "example-key",
      "etag": "d41d8cd98f00b204e9800998ecf8427e",
      "version-id": "1QW9g1Z99LUNbvaaYVpW9xDlOLU.qxgF",
      "sequencer": "617f0837b476e463"
    },
    "request-id": "0BH729840619AG5K",
    "requester": "123456789012",
    "source-ip-address": "1.2.3.4",
    "reason": "DeleteObject",
    "deletion-type": "Delete Marker Created"
  }
}

Object deleted (using lifecycle expiration)

{
  "version": "0",
  "id": "ad1de317-e409-eba2-9552-30113f8d88e3",
  "detail-type": "Object Deleted",
  "source": "aws.s3",
  "account": "111122223333",
  "time": "2021-11-12T00:00:00Z",
  "region": "ca-central-1",
  "resources": [
    "arn:aws:s3:::DOC-EXAMPLE-BUCKET1"
  ],
  "detail": {
    "version": "0",
    "bucket": {
      "name": "DOC-EXAMPLE-BUCKET1"
    },
    "object": {
      "key": "example-key",
      "etag": "d41d8cd98f00b204e9800998ecf8427e",
      "version-id": "mtB0cV.jejK63XkRNceanNMC.qXPWLeK",
      "sequencer": "617b398000000000"
    },
    "request-id": "20EB74C14654DC47",
    "requester": "s3.amazonaws.com",
    "reason": "Lifecycle Expiration",
    "deletion-type": "Delete Marker Created"
  }
}

Object restore completed

{
  "version": "0",
  "id": "6924de0d-13e2-6bbf-c0c1-b903b753565e",
  "detail-type": "Object Restore Completed",
  "source": "aws.s3",
  "account": "111122223333",
  "time": "2021-11-12T00:00:00Z",
  "region": "ca-central-1",
  "resources": [
    "arn:aws:s3:::DOC-EXAMPLE-BUCKET1"
  ],
  "detail": {
    "version": "0",
    "bucket": {
      "name": "DOC-EXAMPLE-BUCKET1"
    },
    "object": {
      "key": "example-key",
      "size": 5,
      "etag": "b1946ac92492d2347c6235b4d2611184",
      "version-id": "KKsjUC1.6gIjqtvhfg5AdMI0eCePIiT3"
    },
    "request-id": "189F19CB7FB1B6A4",
    "requester": "s3.amazonaws.com",
    "restore-expiry-time": "2021-11-13T00:00:00Z",
    "source-storage-class": "GLACIER"
  }
}

Lambda如何对event进行接收和处理

需要根据需要接受的event的结构定义相应的数据结构，然后通过序列化和反序列化来得到event object
S3EventBridgeObject

package com.jessica.aws.lambda;

import java.util.ArrayList;

import com.alibaba.fastjson2.annotation.JSONField;
import com.fasterxml.jackson.annotation.JsonFilter;
import com.fasterxml.jackson.annotation.JsonIgnoreProperties;

import lombok.AllArgsConstructor;
import lombok.Data;
import lombok.NoArgsConstructor;

@Data
@AllArgsConstructor
@NoArgsConstructor
@JsonIgnoreProperties(ignoreUnknown = true)
public class S3EventBridgeObject {
    private String id;
    private String version;
    @JsonFilter(value = "detail-type")
    private String detailType;
    private String source;
    private String account;
    private String time;
    private String region;
    private ArrayList<String> resources;
    private S3EventDetail detail;

    @Data
    @AllArgsConstructor
    @NoArgsConstructor
    @JsonIgnoreProperties(ignoreUnknown = true)
    public class S3EventDetail {
        private String version;
        private S3EventBucketName bucket;
        private S3EventBucketObject object;
        @JSONField(name = "request-id")
        private String requestId;
        private String requester;
        @JSONField(name = "source-ip-address")
        private String sourceIpAddress;
        private String reason;
        @JSONField(name = "deletion-type")
        private String deletionType;
    }

    @Data
    @AllArgsConstructor
    @NoArgsConstructor
    @JsonIgnoreProperties(ignoreUnknown = true)
    public class S3EventBucketName {
        private String name;
    }

    @Data
    @AllArgsConstructor
    @NoArgsConstructor
    @JsonIgnoreProperties(ignoreUnknown = true)
    public class S3EventBucketObject {
        private String key;
        private int size;
        private String etag;
        @JSONField(name = "version-id")
        private String versionId;
        private String sequencer;
    }

}

LambdaHandler

package com.jessica.aws.lambda;

import com.alibaba.fastjson.JSON;
import com.amazonaws.services.lambda.runtime.Context;
import com.amazonaws.services.lambda.runtime.RequestHandler;

import lombok.extern.slf4j.Slf4j;

@Slf4j
public class S3EventHandler implements RequestHandler<Object, Boolean> {
    @Override
    public Boolean handleRequest(Object s3Event, Context context) {
        log.info("Received event data: " + s3Event);
        String jsonString = JSON.toJSONString(s3Event);
        S3EventBridgeObject event = JSON.parseObject(jsonString, S3EventBridgeObject.class);
        log.info(String.format("S3event: event type: %s, bucket name:%s, file key:%s", event.getDetail().getBucket().getName(), event.getDetailType(),
                event.getDetail().getObject().getKey()));
        return true;
    }
}