Skip to content

S3

This extension allows to publish events to Amazon AWS S3 buckets.

Features

  • S3 (Sink)

    S3 sink publishes events as Amazon AWS S3 buckets.

S3 Sink

S3 sink publishes events as Amazon AWS S3 buckets.

Syntax:

@sink(type="s3", 
    credential.provider.class="<STRING>", 
    aws.access.key="<STRING>", 
    aws.secret.key="<STRING>", 
    bucket.name="<STRING>", 
    aws.region="<STRING>", 
    versioning.enabled="<BOOL>", o
    bject.path="<STRING>", 
    storage.class="<STRING>", 
    content.type="<STRING>", 
    bucket.acl="<STRING>", 
    node.id="<STRING>", 
    @map(...)))

QUERY PARAMETERS:

Name Description Default Value Possible Data Types Optional Dynamic
credential.provider.class AWS credential provider class to be used. If blank along with the username and the password, default credential provider will be used. EMPTY_STRING STRING YES No

| aws.access.key | AWS access key. This cannot be used along with the credential.provider.class| EMPTY_STRING | STRING | YES | No |

| aws.secret.key| AWS secret key. This cannot be used along with the credential.provider.class| EMPTY_STRING | STRING | YES | No |

| bucket.name | Name of S3 bucket | | STRING | No | No |

| aws.region | The region to be used to create the bucket | EMPTY_STRING | STRING | YES | No |

| versioning.enabled | Flag to enable versioning support in the bucket.| false | BOOL | YES | No |

| object.path | Path for each S3 object| | STRING | No | YES |

| storage.class | AWS storage class| standard | STRING | YES | No |

| content.type | Content type of the event | application/octet-stream | STRING | YES | YES |

| bucket.acl | Access control list for the bucket.| EMPTY_STRING | STRING | YES | No |

| node.id | The node ID of the current publisher. This needs to be unique for each publisher instance as it may cause object overwrites while uploading the objects to same S3 bucket from different publishers. | EMPTY_STRING | STRING | YES | No |

EXAMPLE 1:

@sink(type='s3', 
    bucket.name='user-stream-bucket',
    object.path='bar/users', 
    credential.provider='software.amazon.awssdk.auth.credentials.ProfileCredentialsProvider', flush.size='3',
    @map(type='json', 
        enclosing.element='$.user', 
        @payload("""{"name": "{{name}}", "age": {{age}}}""")
        )
    ) 

define stream UserStream(name string, age int);  

This creates a S3 bucket named user-stream-bucket. Then this will collect 3 events together and create a JSON object and save that in S3.