Skip to end of metadata
Go to start of metadata

Introduction

The Jitterbit AWS S3 Create plugin is used to upload a file to Amazon AWS S3 as a target within an operation in Design Studio.

This page details the steps to use the AWS S3 Create plugin:

  1. Associate the Plugin with an Agent Group
  2. Assign the Plugin to a Target
  3. Set Global Variables for the Plugin
  4. Complete Operation Setup
NOTE: This page uses Design Studio as an example; you may use this plugin in Cloud Studio using similar steps.

Encryption

The AWS S3 Create plugin passes your provided AWS S3 access key ID and secret access key (described below in Set Global Variables for the Plugin) to Amazon. Security is handled by Amazon as described in Amazon's documentation on AWS Security Credentials.

Version 1.0.1.0 of the Jitterbit AWS S3 Create plugin supports S3 buckets that use one of these encryption options:

  • None: No encryption
  • AES-256: Server-Side Encryption with Amazon S3-Managed Keys (SSE-S3)
  • AWS-KMS: Server-Side Encryption with AWS KMS-Managed Keys (SSE-KMS)

Version 1.0.0.0 of the Jitterbit AWS S3 Create plugin supports using no encryption or AES-256. KMS encryption is not supported with this version.

Examples

The AWS S3 Create plugin can be used whenever you need to upload files to AWS. For reference, a basic Jitterpak AWS S3 Basic Example (JPK) is included with an example setup.

In one common use case, this plugin can be used to move large amounts of data into a Snowflake data warehouse when used with Snowpipe. To use this recommended setup, follow Snowflake's Snowpipe documentation, then configure the AWS S3 Create plugin as described on this page.

NOTE: Using Snowpipe and the AWS S3 Create plugin is recommended for increased performance in place of using Snowflake's JDBC driver, although it is also possible to install the Snowflake JDBC driver and then use Snowflake as a database target in Design Studio.

Additional Functionality

In addition, these options offer additional functionality beyond the AWS S3 Create plugin:

Associate the Plugin with an Agent Group

If using a Cloud Agent Group, you can skip this step, as the AWS S3 Create plugin is already associated with Jitterbit's Cloud Agent Groups by default.

If using a Private Agent Group, you must associate this Jitterbit-provided plugin with your Private Agent Group. This will make the plugin available on all Private Agents within the Private Agent Group. To associate the plugin:

  1. Log in to the Harmony Portal and go to the Management Console. Then use the menu in the top left to navigate to Customizations > Plug-ins.
  2. In the table, locate the row "Jitterbit AWS S3 Create." On the far right, use the Action dropdown to select Associate Agent Group.
  3. In the popup, select your Private Agent Group and click Save.
TIP: Detailed documentation on associating plugins is available in Customizations > Plug-ins.

Assign the Plugin to a Target

In your Design Studio project, you must assign the plugin to a target within an operation. Within your project:

  1. Create an operation. The example Jitterpak uses a Archive type, although any operation using a target, including those with transformations, may be used.
  2. Create a dummy target that the plugin will be assigned to. This target is used as a placeholder and will not actually be written to. We recommend using a file-type source such as temporary storagelocal file, or FTP Site and providing a Filename(s) value of [file].[ext]. This will ensure that the file has the same name and extension as that coming from the source.

    NOTE: A global variable target can also be used if you are writing a single file to S3.
  3. Assign the plugin to the target. To do this, right-click on the source and select Plugin. In the popup, under Available plugins, select "Jitterbit AWS S3 Get" and click Assign. Then click OK.
TIP: Detailed documentation on assigning plugins is available in Applying Pipeline Plug-ins.

Set Global Variables for the Plugin

In order for a plugin assigned to a target to be functional, any global variables required by the plugin must be set. These variables are set in a script, which should be placed before the target in the operation so that the variables can be initialized. Scripts are often set before the source in an operation, which is also before the target.

To add a script before the source and target in the operation:

  1. Right-click on the source in the operation and select Insert Before This > Script.
  2. Create a new Jitterbit Script that sets variables used by the plugin to authenticate with your S3 bucket and upload any files you specify. A script template for required variables is provided below, followed by documentation on all possible variables that may be included within the script.

    AWS S3 Get Script
    <trans>
    $AWS.S3.BUCKET_NAME="<YOUR_BUCKET_NAME>";
    $AWS.S3.ACCESS_KEY_ID="<YOUR_ACCESS_KEY>";
    $AWS.S3.SECRET_ACCESS_KEY="<YOUR_SECRET_KEY>";
    </trans>
NameTypeRequiredDescription
AWS.S3.ENDPOINTString

Optional

The URL of the Amazon AWS S3 endpoint. Defaults to "s3.amazonaws.com" if not set. Valid endpoint names by region are provided in Amazon's documentation AWS Regions and Endpoints.
AWS.S3.BUCKET_NAMEStringRequiredThe name of the S3 target bucket. To write to an existing folder inside the bucket, add the folder name after a forward slash (e.g. "my_bucket/folder"). The folder must already exist.
AWS.S3.ACCESS_KEY_IDStringRequiredThe S3 access key (e.g. "ABCDEFGHI1JK2LM3NOPQ").
AWS.S3.SECRET_ACCESS_KEYStringRequiredThe S3 secret access key (e.g. "1aBCd2e/F34gH56IJ7Kl/Mno8pqRsTUvw9x0yzzz").
AWS.S3.REGIONStringOptionalThe S3 region. Defaults to "DEFAULT_REGION" if not set (not recommended). Current region names are provided in Amazon's documentation AWS Regions and Endpoints.
AWS.S3.OBJECT_KEYStringOptionalThe S3 object key to be written. This is the name of the specific file you are uploading, such as "myfile.txt". If this variable is not included in the script, only one file in the bucket and folder (if specified) will be returned. Specifying multiple files or using wildcards is not supported.

Complete Operation Setup

The plugin portion of the operation setup is complete after you have (1) the script defining the variables to be used with the plugin and (2) the plugin assigned to a target in the operation.

To complete your operation setup, define a source from which the data is coming from, or add both a source and a transformation if you want to transform your data prior to reaching the S3 bucket. If writing a single file to S3, the operation setup is complete.

If you want to write multiple files to S3, you can manually create an additional script to loop through the files in your S3 bucket. Make sure that you have not set any values in the AWS.S3.OBJECT_KEY variable.

Since only one file is returned with the plugin, you can run each file separately through the operation. For example, using a combination of the FileList()While(), and RunOperation() and a newly defined global variable SingleFile, this script picks up each file and then runs the operation with the plugin:

File Loop Script
<trans>
Files = FileList("<TAG>Sources/All Files</TAG>");
i = 0;
while(i< length(Files),
$SingleFile = Files[i];
RunOperation("<TAG>Operations/Read from source and write to S3</TAG>");

i++;
)

</trans>

For use with the above script, you will want to reference the new global variable [SingleFile] in the Get Files of your file-type source. This example may be adjusted for use with other configurations.

For reference, the Jitterpak AWS S3 Basic Example (JPK) provides a complete operation using the AWS S3 Create plugin.

On This Page

Plugin

Jitterbit Harmony

Last updated:  Jan 31, 2020