Snowflake Merge activity

Introduction

A Snowflake Merge activity, using its Snowflake connection, inserts or updates a CSV file of table data into Snowflake and is intended to be used as a target to consume data in an operation. The Snowflake Merge activity does not support deleting an existing record.

Create a Snowflake Merge activity

An instance of a Snowflake Merge activity is created from a Snowflake connection using its Merge activity type.

To create an instance of an activity, drag the activity type to the design canvas or copy the activity type and paste it on the design canvas. For details, see Creating an activity instance in Component reuse.

An existing Snowflake Merge activity can be edited from these locations:

• The design canvas (see Component actions menu in Design canvas).
• The project pane's Components tab (see Component actions menu in Project pane Components tab).

Configure a Snowflake Merge activity

Follow these steps to configure a Snowflake Merge activity:

• Step 1: Enter a name and select a table
  Provide a name for the activity and select a table.

• Step 2: Select merge keys
  Specify the columns to be used for merge keys.

• Step 3: Select an approach
  Different approaches are supported for merging data to Snowflake. Choose from either Direct Mapping or Stage File. When using the Stage File approach, you can select either the Use Existing Stage or Use Temporary Stage options.

• Step 4: Review the data schemas
  The request and response schemas generated from the endpoint are displayed. The schemas displayed depend on the Approach specified in the previous step.

Step 1: Enter a name and select a table

In this step, provide a name for the activity and select a table. Each user interface element of this step is described below.

• Name: Enter a name to identify the activity. The name must be unique for each Snowflake Merge activity and must not contain forward slashes / or colons :.

• Select a Table: This section displays tables available in the Snowflake endpoint. When reopening an existing activity configuration, only the selected table is displayed instead of reloading the entire table list.

  • Selected Table: After a table is selected, it is listed here.

  • Search: Enter any column's value into the search box to filter the list of tables. The search is not case-sensitive. If tables are already displayed within the table, the table results are filtered in real time with each keystroke. To reload tables from the endpoint when searching, enter search criteria and then refresh, as described below.

  • Refresh: Click the refresh icon or the word Refresh to reload tables from the Snowflake endpoint. This may be useful if tables have been added to Snowflake. This action refreshes all metadata used to build the table of tables displayed in the configuration.

  • Selecting a Table: Within the table, click anywhere on a row to select a table. Only one table can be selected. The information available for each table is fetched from the Snowflake endpoint:

    • Name: The name of a table.

    • Type: The type, which is a table.

  Tip

  If the table does not populate with available tables, the Snowflake connection may not be successful. Ensure you are connected by reopening the connection and retesting the credentials.

• Save & Exit: If enabled, click to save the configuration for this step and close the activity configuration.

• Next: Click to temporarily store the configuration for this step and continue to the next step. The configuration will not be saved until you click the Finished button on the last step.

• Discard Changes: After making changes, click to close the configuration without saving changes made to any step. A message asks you to confirm that you want to discard changes.

Step 2: Select merge keys

In this step, specify the columns to be used for merge keys. Each user interface element of this step is described below.

• Merge Keys: Select the columns to be used as update keys for the merge.

• Back: Click to temporarily store the configuration for this step and return to the previous step.

• Next: Click to temporarily store the configuration for this step and continue to the next step. The configuration will not be saved until you click the Finished button on the last step.

• Discard Changes: After making changes, click to close the configuration without saving changes made to any step. A message asks you to confirm that you want to discard changes.

Step 3: Select an approach

Different approaches are supported for merging data to Snowflake. Choose from either Direct Mapping or Stage File. When using the Stage File approach, you select either the Use Existing Stage or Use Temporary Stage options.

• Direct mapping approach
• Stage file approach (use existing stage)
• Stage file approach (use temporary stage)

Direct Mapping approach

• Approach: Use the dropdown to select Direct Mapping.

• Temporary Stage Name: Enter a name to be used for a temporary stage at Snowflake.

• Back: Click to temporarily store the configuration for this step and return to the previous step.

• Next: Click to temporarily store the configuration for this step and continue to the next step. The configuration will not be saved until you click the Finished button on the last step.

• Discard Changes: After making changes, click to close the configuration without saving changes made to any step. A message asks you to confirm that you want to discard changes.

Stage File approach (Use Existing Stage)

• Approach: Use the dropdown to select Stage File.

• Stage Type: Use the dropdown to select Use Existing Stage.

• Select a Stage: This section displays stages available in the Snowflake endpoint. When reopening an existing activity configuration, only the selected stage is displayed instead of reloading the entire stage list.

  • Selected Stage: After a stage is selected, it is listed here.

  • Search: Enter any column's value into the search box to filter the list of stages. The search is not case-sensitive. If tables are already displayed within the table, the table results are filtered in real time with each keystroke. To reload tables from the endpoint when searching, enter search criteria and then refresh, as described below.

  • Refresh: Click the refresh icon or the word Refresh to reload tables from the Snowflake endpoint. This may be useful if you have recently added tables to Snowflake. This action refreshes all metadata used to build the table of tables displayed in the configuration.

  • Selecting a Stage: Within the table, click anywhere on a row to select a stage. Only one stage can be selected. The information available for each stage is fetched from the Snowflake endpoint:

    • Name: The stage name from Snowflake.

    • Type: The stage type from Snowflake. For information on the differences between internal and external stages, see the Snowflake documentation on creating stages.

  Tip

  If the table does not populate with available stages, the Snowflake connection may not be successful. Ensure you are connected by reopening the connection and retesting the credentials.

• Back: Click to temporarily store the configuration for this step and return to the previous step.

• Next: Click to temporarily store the configuration for this step and continue to the next step. The configuration will not be saved until you click the Finished button on the last step.

• Discard Changes: After making changes, click to close the configuration without saving changes made to any step. A message asks you to confirm that you want to discard changes.

Stage File approach (Use Temporary Stage)

• Approach: Use the dropdown to select Stage File.

• Stage Type: Use the dropdown to select Use Temporary Stage.

• Temporary Stage Name: Enter a name to be used for a temporary stage at Snowflake.

• Back: Click to temporarily store the configuration for this step and return to the previous step.

• Next: Click to temporarily store the configuration for this step and continue to the next step. The configuration will not be saved until you click the Finished button on the last step.

• Discard Changes: After making changes, click to close the configuration without saving changes made to any step. A message asks you to confirm that you want to discard changes.

Step 4: Review the data schemas

The request and response schemas generated from the endpoint are displayed. The schemas displayed depend on the Approach specified in the previous step.

These subsections describe the request and response structures for each approach:

• Direct mapping approach
• Stage file approach

These actions are available with each approach:

• Data Schemas: These data schemas are inherited by adjacent transformations and are displayed again during transformation mapping.

  Note

  Data supplied in a transformation takes precedence over the activity configuration.

  The Snowflake connector uses the Snowflake JDBC Driver and the Snowflake SQL commands. Refer to the API documentation for information on the schema nodes and fields.

• Refresh: Click the refresh icon or the word Refresh to regenerate schemas from the endpoint. This action also regenerates the schema in other locations throughout the project where the same schema is referenced, such as in an adjacent transformation.

• Back: Click to temporarily store the configuration for this step and return to the previous step.

• Finished: Click to save the configuration for all steps and close the activity configuration.

• Discard Changes: After making changes, click to close the configuration without saving changes made to any step. A message asks you to confirm that you want to discard changes.

Direct Mapping approach

These are the schema fields visible in the example screenshot above:

• Request

  Request Schema Field/Node	Notes
  updateRequest	Node of the merge request.
  table	Node showing the table name NAME.
  item	Node of the item.
  column_A	Value for first column name, COLOR.
  column_B	Value for second column name, FLOWER.
  . . .	Values for succeeding table columns.

• Response

  Response Schema Field/Node	Notes
  mergeResponse	Node of the merge response.
  noOfRowsUpdated	Count of the rows updated.
  noOfRowsInserted	Count of the rows inserted.
  errors	Node of the errors.
  item	Node of the error items.
  errorMessage	Error message for the error item.

Stage File approach

These are the schema fields visible in the example screenshot above:

• Request

  Request Schema Field/Node	Notes
  mergeRequest	Node of the merge request.
  stageName	Path or prefix under which the data will be uploaded on the Snowflake stage.
  fileContent	Data file contents, in CSV format, that is to be staged for updating in the Snowflake table.
  destPrefix	Destination file prefix to be used on the Snowflake stage.
  destFileName	Destination file name to be used on the Snowflake stage.

  Important

  If Use Existing Stage is selected and the Type of the selected stage is external, stageName and fileContent are excluded from the request schema.

  For information on the differences between internal and external stages, see the Snowflake documentation on creating stages.

• Response

  Response Schema Field/Node	Notes
  mergeResponse	Node of the merge response.
  noOfRowsUpdated	Count of the rows updated.
  noOfRowsInserted	Count of the rows inserted.
  errors	Node of the errors.
  item	Node of the error items.
  errorMessage	Error message for the error item.

Next steps

After configuring a Snowflake Merge activity, complete the configuration of the operation by adding and configuring other activities, transformations, or scripts as operation steps. You can also configure the operation settings, which include the ability to chain operations together that are in the same or different workflows.

Menu actions for an activity are accessible from the project pane and the design canvas. For details, see Activity actions menu in Connector basics.

Snowflake Merge activities can be used as a target with these operation patterns:

• Transformation pattern
• Two-transformation pattern (as the first or second target)

To use the activity with scripting functions, write the data to a temporary location and then use that temporary location in the scripting function.

When ready, deploy and run the operation and validate behavior by checking the operation logs.