Skip to end of metadata
Go to start of metadata

Introduction

Jitterbit Harmony supports these methods for processing a transformation:

A streaming transformation is the preferred method to use when the amount of memory used by a transformation needs to be limited. In cases where you are unable to use a streaming transformation, chunking may be applicable. From the perspective of a transformation, a SOAP web service response/request corresponds to an XML source/target when considering limitations on processing.

This page also covers how Harmony handles the processing of files that have already been processed.

Known Issue

  • Transformations: Using a flat source schema as input may return blank mapped fields
    • Summary: In certain circumstances when using a flat source schema, target fields are not mapped correctly, which causes them to be blank.
    • Additional Info: This does not occur with mirrored schemas or JSON schemas.
    • Workaround: Add a script at the beginning of the operation that disables streaming transformations by setting $jitterbit.transformation.auto_streaming = false;.

Streaming Transformation

streaming transformation loads one record at a time into memory, performs the transformation of the record, and writes the target to disk. This minimizes the amount of memory that is used during the transformation to what is required to transform that one record.

Streaming is automatically applied to transformations where the source and target are both flat structures (for example, a single database table or a single CSV file) and these requirements are fulfilled:

No other action is needed; streaming will automatically be used by the transformation when it is processed.

EXAMPLES: Transformations that automatically use streaming include these data structures:

  • CSV to CSV
  • Single table to single table
  • Single table to CSV
  • CSV to single table

Chunking

In a situation where a streaming transformation is neither applicable or possible, you may be able to use chunking to make the transformation require less memory. For very large XML sources and targets, chunking may be the only option. (If memory use is not an issue, a streaming transformation is always the preferred choice.)

Chunking is configured at the operation level from the operation options, so you can use the same transformation in both chunked and non-chunked mode.

Operation options can be accessed from the operation settings that are accessible from the project pane in both the Workflows and Components tabs, and from the design canvas:

  • Project Pane: In the Workflows or Components tab of the project pane, hover over an operation name and click the actions menu icon  to open the actions menu. From the menu, select Settings to open the operation settings. Then select the Options tab.
  • Design Canvas: In the top right of an operation, hover over the actions menu icon  to open the actions menu. From the menu, select Settings to open the operation settings. Then select the Options tab.

Use as large a chunk size as possible, making sure that the data for one chunk fits into available memory.

For instructions and best practices on using chunking, see Operation Options.

Repeating File Transfers

Harmony ensures that text files are transferred only a single time. When an operation starts, but before it moves any file, Harmony will check the transfer history list of source files to see if the same source information has already been used. Three criteria are used to ensure unique file transfers:

  • Filename
  • Modification date
  • Operation ID

If all of these criteria for any given source text file are the same, the system assumes the file has been processed and skips transferring this particular file.

On This Page

Last updated:  Sep 21, 2020

  • No labels