Skip to Content

Transformation Processing

Introduction

Harmony supports these methods for processing a transformation:

A streaming transformation is the preferred method to use when the amount of memory used by a transformation needs to be limited. In cases where you are unable to use a streaming transformation, chunking may be applicable. From the perspective of a transformation, a SOAP web service response/request corresponds to an XML source/target when considering limitations on processing.

This page also covers how Harmony handles the processing of files that have already been processed.

Known Issue

This issue is known to occur with transformations:

  • Using a flat source schema as input may return blank mapped fields
    • Summary: In certain circumstances when using a flat source schema, target fields are not mapped correctly, which causes them to be blank.
    • Additional Information: This does not occur with mirrored schemas or JSON schemas.
    • Workaround: Add a script at the beginning of the operation that disables streaming transformations by setting $jitterbit.transformation.auto_streaming = false;.

Streaming Transformation

A streaming transformation loads one record at a time into memory, performs the transformation of the record, and writes the target to disk. This minimizes the amount of memory that is used during the transformation to what is required to transform that one record.

Streaming is automatically applied to transformations where the source and target are both flat structures (for example, a single database table or a single CSV file) and these requirements are fulfilled:

  • Streaming has not been explicitly disabled by setting AutoStreaming=0 in the jitterbit.conf file for a Private Agent.

  • Streaming has not been explicitly disabled by setting the Jitterbit variable jitterbit.transformation.auto_streaming to 0 or false.

  • No instance-resolving functions are used, such as FindByPos, FindValue, or Sum.

  • These dictionary and array functions are not present: GetSourceInstanceMap, GetSourceAttrNames, GetSourceElementNames, GetSourceInstanceElementMap, GetSourceInstanceArray, GetSourceInstanceElementArray.

  • The XML function GetXmlString is not present.

  • There are no multiple mappings in the target.

  • The transformation does not have a condition defined in the target.

No other action is needed; streaming is automatically be used by the transformation when it is processed.

Examples

Transformations that automatically use streaming include these data structures:

  • CSV to CSV
  • Single table to single table
  • Single table to CSV
  • CSV to single table

Chunking

In a situation where a streaming transformation is neither applicable or possible, you may be able to use chunking to make the transformation require less memory. For very large XML sources and targets, chunking may be the only option. (If memory use is not an issue, a streaming transformation is always the preferred choice.)

Chunking is configured at the operation level from the operation options, so you can use the same transformation in both chunked and non-chunked mode.

The Settings option for operations is accessible from these locations:

Once the operation settings screen is open, select the Options tab.

Use as large a chunk size as possible, making sure that the data for one chunk fits into available memory.

For instructions and best practices on using chunking, see Operation Options.

Repeating File Transfers

Harmony ensures that text files are transferred only a single time. When an operation starts, but before it moves any file, Harmony checks the transfer history list of source files to see if the same source information has already been used. Three criteria are used to ensure unique file transfers:

  • Filename
  • Modification date
  • Operation ID

If all of these criteria for any given source text file are the same, the system assumes the file has been processed and skips transferring this particular file.