Jitterbit Harmony supports these methods for processing a transformation:
A streaming transformation is the preferred method to use when the amount of memory used by a transformation needs to be limited. In cases where you are unable to use a streaming transformation, chunking may be applicable. From the perspective of a transformation, a SOAP web service response/request corresponds to an XML source/target when considering limitations on processing.
This page also covers how Harmony handles the processing of files that have already been processed.
- Transformations: Using a flat source schema as input may return blank mapped fields
- Summary: In certain circumstances when using a flat source schema, target fields are not mapped correctly, which causes them to be blank.
- Additional Info: This does not occur with mirrored schemas or JSON schemas.
- Workaround: Add a script at the beginning of the operation that disables streaming transformations by setting
$jitterbit.transformation.auto_streaming = false;.
A streaming transformation loads one record at a time into memory, performs the transformation of the record, and writes the target to disk. This minimizes the amount of memory that is used during the transformation to what is required to transform that one record.
Streaming is automatically applied to transformations where the source and target are both flat structures (for example, a single database table or a single CSV file) and these requirements are fulfilled:
- Streaming has not been explicitly disabled by setting
jitterbit.conffile for a Private Agent.
Streaming has not been explicitly disabled by setting the Jitterbit variable
These dictionary and array functions are not present:
- The XML function
GetXmlStringis not present.
There are no multiple mappings in the target.
The transformation does not have a condition defined in the target.
No other action is needed; streaming will automatically be used by the transformation when it is processed.
EXAMPLES: Transformations that automatically use streaming include these data structures:
- CSV to CSV
- Single table to single table
- Single table to CSV
- CSV to single table
In a situation where a streaming transformation is neither applicable or possible, you may be able to use chunking to make the transformation require less memory. For very large XML sources and targets, chunking may be the only option. (If memory use is not an issue, a streaming transformation is always the preferred choice.) Chunking is configured at the operation level from the operation options, so you can use the same transformation in both chunked and non-chunked mode. Use as large a chunk size as possible, making sure that the data for one chunk fits into available memory. For instructions and best practices on using chunking, see Operation Options.
Chunking is configured at the operation level from the operation options, so you can use the same transformation in both chunked and non-chunked mode.
Use as large a chunk size as possible, making sure that the data for one chunk fits into available memory.
For instructions and best practices on using chunking, see Operation Options.
Repeating File Transfers
Harmony ensures that text files are transferred only a single time. When an operation starts, but before it moves any file, Harmony will check the transfer history list of source files to see if the same source information has already been used. Three criteria are used to ensure unique file transfers:
- Modification date
- Operation ID
If all of these criteria for any given source text file are the same, the system assumes the file has been processed and skips transferring this particular file.
Last updated: Sep 21, 2020
- No labels