Skip to Content

Split a File into Individual Records Using SourceInstanceCount

Introduction

This design pattern can be used to split a small set of source data in a multi-record JSON or XML file into multiple files, each containing a single record, using the SourceInstanceCount function.

Tip

This pattern is recommended to be used only with a small set of source records when the source data is complex (hierarchical). For flat (non-hierarchical) source data, see Split a File Into Individual Records Using SCOPE_CHUNK. This pattern is not recommended for large sets of data because the payload must be repeatedly processed for each source record.

Use Case

In this scenario, the source data is a complex JSON file that contains a small number of multiple records, and the target endpoint requires records to be processed individually.

Design Pattern and Example

This example operation chain is used to illustrate the design pattern, whose key characteristics are described below:

operation chain sourceinstancecount

  1. The operation Get File with Multiple Records retrieves the multi-record JSON file from the source and writes the contents to a variable (named Variable).

  2. On success of that operation, the operation Control executes a controller script, the parts of which are described below:

    <trans>
ordercount = CountSubString($Variable, '"assigned_location":');
$gv_record_no=1;
While($gv_record_no<=ordercount,
RunOperation("<TAG>operation:Split into Single-record Files</TAG>");
$gv_record_no++;
)
</trans>

    • The CountSubString function is used to count the number of records from the source (replacing assigned_location with any field that appears only once for each record in the source data).

    • A global variable (gv_record_no) is initialized at 1 and will be used to increment the number of records.

    • The While function is used to step through the record count and execute the next operation for each record.

  3. As the script cycles through records, the operation Split into Single-record Files is repeatedly executed. These are the steps of the operation:

    1. The contents of the multi-record JSON file contained in Variable are used as a source for the transformation.

    2. In the transformation, a condition script is defined on the item node to iterate through each record:

      If(SourceInstanceCount() == $gv_record_no, true, false)

      split into single records

    3. On each operation run, the transformation will output a single JSON record, which can be written to the desired target.