Filter Records Using Conditions¶
A common use case for using conditions in a transformation is as a filter to determine if a record (or part of a record) should be processed or skipped.
When to Use Conditions¶
Conditions can be thought of like the WHERE clause in a SQL statement.
Conditions are typically built using scripting. The output of a condition is either
Since transformations execute from the top of the target transformation down, frequently the condition of the top node is used for actions not specific to any single data node, nor related to the determination of processing or skipping the record.
Such general-purposes tasks include these:
- Setting variables
- Performing lookups
- Updating record counts
- Logging values
If the condition node is used in this way, take care to ensure that all records are processed, such as simply putting
true as the last line in the script.
Conditions can be set not only at the topmost node, but also at subnodes, meaning that just that section would be skipped or processed.