Skip to Content

Filter Records Using Conditions

Introduction

A common use case for using conditions in a transformation is as a filter to determine if a record (or part of a record) should be processed or skipped.

When to Use Conditions

Conditions can be thought of like the WHERE clause in a SQL statement.

Conditions are typically built using scripting. The output of a condition is either true or false.

Since transformations execute from the top of the target transformation down, frequently the condition of the top node is used for actions not specific to any single data node, nor related to the determination of processing or skipping the record.

Such general-purposes tasks include these:

  • Setting variables
  • Performing lookups
  • Updating record counts
  • Logging values

If the condition node is used in this way, take care to ensure that all records are processed, such as simply putting true as the last line in the script.

Conditions can be set not only at the topmost node, but also at subnodes, meaning that just that section would be skipped or processed.