Schema Processing¶

Introduction¶

When using a user-defined schema, in certain circumstances data may be converted during processing. This page describes when and how data is converted.

Names of Fields and Nodes¶

Names of fields and nodes are converted to valid namespaces following the XML standard. For reference, see NCNameChar for the allowed characters as defined by the W3C.

Names with Spaces¶

Spaces included as part of a field or node name are replaced with an underscore (_).

Special Characters in Schemas¶

Special characters in user-defined schemas are processed differently depending on the schema type. These conversions are limited to the schema structures only and do not affect the actual data.

The handling of special characters depends on the schema type, as described in the table below. Special characters include, but are not limited to, these non-alphanumeric characters:

. _ - $ # @ ?

Support for special characters by Cloud Studio is limited by support for special characters by the schema type itself. For example, including a $, #, @, or ? in an XML schema field or node name is not supported because those special characters are not supported by XML.

When mirroring a schema provided by a database, any special characters in a schema field or node name are replaced by underscores as required to create a valid XML schema.

Handling of Special Characters in Schemas	Custom Flat or Hierarchical Schema	Uploaded CSV Schema	Uploaded JSON Schema	Uploaded XML Schema
Special characters on the source or target side of a transformation are retained as-is
Special characters on the source or target side of a transformation are replaced by a question mark (`?`)
Special characters on the source or target side of a transformation are replaced by an underscore (`_`)
Special characters in a script on a target field in a transformation are replaced by a question mark (`?`)

CSV Files with Headers¶

If providing a CSV file with a header row as a sample schema file, these rules are applied to generate column names:

Special characters are replaced with a question mark (?).
Spaces are replaced with an underscore (_).
If a column name is blank, it is replaced with f1, f2, f3, and so on.
If a column name starts with a number, it is prefixed with an underscore (_).
If a column name is repeated, it is appended with 2, 3, 4, and so on.

Data Types¶

These sections describe Jitterbit's support of certain data types. For all data types supported in Jitterbit Scripts, see Data Types in Jitterbit Script.

Unlimited-Precision Data Types¶

Unlimited-precision data types, such as XML decimal, are converted to double data types and therefore have a precision limit imposed on them that could, as a result, truncate data.

The precision limit is within the range of the minimum and maximum values of a signed long, which is –2,147,483,648 to 2,147,483,647. If outside of this range, consider using a string data type instead to avoid truncating data.

Non-Primitive Data Types¶

The primitive data types boolean, date, double, float, integer, long, and string, and the non-primitive data type decimal are supported in all schema types. When creating or editing a custom flat or hierarchical schema or editing any schema uploaded in a transformation, these data types are available to choose in the Type dropdown. For new custom schemas, other non-primitive data types such as datetime are not supported.

However, other non-primitive data types are supported for schemas that are automatically generated from a connector-generated schema or were mirrored from such a schema. After they are generated, such schemas can also be manually edited in the custom schema editor. If the schema contains any non-primitive data types, these are also listed in the Type dropdown when editing such a schema.

Null Values¶

Fields that have a null value are included in the resulting data schema despite having no data. As they also have no defined data type, these fields are treated as having a string data type.

XML Schema Namespaces¶

Namespaces in XML schemas are supported. If more than one namespace is used in an XML schema, the Harmony Agent converts the XML schema to multiple XSD files during processing.

Troubleshooting¶

Due to changes made in Harmony versions 10.25 and 10.27, you may see different behavior in projects created prior to these releases. These changes affect XML schemas that have been mirrored and are most likely to impact mappings that use XML functions that involve namespaces, such as the SelectNodes function. In this case, mappings that were previously valid may now be invalid with an error related to the syntax of the XML function.

Compare the differences between this example mirrored XML schema prior to 10.25 with one created in 10.25 or later:

xml schema annotated

Example XML Schema Prior to 10.25: In projects created prior to 10.25, mirrored XML schemas use the default namespace prefix for XML documents: xsi. As shown highlighted in red above, xmlns:xsi declares the namespace and unmapped fields are displayed in the schema with the attribute xsi:nil="true".
Example XML Schema in 10.25 and Later: In projects created in 10.25 and later and highlighted in green above, mirrored XML schemas use the namespace prefix ns to declare a qualified namespace. Unmapped fields are not displayed in the schema.

In versions 10.25 and 10.26, if you imported a project with a mirrored XML schema that was created prior to 10.25, the schema was reprocessed and changed to use the qualified namespace prefix.

As of version 10.27, imported projects whose mirrored XML schemas were created prior to 10.25 retain the default XML schema namespace prefix so that the schema is identical to that when it was created. This change means that any pre-10.25 projects that are imported into the current version should function as originally designed.

To force a refresh of a pre-10.25 XML schema to use the updated namespace prefix, you can regenerate the schema by refreshing it or reconfiguring the activity that provides the schema.

XML Schema Self-Closing Tags¶

Self-closing tags on elements in XML schemas are supported with some manipulation of the sample XML used to generate the schema. To send the XML element in the payload without a mapped value, you can use a Jitterbit function and Jitterbit variable as described below.

Manipulate the Sample XML¶

Manipulation is required when an XML sample file uses shorthand notation for an opening and closing tag in an element such as this:

<tag/>

If such shorthand is used in an XML schema directly, the node will not be able to be mapped to when used as the target schema of a transformation.

To resolve, manipulate the sample XML being used to generate the schema to expand the opening and closing tags surrounding the element, and provide a sample value so that a data type is assigned to the element:

<tag>example</tag>

The element will then show up as a field in the XML schema and, when used as the target schema in a transformation, you can map to that field.

Map to the XML Field¶

If you do not have a source object or variable to map to the target field, you can place the Jitterbit function Null in the transformation script to use as the mapped value:

<trans>
Null()
</trans>

Upstream of the transformation using the XML schema, set one of these Jitterbit variables (depending on which is appropriate to your use case) to control what is sent:

Empty XML: To send an empty XML node in the payload, use jitterbit.target.xml.include_empty_xml:
```
$jitterbit.target.xml.include_empty_xml=true;
```
Null XML: To send a nil value in the XML payload, use jitterbit.target.xml.include_null_xml:
```
$jitterbit.target.xml.include_null_xml=true;
```
Exclude Empty XML: To exclude an empty XML node with a boolean data type, use jitterbit.target.xml.exclude_empty_data:
```
$jitterbit.target.xml.exclude_empty_data=true;
```