Skip to Content

Jitterbit iPaaS best practices

Introduction

This document serves as a general guide to using Jitterbit integration platform-as-a-service (iPaaS). It provides best practice guidance for common integration scenarios and recommendations using the available tools. This document is not comprehensive and does not cover all scenarios.

This document is for users of Integration Studio, the web-based version of Jitterbit's project design application. For best practices using Design Studio, Jitterbit's desktop-based project design application, see Best practices with Design Studio.

Best practices for App Builder, Jitterbit's application for creating web and mobile apps, are covered separately.

You should already be familiar with Jitterbit iPaaS and Integration Studio from the following resources:

At that point, you should know the basic concepts and terms used in Jitterbit iPaaS, and understand what we mean by projects, operations, endpoints, scripting, migration, and deployment.

See the Additional resources section at the end of this page for links to videos and other documents that expand on these best practices.

Support, Customer Success Managers, and documentation

Access to Jitterbit support is included as part of a Harmony customer license. When questions or technical issues arise, you can get expert assistance from Jitterbit support. The Jitterbit support page describes special instructions for production-down situations in order to escalate time-sensitive issues.

You can also contact your Customer Success Manager (CSM) with questions related to licensing or other topics.

This documentation site (Jitterbit Documentation) and our developer documentation site (Jitterbit Developer Portal) contain more than 3,600 unique URLs of technical material.

To help you locate relevant material, the search box is pre-filtered to limit search results to the section of documentation that is currently being accessed. You can search all documentation by clearing the search filter. To search for specific phrases, enclose them in double quotes.

Recipes and templates

The Marketplace provides more than 400 ready-made Integration Studio recipes and templates to use as the basis for integration designs:

  • An integration recipe moves data in one direction between objects across two applications or systems.

  • A process template accelerates the execution of a specific business process using numerous objects across multiple applications or systems. Process templates are designed to reduce the time to deployment by 50 to 80 percent.

Jitterbit product updates

Jitterbit product updates are released frequently (see Release schedule). Even a minor release contains new features and enhancements along with bug fixes.

Web applications accessed through the Harmony portal are updated automatically and always run the latest released version.

Cloud API gateway and cloud agent group updates are applied automatically. For the cloud agent groups, there are two sets: Production and Sandbox. The latter set is used to test compatibility with pre-releases of agent software and is not a development environment.

Locally installed applications are upgraded by downloading and running an installer:

It is advisable to stay current with releases, particularly releases that include feature upgrades.

Project design simplicity and feature preference

During the process of designing an integration project, you may encounter multiple methods of implementation that can result in the same functionality.

While there are often advantages or disadvantages to one method over another depending on the integration use case (as documented), we recommend always keeping the overarching guiding principle of simplicity in mind. Projects that are designed as simply as possible usually have greater longevity and are much easier for an outsider to get up to speed on if a change needs to be made. On the other hand, overly complex projects can be more difficult to maintain and hand off to others during a transition of responsibilities.

To design simpler projects, we recommend the following order of feature preference:

  1. Make use of no-code features whenever possible, such as visually designing operations on the design canvas, grouping components together to help organize them, adding tags and comments when deploying, and publishing operations as APIs.
  2. Leverage low-code features, such as using variables within the wizard-based component configuration screens or manipulating field-level data in a transformation mapping using a script function.
  3. On an as-needed basis, implement more complex logic using scripts written in Jitterbit Script language.
  4. Only when a Jitterbit Script equivalent is unavailable, implement scripts in JavaScript.

Project design and reusability

A typical scenario for reusing a project involves the development of a starter project with the extensive use of global variables and, especially, project variables. Configurable items — such as endpoint credentials, optional field mappings, parameterized queries, email addresses, and filenames — can be exposed as project variables. The starter project can also contain common functions such as error handling or the use of environment-wide caches. The starter project is exported and then imported into new projects to form a consistent basis for development.

Endpoint reuse

Endpoints, created by configuring a connection and associated activities using connectors, are frequently used in operations. However, a unique endpoint doesn't necessarily need to be built for each operation. Since activity configurations accept variables for paths and filenames, generic endpoints can be built one time and then dynamically configured using global and project variables.

For example, assume an HTTP connection and an associated activity are created, and the activity configuration specifies a path defined by a global variable, such as $gv_http_path. A controller script can be used to populate the $gv_http_path as required.

Another example is a Database Query activity with a condition. Its WHERE condition can be assigned to a global variable, such as $gv_database_condition.

Most endpoints have the capability to be configured using variables.

Script reuse

Stand-alone scripts that perform a specific function, such as returning a database lookup or calculating a result from a series of arguments, can be candidates for reuse, particularly if used in multiple operations.

For example, if a script uses the DBLookup function against a database table, and this function is used throughout the project, then a stand-alone script (separate from an operation) can be built. Using the ArgumentList function or simple global variables, the script can accept arguments and return a result. Since every operation chain is a different scope, the same script can safely be called from multiple simultaneous operations.

Project organization

Workflows are used as a means of project organization. A workflow typically holds related operations that process data from start to finish: Create Orders, Sync Customer Master, Process Confirmations, etc. Processes that are common across different workflows such as querying an endpoint or handling an operation error condition, can be held in their own workflow and referred to by other workflow operations.

You can also create custom groups where project components can be collected for ease of reference.

The numbers assigned to the operations that appear in the project designer are assigned automatically and are based on the display position of the operation in the project designer. Those numbers are not shown in the operation logs. If an operation numbering scheme is needed, then it can be implemented by incorporating numbering into the operation name.

Manage asynchronous operations

When using the RunOperation function in its asynchronous mode, operations execute without returning control to the calling function. Use of asynchronous operations can lead to race conditions.

For example, if Operation A updates a database table and is chained to Operation B, which reads that same table (both are synchronous), no race conditions are encountered. But if Operation A is called asynchronously followed immediately by Operation B, then B may execute before A is finished.

In addition, the number of simultaneous asynchronous calls must be managed, as the number of simultaneous operations running on an agent is capped (see the [OperationEngine] section of the private agent configuration file).

Endpoint credentials

We recommend using a system ID with administration permissions for endpoint credentials, rather than a user-level ID. User IDs typically expire or have to be disabled when a user leaves a company.

By using project variables (whose values can be hidden) for credentials management, a Harmony organization administrator does not have to enter production credentials. By setting up the appropriate user permissions, a user can apply the production credentials through the Management Console Projects page.

If using private agents, as an alternative to using the Management Console, global variables can be managed using the [PredefinedGlobalVariables] section of the private agent configuration file.

Persisting integration data

There are multiple methods for storing data in the Harmony cloud, including using project variables, cloud caching functions, or temporary storage.

Project variables

Project variables are preinitialized static variables that can be thought of as project constants. They can be edited from Integration Studio (see Project variables) or the Management Console (see Projects).

One example use of project variables is for endpoint credentials. By using project variables, different endpoint environments (which usually have different credentials) can be applied to different Harmony environments and edited through the Management Console. This enables a business process where a user with Management Console rights can change credentials without requiring access to Integration Studio and the project designer.

A second example is to use project variables to hold flags used by integration logic that can customize the behavior of the integration. If a single project is developed but used for different endpoints, then a boolean project variable (such as Send_PO_Number) could be checked by the transformation's logic for the PO Number field. If the project variable's value is false, then the UnMap function could be used to "turn off" that field and not include it in a relevant transformation.

Cloud caching functions

The cloud caching functions ReadCache and WriteCache are used to assign data spaces that are available across projects and across environments. A cached value is visible to all operations running in the same scope until it expires, regardless of how that operation was started or which agent it runs on. By caching data in Harmony, rather than relying on local or agent-specific data stores such as Temporary Storage, data can be shared between separate operations and across projects.

These are additional uses of cloud caching:

  • Data can be shared between asynchronous operations within a project.
  • Errors that are generated across different operations could be stored to a common cache. By accumulating operation results in this manner, more comprehensive alerts can be built.
  • Login tokens can be shared across operations.

Manage Temporary Storage

Temporary Storage endpoints are frequently used in operations on both cloud and private agents. These are distinct from Local Storage endpoints, which can only be used on private agents.

When using a Temporary Storage endpoint, temporary files are written to and read from the default operating system's temp directory on the agent that is performing the work:

  • In the case of a single private agent, the temporary storage directory is that private agent server's default temp directory.
  • If you are using more than one private agent, clustered in a private agent group, the temporary storage directory is the default temp directory of the specific private agent server doing the work.
  • As cloud agents are clustered in a cloud agent group, the temporary storage directory is the default temp directory of the specific cloud agent server doing the work.

In a clustered agent group (private or cloud agents), as long as the operations using the temporary storage are linked (chained) together, then all the temporary storage reads and writes will happen on the same agent server. However, if Chain A writes to its temp storage myfile and Chain B reads from its temp storage myfile, and the two chains are not themselves chained to one another, the temporary storage read action in Chain B may not be read from the same agent as Chain A.

When using temporary storage, keep these guidelines in mind:

  • When using private agents, to make your project upgrade-proof, use temporary storage in such a way that moving from a single private agent to a multiple-agent agent group does not require refactoring.

  • When using a clustered agent group (private or cloud agents), to ensure that the agent server where temporary storage is written is the same server where temporary storage will be read from, make sure that any references to the temporary storage Read and Write activities are in the same operation chain.

  • Temporary storage on private agents is deleted after 24 hours by default by the Jitterbit file cleanup service. The cleanup service frequency can be configured through the private agent configuration file under the [FileCleanup] section. However, on cloud agents, temporary files may be deleted immediately.

  • Cloud agents have a temporary storage file size limit of 50 GB per file. Temporary files larger than 50 GB are possible only when using private agents.

  • When writing to temporary storage, the default is to overwrite files. This can be changed with the Append to File checkbox in a Temporary Storage Write activity. Usually this then requires that after the source is read that the file be deleted or archived. A simple way to do this is to use the post-processing options Delete File or Rename File in a Temporary Storage Read activity.

  • Filename keywords are available that can be used when creating a filename.

  • A file in temporary storage can be read by building a script with the ReadFile function. For example: ReadFile("<TAG>activity:tempstorage/Temporary Storage/tempstorage_read/Read</TAG>"). Bear in mind that this works reliably only if there is a single private agent.

In some cases, it may be advantageous to use a Variable endpoint instead of a Temporary Storage endpoint. See Variable versus Temporary Storage in Global variable versus Temporary Storage for a comparison of these two different types and for recommendations on when each is appropriate.

Scripting

Scripts written in Jitterbit Script language or JavaScript can be used almost anywhere in operations and within transformation mappings.

When to use scripting

Operations can be organized into operation chains in two ways: (1) by linking operations using On Success and On Fail conditions using operation actions, or (2) by using a controller script.

Instead of using operation actions, a controller script uses the RunOperation function to link operations together using a script.

To capture a failed operation, the If function can be used in conjunction with RunOperation. For example: If(!RunOperation(<operation tag>),<condition>), where the condition can use GetLastError to capture the error, and can elect to stop the entire process using RaiseError, and/or run another process to accumulate error text.

A controller script can be beneficial in situations such as these:

  • To run an operation that is dependent on external factors such as project variables or data.
  • To call sub-operations from within a loop, where data is passed to the operation from a list.
  • To trace operation chain activities. For example: (WriteToOperationLog("count of records to process: " + cnt), WriteToOperationLog("Starting update operation at: " + Now()), WriteToOperationLog("Database query: " + sql), etc.)

Other areas where scripting is frequently used are within the mapped fields in transformations and in other stand-alone scripts. If the same script is being used within more than one transformation, consider setting up that script as a stand-alone script and calling it from within each transformation using RunScript.

Naming convention for variables

Jitterbit iPaaS has four types of variables:

As the scope of a local variable is limited to a single script, a naming convention for them can be very simple, such as all-lowercase letters or an initial word, such as return or myVariable. Periods are not allowed in local variables.

Global variables, as their scope is larger (a global variable is available to be referenced in the same or downstream operations and scripts within an operation chain), should use a consistent naming convention to avoid confusion. For example, using multiple components for a variable name, separated by periods, you could follow a pattern such as this:

type.category.subcategory
Component Description
type A short abbreviation identifying the variable type, such as pv (project variable), gv (global variable), io (endpoint source/target name), dict (dictionary), etc.
category A logical category for the variable, such as sfdc, shipearly, confirm, order, etc.
subcategory A logical subcategory for the variable, such as purchase_orders, categories, ids, etc.

Combining these components, these are possible variable names:

  • $pv_shopify_base_url
  • $dict_staples_po_line_items
  • $io_request
  • $gv_sfdc_workorder_id

Since variables are sorted alphabetically in various places throughout the UI, organizing them hierarchically will assist with managing and using variables.

Whatever convention you choose to use, we recommend codifying and documenting it so that all team members can consistently use it in all projects.

Note

If you plan to use Jitterbit Script global variables in a JavaScript script, it is important to use underscores instead of periods:

  • $example_arr_names
  • $example_sfdc_success_message

Environments

Jitterbit enables software development lifecycle methodologies through the use of environments. You can set up both production and non-production environments.

For example, assume that a Development and a Production environment are set up in the Management Console and both are associated with the same agent group. Assume that a project is first developed in the Development environment.

Integration Studio has a migration feature that will copy that project to the Production environment, after which the endpoint credentials are changed to the Production endpoint credentials using project variables. Other source and target endpoints are also changed. After the initial migration, any further migrations of the same project from Development to Production exclude migrating project variable values unless they are new project variables.

Testing

Jitterbit enables rapid integration development and unit testing by making visible the actual integration data during design time. The obvious advantage is to enable an iterative development process by showing the data before and after field transformations, rather than building the entire operation, running it, and inspecting the output. Data is made visible by using the preview feature in a transformation.

After the sample source data is imported or generated, the transformation will show the output of any mappings and embedded scripts.

Troubleshooting

A key concept for a healthy integration architecture is to recognize that there will be questions raised by the business concerning the accuracy of the integration work, particularly when discrepancies appear in the endpoint data. The integration may or may not be at fault. It is incumbent on the integration project to provide a high degree of transparency in order to help resolve questions about data accuracy.

For example, if data in a target endpoint appears to be incorrect, then typically integration support is called upon to provide details concerning any integration actions, such as times, sources, transformation logic, success or failure messages, etc. The troubleshooting process will benefit by making that information available as a standard part of the integration architecture. In Jitterbit iPaaS, this is supported through logging and alerting features.

Logging

Operation logs capture key data by default, such as operation run times and success, failure, or cancellation messages. If there are failures and the endpoint returns failure information, then the log will capture that information.

With respect to failures, the response is used to make the status determination. For example, if an HTTP status code 400 or more is received in a response, this is considered a failure. If the request has a status of 200 but the response contains data errors, this is treated as a success.

When developing an integration project, use the WriteToOperationLog function in the mappings and scripts to capture key data and steps in the process. This typically is as simple as: WriteToOperationLog("The id is: "+sourcefieldid).

If capturing the entire output of a transformation is desired, this can be done by building an operation that reads the source, performs the transformation, and writes output to a Variable or Temporary Storage endpoint instead of the target endpoint. A post-operation script can read the output and log it. Then the "real" operation can be performed.

Logs can be viewed in either the Integration Studio operation log screen or the Management Console Runtime Operations page. The Management Console Runtime Operations page can be accessed by support personnel without needing to navigate to the project.

Data in the logs is searchable. To filter just the logs you need, you can use the search syntax of message=%\<your text>% in both the Integration Studio and Management Console operation logs.

Frequently, APIs have an informative success or non-success response message. If debug logging is enabled for the API, the request body will be captured in the API logs (which are distinct from the operation logs).

Operation logs, including detailed log messages from both cloud agents and private agents, are retained for 30 days by Harmony.

Alerting

Frequently, integration results not only need to be logged, but they also need to be escalated. Email notifications can easily be attached to operations and success/failure paths or called from scripts. You can alternatively use the Email connector to configure a Send email activity as the target of an operation.

For additional information, refer to Set up alerting, logging, and error handling.

Additional resources

These sections and pages in the documentation provide additional best practices.

Tech talks

Jitterbit tech talks are video presentations that cover areas of interest to users of all levels:

Tech Talk Duration Release Date
API proxy 39:53 2019.01.09
APIs 49:22 2018.08.07
Integration Studio 43:53 2019.05.14
Complex project orchestration best practices 50:46 2018.10.16
Connector Builder 50:19 2019.07.16
Environments 1:04:22 2018.04.04
External credentials storage and management 46:58 2020.10.09
Error handling best practices 27:22 2018.03.13
Jitterbit best practices 1:04:38 2020.03.16
Logging best practices 1:03:02 2019.02.12
Open API Portal Manager 57:21 2019.11.05
Pass-through sources and global variables best practices 42:44 2018.12.05
Private agents best practices 42:43 2018.07.05
Process templates 43:27 2020.07.08
Project organization best practices 1:08:39 2018.06.08

Documentation

Jitterbit documentation has best practices included with our pages on using Jitterbit products:

Security

Integration project methodology

  • Integration Project Methodology
    Key items that a Project Manager for an Integration Studio project should know, including how to organize your team, gather and validate requirements clearly and concisely, and leverage the strengths of Jitterbit iPaaS to deliver a successful integration project.

Create projects

Logging

Private agents