Jitterbit Listening Service

Overview

The Jitterbit Listening Service is Jitterbit's application service that runs on a private agent and is consumed by certain Integration Studio connectors that have listening capabilities. The listening service provides asynchronous event handling for operations deployed on the agent.

Connectors with listening activities

The following connectors and activities use the listening service:

Prerequisites

To use a listening activity, a private agent group with one or more agents is required. Listening activities cannot be used with Jitterbit's cloud agent groups. Not all connectors support the listening service.

These prerequisites must be met when the private agent group contains a single agent or multiple agents:

• The listening service must be enabled on each agent as described in Enable the listening service on the agent below. This is a manual step in configuration and is not enabled by default.

For private agent groups that contain multiple agents, these additional prerequisites must be met for the agents to communicate with each other:

• The agent version must be 10.78 or higher (for 10.x private agents), or 11.8 or higher (for 11.x private agents).
• All agents must have these ports available:
  • 5701
  • 5801
• All agents must be within the same network group.
• If there are \(N\) agents in the agent group, at least \((N / 2) + 1\) must be running for the listening service to work.
• Set the value for agent.sdk_framework.listeners.eventsQueue in each agent's jitterbit-agent-config.properties file to be less than or equal to the number of processor cores on the agent host server. Alternatively, set agent.sdk_framework.listeners.matchEventsQueueToAvailableCores to true.

Once an operation is deployed, the Integration Studio project must have the listening service enabled at both the operation level and the activity level as described in Enable the listening service on the operation and activity below. This is manual step performed at the time of project design or project management.

Enable the listening service on the agent

The listening service is disabled by default on the agent and must be enabled manually following these steps:

1. On each private agent, open jitterbit-agent-config.properties in a text editor.

   This file is located in <JITTERBIT_HOME>/Resources/, where <JITTERBIT_HOME> is replaced with the private agent root directory.

2. Add this line to the file to enable the listener framework:

   agent.sdk_framework.listener.enabled=true
   

3. Restart the agent.

Enable the listening service on the operation and activity

The listening service must be enabled on both the operation and the listening activity. The service can be toggled at the operation level from the Integration Studio operation or at the operation or activity level from the Management Console Projects page as described below.

Integration Studio operation

Once an operation with a listener activity is deployed, an Events Disabled toggle appears at the bottom of the operation on the design canvas. By default, event listening is disabled:

To enable event listening for the operation, click the toggle:

This action will also enable event listening on any listening activities that it contains, as indicated by the color of the listener icon in the top right of the activity:

• A blue listener icon indicates listening is enabled for this activity.
• A gray listener icon indicates listening is not enabled for this activity.

Disabling event listening in an operation also disables event listening on any listening activities that it contains. Events received but not yet processed when listening is disabled are not lost and are processed as normal.

Management Console Projects page

Once an operation with a listener activity is deployed, the operation and its listening activities are displayed on the Management Console Projects page in the Listeners tab:

• Search: Enter any part of the activity, operation name, or listener status into the search box to filter the list of listening activities. The search is not case-sensitive.

• Refresh: Click the Refresh icon to refresh the list of listening activities.

• Activity: The name of the listening activity using a shortened form of the connector activity type followed by the user-provided connection name.

  Note

  The user-provided name of the listening activity is not shown.

  Click the disclosure triangle next to the activity to reveal the names of each operation that the activity is used in.

• Listener Status: The state of listening is provided for each activity and operation. Click to toggle the listening state between On (enabled) and Off (disabled). Note:

  • Disabling event listening for an activity will automatically disable listening for all operations it is used within. However, enabling event listening for an activity does not affect the state of any operations it is used in.
  • Operations can be enabled and disabled for event listening individually.
  • Changing the state is synced with the Integration Studio UI in both directions. That is, toggling the listening state in the Management Console affects the state in Integration Studio. Similarly, enabling or disabling events on an Integration Studio operation affects the state in the Management Console. A refresh may be required to update the display.

System architecture

This diagram displays how an event message moves through the system architecture when using a single private agent:

1. An operation containing a connector configured with an event listening activity is deployed and enabled for listening. One listening activity can be used in many operations and projects to receive the same event but process it differently.
2. The listening service within the agent will start a listener for that operation.
3. The listener will start actively listening for any event notifications from the endpoint.
4. When an event happens at the endpoint, it publishes an event notification that can be received by its subscribers.
5. The listener picks up the event notification message.
6. If there is one agent in the agent group, the listener relays the event message to the operation. If the agent group contains the minimum number to allow full listening service capabilities, the event message is passed to the agent with the least workload.
7. On receiving the event notification, the operation will trigger a downstream operation.

When the listening service is disabled, agents in an agent group communicate directly with Harmony. When enabled, and a minimum number of agent nodes are active, agents communicate with each other to form a cluster. The first agent registered is nominated as the cluster leader. The leader is responsible for receiving messages and distributing them to cluster members for processing. The cluster leader distributes the load among all the agents and ensures that no two agents process the same message.

Persistence

Messages sent to an agent can be stored in a database. If the agent fails, the database provides a persistent store of messages that can be resent when the agent comes back online. Persistence using the internal PostgreSQL database is enabled by default for single-agent clusters. For clusters with more than one agent, persistence must be manually enabled as follows:

1. On each private agent, edit <JITTERBIT_HOME>/Resources/jitterbit-agent-config.properties where <JITTERBIT_HOME> is replaced with the private agent root directory.

2. Add these lines to the file:

   agent.sdk_framework.queueStore.enabled=true
   agent.sdk_framework.queueStore.type=dbinternal
   
   agent.sdk_framework.persistence.enabled=true
   agent.sdk_framework.persistence.type=dbinternal
   

3. Restart the agent.

The settings agent.sdk_framework.queueStore.type=dbinternal and agent.sdk_framework.persistence.type=dbinternal cause the agent to use the local PostgreSQL database for persisting messages.

To configure an external JDBC-enabled database or Redis server, all agents within the agent group must use the following settings, where the database user has permissions to create, read, update, and delete data, and to create tables:

agent.sdk_framework.queueStore.enabled=true
agent.sdk_framework.queueStore.type=db

agent.sdk_framework.persistence.enabled=true
agent.sdk_framework.persistence.type=db

agent.sdk_framework.datastore.db.url=jdbc:sqlserver://harmony:1433;database=cloud
agent.sdk_framework.datastore.db.user=sa
agent.sdk_framework.datastore.db.password=******
agent.sdk_framework.datastore.db.databaseName=cloud
agent.sdk_framework.datastore.db.dialect=org.hibernate.dialect.SQLServerDialect
agent.sdk_framework.datastore.db.driver_class=com.microsoft.sqlserver.jdbc.SQLServerDriver

agent.sdk_framework.queueStore.enabled=true
agent.sdk_framework.queueStore.type=redis

agent.sdk_framework.persistence.enabled=true
agent.sdk_framework.persistence.type=redis

agent.sdk_framework.datastore.redis.url=redis://redis:6379

#Optional - pool configuration

agent.sdk_framework.datastore.redis.maxTotal=8
agent.sdk_framework.datastore.redis.maxIdle=8
agent.sdk_framework.datastore.redis.minIdle=0
agent.sdk_framework.datastore.redis.blockWhenExhausted=true
agent.sdk_framework.datastore.redis.maxWaitMillis=-1
agent.sdk_framework.datastore.redis.testOnBorrow=false
agent.sdk_framework.datastore.redis.testOnReturn=false
agent.sdk_framework.datastore.redis.jmxEnabled=true

Run the following command on the agent host server to get the status of the agent and the restore process:

curl localhost:46912/axis/v1/cdk/internal/members

Troubleshooting

Message not delivered

The cluster message delivery retry mechanism uses the following setting in the <JITTERBIT_HOME>/Resources/jitterbit-agent-config.properties file:

agent.sdk_framework.retry.deleteRetryableMessageAfter=60

This is the number of minutes after which a retried message is deleted.

If set to -1, messages are never deleted and are stored indefinitely.

Restore after agent failure

If an agent fails with persistence enabled, you can manually restore it using these steps:

1. On the failed agent's server, edit <JITTERBIT_HOME>/Resources/jitterbit-agent-config.properties and add the following:

   agent.sdk_framework.listener.running.mode=restore
   

2. Start the agent.

3. Use the REST API to query the status of the agent.
4. When the status for leaderNodeState is RESTORED, stop the agent.
5. Edit <JITTERBIT_HOME>/Resources/jitterbit-agent-config.properties and remove or comment out the line added in step 1.
6. Start the agent.