Data, such as master data changes or new transactions, must be moved from a source to a target. A periodic query, underpinned by a schedule, will include a condition that uses a timestamp to select the records from the source that should be processed into the target.
There are several considerations:
- Persistent storage of the timestamp data
- Avoid an 'infinite loop', where the changes made by Jitterbit to the target are picked up and applied back to the source, and so on.
- Isolation of intended changes from unintended changes in the source, such as data changed by a trigger should not be included, but data changed by a user should.
There are pros and cons to this approach vs an event-based/real-time push approach.
- If target is unavailable, the source will continue to try to move the data until it is complete.
- Can 'move back' the timestamp to allow for re-runs
- Can get started relatively quickly
- Easier to develop and test by using smaller data sets.
- Can better control the size of data to process.
- If a 'write-back' to the source is implemented, then you can trace the integration activity from source to target.
- Only applicable if a timestamp is available, or a query condition can be constructed.
- Polling/Queries can be more complicated than Event-Based.
- The former requires a timestamp data store, queries with conditions and possibly other filters, and is sensitive to schema changes in the source. In the latter, Jitterbit is simply receiving a payload. As long as that schema does not change, then the systems are more loosely coupled, which is an advantage from a development standpoint.
- Move integration tool administrative overhead. For example, if you need to stop a flow or need to change/update the schedule in Jitterbit, which is a software configuration change. For many companies, making a change comes under a software-control process and requires more steps. With an event-driven process, the change is made in the source.
- Since timestamps are usually system generated, they do not accommodate business rules, which leads to including business rules in the middleware and extensive scripting. From an enterprise architecture standpoint, business rules should be held in the applications only. As we will see in this example, there is extensive business rule scripting.
Query NetSuite, update Salesforce
JB Sync: Get timestamp Script
The timestamps are stored in SFDC, and a query gets the sync ID and the timestamp itself. Note that timestamps are UTC, and the values are returned in an array.
Use a salesforce function to get the current time, which will be used to update the JB Sync record. Note that this is being done before the NS query instead of after the SFDC upsert, which is the more traditional method. However, it was discovered that records were being missed since changes were being made in the source that happened between the time the query was made and the time the upsert finished. The decision was made to use this approach, even at the expense of picking up the same records in the source query. Also note that there is no 'write-back' to the selected records to update it based on a successful upsert into SFDC. If that were implemented, then a query could be constructed that would not pick up duplicate records.
Pass the synctimestampticket global variable to the query
A condition is used to further filter the data.
The requirement was to filter out certain sites based on dates until the nationwide roll out was complete. Since the query does not filter based on the changing date requirements, a condition is used inside the transformation.
The incoming data is written to the operation log. Then the location internal IDs are checked, and if the location is in scope, the record's last modified by value is checked and if made by Jitterbit, is flagged to be skipped.
Since the LMB values are environment-dependent, then a check is made to the current environment.
Next, the record is checked against a list of date ranges since records before a certain time are excluded. If record passes all the tests, it will be used.
After the data is written to a temp file, Update Jitterbit Syncs is called:
Finally, the A02 operation is called if there are any records to process.
Last updated: Apr 19, 2019