Skip to Content

Google BigQuery Connector

Summary

The Harmony Google BigQuery Connector establishes access to Google BigQuery.

The Google BigQuery connector provides an interface for creating a Google BigQuery connection, the foundation used for generating instances of Google BigQuery activities. These activities, once configured, interact with Google BigQuery through the connection.

The Google BigQuery connector is accessed from the design component palette's Connections tab (see Design Component Palette).

Connector Overview

This connector is used to first configure a Google BigQuery connection. Activity types associated with that connection are then used to create instances of activities that are intended to be used as sources (to provide data in an operation) or targets (to consume data in an operation).

Together, a specific Google BigQuery connection and its activities are referred to as a Google BigQuery endpoint:

Google BigQuery activity types

  • Data Transfer: Transfers data from a data source to a dataset in Google BigQuery and is intended to be used as a target in an operation.

  • Invoke Routine: Invokes a routine (such as a stored procedure, user-defined function, or table function) for a dataset in Google BigQuery and is intended to be used as a target in an operation.

  • Query: Queries a table in a dataset in Google BigQuery and is intended to be used as a source in an operation.

  • Insert Record: Inserts records into a table in Google BigQuery and is intended to be used as a target in an operation.

  • Create Structure: Creates datasets or tables in Google BigQuery and is intended to be used as a target in an operation.

  • Update Structure: Updates structures in Google BigQuery and is intended to be used as a target in an operation.

  • Update Record: Updates records in a dataset in Google BigQuery and is intended to be used as a target in an operation.

  • Delete Structure: Deletes structures from Google BigQuery and is intended to be used as a target in an operation.

  • Execute Custom Query: Executes custom queries against tables in datasets in Google BigQuery and is intended to be used as a source in an operation.

Note

This connector is a Connector SDK-based connector, which may be referred to by Jitterbit when communicating changes made to connectors built with the Connector SDK.

Prerequisites and Supported API Versions

The Google BigQuery connector requires the use of an agent version 10.1 or later. These agent versions automatically download the latest version of the connector when required.

The Google BigQuery connector uses the Google SDK version 25.4.0. Refer to the SDK documentation for information on the schema nodes and fields.

To use the Google BigQuery connector, you must have a Google service account for your project with appropriate permissions set, and certain credentials obtained from the service account (as described in Google BigQuery Prerequisites).

Troubleshooting

If you experience issues with the Google BigQuery connector, these troubleshooting steps are recommended:

  1. Click the Test button in the connection configuration to ensure the connection is successful and to ensure the latest version of the connector is downloaded to the agent (unless using the Disable Auto Connector Update organization policy).

  2. Check the operation logs for any information written during execution of the operation.

  3. Enable operation debug logging (for Cloud Agents or for Private Agents) to generate additional log files and data.

  4. If using Private Agents, you can enable connector verbose logging for this connector using this specific configuration entry of logger name and level:

    <logger name="org.jitterbit.connector.google.bigquery" level="DEBUG"/>

  5. If using Private Agents, you can check the agent logs for more information.