Explore

5X - Bring your own Google BigQuery

Introduction

5X is an enterprise-ready, full-stack data readiness platform designed to streamline your data engineering processes. This guide outlines the process of securely connecting your Google BigQuery instance to 5X, providing options that balance functionality with your specific security requirements.

Integration Options

We offer two methods for integrating Google BigQuery with 5X:

Standard Integration: Recommended for most users, providing full access to all 5X features, including BigQuery user management via the 5X platform.

Restricted Access Integration: Requires only the minimum permissions necessary for essential 5X functions, ideal for organizations with strict security policies or compliance requirements.

Platform Capabilities

Understanding how 5X interacts with Google BigQuery is crucial for choosing the right integration method. Here's an overview:

Platform Capability

Google BigQuery Credentials Used

Notes

Ingestion

Dedicated service account

Separate service account for data ingestion tasks.

IDE

Individual user's Google credentials

Each user connects with their own Google account via OAuth. Permissions based on user's BigQuery role.

Jobs

Dedicated service account

Separate service account for executing data jobs. Needs appropriate read/write permissions.

Business Intelligence (BI)

Dedicated service account

Separate service account for BI operations. Typically requires read-only access.

BigQuery User Management

Imported service account

Requires Project IAM Admin role. Only available in Standard Integration. Manages BigQuery users/roles via 5X.

There are no rows in this table

⁠

Note: BigQuery utilization tracking within the 5X platform is currently not supported.

Setup Process

1. Prepare Your Google Cloud Environment

a. Enabling Required APIs

Use an account with GCP Project Admin privileges

Navigate to "APIs & Services" > "Enabled APIs & services" in the left sidebar menu

Click the "+ ENABLE APIS AND SERVICES" button at the top

Search for and enable these APIs:

BigQuery API

Cloud Resource Manager API

Ensure each API shows as "Enabled" in your API dashboard

Note: These APIs are crucial for BigQuery operations and project resource management. It may take a few minutes for the changes to propagate after enabling.

b. Creating a Service Account

Navigate to Left Navbar Menu > IAM & Admin > Service Accounts

Click on +CREATE SERVICE ACCOUNT

Fill in the Service account details

Click on CREATE AND CONTINUE

Note: Use a descriptive name like "5X_Service_Account" that clearly identifies its association with 5X. This service account is crucial for maintaining the connection between 5X and your BigQuery instance.

c. Granting Required Roles

Navigate to "IAM & Admin" > "IAM" in the Google Cloud Console

Locate the service account you just created in the list

Click the pencil icon next to your service account to edit its roles

Click "ADD ANOTHER ROLE" and grant the following:

Project IAM Admin (Optional - required for Standard Integration)

BigQuery User

Click "SAVE" to apply the new roles

Click "DONE" to complete the service account setup

Note: Grant the Project IAM Admin role to enable full functionality, including BigQuery user management via 5X. For Restricted Access Integration, you can omit this role, acknowledging that BigQuery user management will not be available through 5X.

d. Creating a JSON file

Search and click on the newly created service account

Navigate to the "KEYS" tab in the service account details

Click on "ADD KEY" and select "Create new key"

Select "JSON" as the key type

Click "CREATE" to generate and automatically download the JSON file

Note: Keep this JSON file secure, as it contains sensitive credentials for accessing your Google Cloud resources.

2. Import to 5X

Use the created service account credentials (JSON file) to import your Google BigQuery instance into 5X.

⁠

Impact of Integration Choice

Feature

Standard Integration

Restricted Access Integration

BigQuery User Management

Available through 5X platform

Manage directly in Google Cloud Console

There are no rows in this table

⁠

Security Considerations

All actions performed by 5X are logged and auditable.

You retain full control over the BigQuery service account and can modify or revoke access at any time.

5X employs industry-standard security measures to protect your data and access credentials.

5X uses static IPs to connect to your BigQuery instance. Ensure your firewall allows connections from:

3.217.202.216

3.211.173.46

174.129.146.62

Troubleshooting

If you encounter issues:

Verify that the service account has the necessary permissions for your chosen integration method.

Check Google Cloud audit logs for any access issues.

Ensure your firewall allows connections from the 5X static IP addresses.

Contact 5X support if problems persist.

Conclusion

Our flexible integration options ensure you can leverage 5X's powerful data management capabilities while maintaining your organization's security standards. Whether you choose Standard Integration for full functionality or Restricted Access Integration for enhanced security control, 5X is committed to supporting your data readiness journey securely and efficiently across multiple Google Cloud projects and BigQuery instances.

Want to print your doc?
This is not the way.

Try clicking the ⋯ next to your doc name or using a keyboard shortcut (

CtrlP

) instead.