Integration for Google BigQuery

Automating your privacy operations by integrating MineOS with Google BigQuery

This integration allows you to:

• Automate Data Subject Requests (DSR) for user data from Google BigQuery.
• Preview and search user data in BigQuery.
• Perform data classification to detect data types stored in your BigQuery tables.

Before you start

• Make sure your MineOS plan supports integrations.
• Make sure you have access to Google Cloud Platform and Google BigQuery.

Setting up

To connect the BigQuery integration, follow these steps:

1. On the left sidebar, click Data Inventory and then Data Sources
2. Click on Add data source
3. Select Google Big Query from the catalog, then enter the Google Big Query page from your data sources list
4. In the RequestHandling tab, check the relevant checkboxes and choose the Integration handling style:
   1. Handle this data source in privacy requests - To use the integration in handling Data Subject Requests (DSR) 
   2. Scan this source using Data Classifier - To use the integration in preforming Data Classification and scanning for PII data types
5. Click "Connect" and follow the on-screen instructions for granting the required permissions.
6. Some organizations have a session timeout enabled for Google Cloud scopes. To avoid session timeout for this integration, you need to mark it as a Trusted App, as well as exempt trusted apps from session timeout. Refer to Google's article for help.
   1. Note: failing to do so will expire this integration and you will have to reconnect it every time.
7. Enter your Google Cloud Project ID, click Test & Save

Content Discovery

1. When using Content Discovery for your Google BigQuery Database, you need to make sure you have given the Google BigQuery OAuth user the required permissions to your relevant data.
2. In the RequestHandling tab, make sure to check the relevant checkbox- Scan this source using Data Classifier, To use the integration in preforming Data Classification and scanning for PII data types.
   
   
3. Choose the wanted Integration Type- 'BigQuery' to scan your tables, or 'BigQuery for Views' in case you want to scan the Views in your DB instead of the tables.
4. Choose 'Connect' to connect your Google BigQuery OAuth Account to MineOS and allow the app to read your DB content.
5. After the connection is successful, click on the 'Save' button, and go to the Data Classification pane below and select 'Start Scan' with the desired confidence level.
   
   
6. Upon content discovery, MineOS will scan your data, and analyze each table's data in our PII Processing Engine. These scan results will be added to the data types of your integration general info (according to the likelihood of the data type found).

Optional: Defining queries for DSR flow

To manage your DSR handling actions, you need to define the queries that will run in Big Query for each possible action:

Optional: Preview Query

The Preview query will be used when opening the ticket processing screen and is responsible for showing how many records were found as well as showing a sample (preview) of the data.

Example Query:

SELECT name, age, country FROM `dataset.table` WHERE email = {{endUserEmail}} 

* Preview will return the count of the number of records returned in the query response

* Preview will show the first 3 values from the query response, it supports strings and long types

* The variable is mandatory

Optional: Copy Query

The Copy query will be used for the Copy action. Copy action is running on the ticket processing page in ticket of type Copy when clicking on Generate Copy

Example Query:

SELECT * FROM `dataset.table` WHERE email = {{endUserEmail}} 

* Copy will show all records returned in the query response

* The variable is mandatory

Required: Delete Query

The Delete query will be used for the Delete action. Delete action is running on the ticket processing page in ticket of type Deletion when clicking on Delete from X sources

Example Query:

DELETE FROM `dataset.table` WHERE email = {{endUserEmail}} 

* The variable is mandatory

Paste the project id and queries in the Request handling tab in the correct inputs and click Save

Working with custom identifiers

If you wish to use this integration with a user identifier that is not email, you will need to provide it for each request by making an UpdateMetadata API call.

When making the API call, make sure to use the following field name: $googlebq

"customFields": {
    "$googlebq": "1234567"
  }

You can use any value you like, and it will be used to replace the {endUserName} parameter when running queries.

Note: you still need to use the {endUserName} parameter in queries.

Permissions required to use this integration

Mine will request the following BigQuery permissions when you activate this integration:

Current Limitations

• BigQuery data classification does not support scanning partitioned tables.

What's next?

Read more about the deletion process using integrations here.

Read more about the get a copy process using integrations here.

Talk to us if you need any help with Integrations via our chat or at portal@saymine.com, and we'll be happy to assist!🙂