This guide will walk you through integrating Common Fate with BigQuery. At the end of this guide you’ll have a functioning integration with Common Fate reading your dataset inventory and provisioning access to BigQuery data.

​
Prerequisites

If you’re running a BYOC (“Bring-Your-Own-Cloud”) deployment of Common Fate in your own AWS account, you’ll need to be on v1.38.0 or later of the common-fate/common-fate-deployment/aws Terraform module.

You’ll also need to have set up the Common Fate GCP integration before adding BigQuery.

​
BigQuery Setup

To provision access to BigQuery, the Common Fate provisioner role needs the following additional permissions:

  • bigquery.tables.getIamPolicy
  • bigquery.tables.setIamPolicy
  • bigquery.datasets.getIamPolicy
  • bigquery.datasets.setIamPolicy

If you used our reference integration Terraform module to deploy the GCP roles, you can add these permissions by providing the permit_bigquery_provisioning variable:

module "common-fate-gcp-roles" {
  source                                = "common-fate/common-fate-deployment/aws//modules/gcp-integration/workload-identity-roles"
  version                               = "1.38.0"
  gcp_project                           = "<ID of the GCP project you created above, excluding the '/project' prefix>"
  common_fate_aws_account_id            = "<Common Fate AWS Account ID>"
  gcp_organization_id                   = "<Your GCP organization ID>"
  common_fate_aws_reader_role_name      = "common-fate-prod-control-plane-ecs-tr"
  common_fate_aws_provisioner_role_name = "common-fate-prod-provisioner-ecs-tr"
+ permit_bigquery_provisioning          = true
}

​
Configuring Common Fate

To grant and revoke access to BigQuery, add the following Provisioner registration inside your Application Configuration repository:

resource "commonfate_webhook_provisioner" "prod" {
 url = "http://common-fate-prod-builtin-provisioner.common-fate-prod-builtin.internal:9999"
  capabilities = [
    {
      target_type = "GCP::BigQuery::Dataset"
      role_type   = "GCP::Role"
      belonging_to = {
        type = "GCP::Organization"
        id   = "organizations/123456789123"
      }
    },
    {
      target_type = "GCP::BigQuery::Table"
      role_type   = "GCP::Role"
      belonging_to = {
        type = "GCP::Organization"
        id   = "organizations/123456789123"
      }
    }
  ]
}

​
Just-In-Time access to BigQuery Datasets

To make BigQuery datasets available for Just-In-Time (JIT) access you can add a commonfate_gcp_bigquery_dataset_selector Selector resource to your Common Fate application Terraform code. As shown below, the when clause in the resource is a Cedar expression. You can use any Cedar operator in the when clause, such as && and || to combine conditions.

You’ll need to use the commonfate_bigquery_dataset_selector in conjunction with a commonfate_bigquery_dataset_availabilities and commonfate_access_workflow resources.

We’ve included some examples below.

​
Select a dataset by ID

resource "commonfate_bigquery_dataset_selector" "example" {
  id                  = "example"
  name                = "Example"
  gcp_organization_id = = "<Your GCP Organization ID>"
  when                = <<EOT
  resource == GCP::BigQuery::Dataset::"replace-this-with-your-dataset-id"
  EOT
}

​
Select multiple datasets by ID

resource "commonfate_bigquery_dataset_selector" "example" {
  id                  = "example"
  name                = "Example"
  gcp_organization_id = = "<Your GCP Organization ID>"
  when                = <<EOT
  resource == GCP::BigQuery::Dataset::"replace-this-with-your-dataset-id" || resource == GCP::BigQuery::Dataset::"some-other-dataset-id"
  EOT
}

​
Select datasets based on a naming pattern

Select datasets with a name ending in -prod:

resource "commonfate_bigquery_dataset_selector" "example" {
  id                  = "example"
  name                = "Example"
  gcp_organization_id = = "<Your GCP Organization ID>"
  when                = <<EOT
  resource.name like "*-prod"
  EOT
}

Select datasets with a name beginning with develop:

resource "commonfate_bigquery_dataset_selector" "example" {
  id                  = "example"
  name                = "Example"
  gcp_organization_id = = "<Your GCP Organization ID>"
  when                = <<EOT
  resource.name like "develop*"
  EOT
}

​
Select datasets in a particular project

resource "commonfate_bigquery_dataset_selector" "example" {
  id                  = "example"
  name                = "Example"
  gcp_organization_id = = "<Your GCP Organization ID>"
  when                = <<EOT
  resource in GCP::Project::"my-example-project"
  EOT
}

​
Select datasets in a particular folder

resource "commonfate_bigquery_dataset_selector" "example" {
  id                  = "example"
  name                = "Example"
  gcp_organization_id = = "<Your GCP Organization ID>"
  when                = <<EOT
  resource in GCP::Folder::"folders/1234567890"
  EOT
}

​
Just-In-Time access to BigQuery Tables

To make BigQuery tables available for Just-In-Time (JIT) access you can add a commonfate_gcp_bigquery_table_selector Selector resource to your Common Fate application Terraform code. As shown below, the when clause in the resource is a Cedar expression. You can use any Cedar operator in the when clause, such as && and || to combine conditions.

You’ll need to use the commonfate_gcp_bigquery_table_selector in conjunction with a commonfate_bigquery_table_availabilities and commonfate_access_workflow resources.

We’ve included some examples below.

​
Select a table by ID

resource "commonfate_bigquery_table_selector" "example" {
  id                  = "example"
  name                = "Example"
  gcp_organization_id = = "<Your GCP Organization ID>"
  when                = <<EOT
  resource == GCP::BigQuery::Dataset::"replace-this-with-your-dataset-id"
  EOT
}

​
Select multiple tables by ID

resource "commonfate_bigquery_table_selector" "example" {
  id                  = "example"
  name                = "Example"
  gcp_organization_id = = "<Your GCP Organization ID>"
  when                = <<EOT
  resource == GCP::BigQuery::Dataset::"replace-this-with-your-dataset-id" || resource == GCP::BigQuery::Dataset::"some-other-dataset-id"
  EOT
}

​
Select tables based on a naming pattern

Select tables with a name ending in -prod:

resource "commonfate_bigquery_table_selector" "example" {
  id                  = "example"
  name                = "Example"
  gcp_organization_id = = "<Your GCP Organization ID>"
  when                = <<EOT
  resource.name like "*-prod"
  EOT
}

Select tables with a name beginning with develop:

resource "commonfate_bigquery_table_selector" "example" {
  id                  = "example"
  name                = "Example"
  gcp_organization_id = = "<Your GCP Organization ID>"
  when                = <<EOT
  resource.name like "develop*"
  EOT
}

​
Select tables in a particular dataset

resource "commonfate_bigquery_table_selector" "example" {
  id                  = "example"
  name                = "Example"
  gcp_organization_id = = "<Your GCP Organization ID>"
  when                = <<EOT
  resource in GCP::BigQuery::Dataset::"example-project-2-411516:example_dataset"
  EOT
}

​
Select tables in a particular project

resource "commonfate_bigquery_table_selector" "example" {
  id                  = "example"
  name                = "Example"
  gcp_organization_id = = "<Your GCP Organization ID>"
  when                = <<EOT
  resource in GCP::Project::"my-example-project"
  EOT
}

​
Select tables in a particular folder

resource "commonfate_bigquery_table_selector" "example" {
  id                  = "example"
  name                = "Example"
  gcp_organization_id = = "<Your GCP Organization ID>"
  when                = <<EOT
  resource in GCP::Folder::"folders/1234567890"
  EOT
}
Auth0DataStax