Skip to content
Common Fate
Docs Granted

BigQuery

This guide will walk you through integrating Common Fate with BigQuery. At the end of this guide you’ll have a functioning integration with Common Fate reading your dataset inventory and provisioning access to BigQuery data.

Prerequisites

If you’re running a BYOC (“Bring-Your-Own-Cloud”) deployment of Common Fate in your own AWS account, you’ll need to be on v1.38.0 or later of the common-fate/common-fate-deployment/aws Terraform module.

You’ll also need to have set up the Common Fate GCP integration before adding BigQuery.

BigQuery Setup

To provision access to BigQuery, the Common Fate provisioner role needs the following additional permissions:

  • bigquery.tables.getIamPolicy
  • bigquery.tables.setIamPolicy
  • bigquery.datasets.getIamPolicy
  • bigquery.datasets.setIamPolicy

If you used our reference integration Terraform module to deploy the GCP roles, you can add these permissions by providing the permit_bigquery_provisioning variable:

module "common-fate-gcp-roles" {
  source                                = "common-fate/common-fate-deployment/aws//modules/gcp-integration/workload-identity-roles"
  version                               = "1.38.0"
  gcp_project                           = "<ID of the GCP project you created above, excluding the '/project' prefix>"
  common_fate_aws_account_id            = "<Common Fate AWS Account ID>"
  gcp_organization_id                   = "<Your GCP organization ID>"
  common_fate_aws_reader_role_name      = "common-fate-prod-control-plane-ecs-tr"
  common_fate_aws_provisioner_role_name = "common-fate-prod-provisioner-ecs-tr"
 permit_bigquery_provisioning          = true
}

Configuring Common Fate

To grant and revoke access to BigQuery, add the following Provisioner registration inside your Application Configuration repository:

resource "commonfate_webhook_provisioner" "prod" {
 url = "http://common-fate-prod-builtin-provisioner.common-fate-prod-builtin.internal:9999"
  capabilities = [
    {
      target_type = "GCP::BigQuery::Dataset"
      role_type   = "GCP::Role"
      belonging_to = {
        type = "GCP::Organization"
        id   = "organizations/123456789123"
      }
    },
    {
      target_type = "GCP::BigQuery::Table"
      role_type   = "GCP::Role"
      belonging_to = {
        type = "GCP::Organization"
        id   = "organizations/123456789123"
      }
    }
  ]
}

Just-In-Time access to BigQuery Datasets

To make BigQuery datasets available for Just-In-Time (JIT) access you can add a commonfate_gcp_bigquery_dataset_selector Selector resource to your Common Fate application Terraform code. As shown below, the when clause in the resource is a Cedar expression. You can use any Cedar operator in the when clause, such as && and || to combine conditions.

You’ll need to use the commonfate_bigquery_dataset_selector in conjunction with a commonfate_bigquery_dataset_availabilities and commonfate_access_workflow resources.

We’ve included some examples below.

Select a dataset by ID

resource "commonfate_bigquery_dataset_selector" "example" {
  id                  = "example"
  name                = "Example"
  gcp_organization_id = = "<Your GCP Organization ID>"
  when                = <<EOT
  resource == GCP::BigQuery::Dataset::"replace-this-with-your-dataset-id"
  EOT
}

Select multiple datasets by ID

resource "commonfate_bigquery_dataset_selector" "example" {
  id                  = "example"
  name                = "Example"
  gcp_organization_id = = "<Your GCP Organization ID>"
  when                = <<EOT
  resource == GCP::BigQuery::Dataset::"replace-this-with-your-dataset-id" || resource == GCP::BigQuery::Dataset::"some-other-dataset-id"
  EOT
}

Select datasets based on a naming pattern

Select datasets with a name ending in -prod:

resource "commonfate_bigquery_dataset_selector" "example" {
  id                  = "example"
  name                = "Example"
  gcp_organization_id = = "<Your GCP Organization ID>"
  when                = <<EOT
  resource.name like "*-prod"
  EOT
}

Select datasets with a name beginning with develop:

resource "commonfate_bigquery_dataset_selector" "example" {
  id                  = "example"
  name                = "Example"
  gcp_organization_id = = "<Your GCP Organization ID>"
  when                = <<EOT
  resource.name like "develop*"
  EOT
}

Select datasets in a particular project

resource "commonfate_bigquery_dataset_selector" "example" {
  id                  = "example"
  name                = "Example"
  gcp_organization_id = = "<Your GCP Organization ID>"
  when                = <<EOT
  resource in GCP::Project::"my-example-project"
  EOT
}

Select datasets in a particular folder

resource "commonfate_bigquery_dataset_selector" "example" {
  id                  = "example"
  name                = "Example"
  gcp_organization_id = = "<Your GCP Organization ID>"
  when                = <<EOT
  resource in GCP::Folder::"folders/1234567890"
  EOT
}

Just-In-Time access to BigQuery Tables

To make BigQuery tables available for Just-In-Time (JIT) access you can add a commonfate_gcp_bigquery_table_selector Selector resource to your Common Fate application Terraform code. As shown below, the when clause in the resource is a Cedar expression. You can use any Cedar operator in the when clause, such as && and || to combine conditions.

You’ll need to use the commonfate_gcp_bigquery_table_selector in conjunction with a commonfate_bigquery_table_availabilities and commonfate_access_workflow resources.

We’ve included some examples below.

Select a table by ID

resource "commonfate_bigquery_table_selector" "example" {
  id                  = "example"
  name                = "Example"
  gcp_organization_id = = "<Your GCP Organization ID>"
  when                = <<EOT
  resource == GCP::BigQuery::Dataset::"replace-this-with-your-dataset-id"
  EOT
}

Select multiple tables by ID

resource "commonfate_bigquery_table_selector" "example" {
  id                  = "example"
  name                = "Example"
  gcp_organization_id = = "<Your GCP Organization ID>"
  when                = <<EOT
  resource == GCP::BigQuery::Dataset::"replace-this-with-your-dataset-id" || resource == GCP::BigQuery::Dataset::"some-other-dataset-id"
  EOT
}

Select tables based on a naming pattern

Select tables with a name ending in -prod:

resource "commonfate_bigquery_table_selector" "example" {
  id                  = "example"
  name                = "Example"
  gcp_organization_id = = "<Your GCP Organization ID>"
  when                = <<EOT
  resource.name like "*-prod"
  EOT
}

Select tables with a name beginning with develop:

resource "commonfate_bigquery_table_selector" "example" {
  id                  = "example"
  name                = "Example"
  gcp_organization_id = = "<Your GCP Organization ID>"
  when                = <<EOT
  resource.name like "develop*"
  EOT
}

Select tables in a particular dataset

resource "commonfate_bigquery_table_selector" "example" {
  id                  = "example"
  name                = "Example"
  gcp_organization_id = = "<Your GCP Organization ID>"
  when                = <<EOT
  resource in GCP::BigQuery::Dataset::"example-project-2-411516:example_dataset"
  EOT
}

Select tables in a particular project

resource "commonfate_bigquery_table_selector" "example" {
  id                  = "example"
  name                = "Example"
  gcp_organization_id = = "<Your GCP Organization ID>"
  when                = <<EOT
  resource in GCP::Project::"my-example-project"
  EOT
}

Select tables in a particular folder

resource "commonfate_bigquery_table_selector" "example" {
  id                  = "example"
  name                = "Example"
  gcp_organization_id = = "<Your GCP Organization ID>"
  when                = <<EOT
  resource in GCP::Folder::"folders/1234567890"
  EOT
}