Introduction to BigQuery Admin Resource Charts

Introduction

BigQuery Admin Resource Charts let BigQuery administrators observe how their organization uses BigQuery slots and how their jobs perform over time. Primary use cases for Admin Resource Charts include root-cause analysis for capacity planning based on historical resource utilization and diagnosing performance changes. Data in the Admin Resource Charts is based on Jobs INFORMATION_SCHEMA tables and updates in real-time, going back a maximum of 30 days.

BigQuery Admin Resource Charts overview.

Enabling Admin Resource Charts

The Admin Resource Charts is only available for BigQuery Reservations customers. If you use on-demand pricing for bytes processed, your jobs data will not be visible from this UI.

Permissions

Admin Resource Charts require the following permissions:

  • To see all data in the Admin Resource Charts, you must have bigquery.jobs.listAll for the organization and is available to the following roles:

    • Organization Owner
    • Organization Admin
    • Organization-level BigQuery Admin
    • Organization-level BigQuery Resource Admin
    • Custom Identity and Access Management (IAM) roles

  • To see partial data at the reservation level, you must have bigquery.reservations.list and bigquery.reservationAssignments.list permissions to the reservation admin project. Usage data for jobs using reservations belonging to the admin project is available, but more granular aggregation options are not available.

  • Viewing reservation capacity data on Admin Resource Charts requires the bigquery.capacityCommitments.list permission to the admin project and is available to the following roles:

    • BigQuery User
    • BigQuery Resource Admin
    • BigQuery Admin roles

Using Admin Resource Charts

  1. To start using Admin Resource Charts, in the Cloud Console, open the BigQuery page.

    Go to BigQuery

  2. In the project picker, select the admin project that you used to purchase slots and create reservations.

  3. In the navigation panel, select the Monitoring section under Administration.

Overview tab.

Admin Resource Charts have three primary interactive components:

  • The event timeline chart displays an overview of data across a maximum of 30 days. The default is 1 day.

  • The main chart displays the selected metrics (slot usage, job concurrency, job performance, errors, or bytes processed) over time for the organization or reservation admin project. It includes a table legend that correlates with data in the chart, showing more detailed statistics aggregated over the time period.

  • A side panel to pick different charts, breakdowns, and filter options.

Main chart view

Metrics type options

You can select different metrics types to display in the charts: Slot Usage, Job Concurrency, Job Performance, Job Errors, or Bytes Processed.

Metrics type.

  • Slot Usage: Average slot usage in the selected time period.

  • Job Concurrency: Maximum number of jobs that ran simultaneously in the selected time period.

  • Job Performance: Average job run time of completed jobs, calculated from job creation time to job end time.

  • Job Error: Average slot usage in the selected time period from jobs that failed.

  • Bytes Processed: Total bytes processed by completed jobs in the selected time period.

Group-by options

You can group data in the chart view by several dimensions. For example, the slot usage chart can display the following:

Break down dimensions.

  • The Overall view is a high-level overview of the selected metric over time across the entire organization. For slot usage, capacity and usage are shown on the same chart to give an idea of overall slot utilization over time.

  • The Project view shows the selected metric grouped by project. If an organization has more than 10 projects, then only the top 10 projects with the highest slot usage, job concurrency, job duration, or failed jobs in the selected time period are shown. For slot usage, all other projects are grouped into the Other category at the bottom of the chart for easier comparison to overall capacity.

  • All other group-by views are similar to the project view, with data grouped by the respective dimension.

You can modify the time period in the following ways:

  • Select and drag the time period into the event timeline chart.

  • Select and drag the time period into the main chart.

The alignment period updates automatically as the selected time frame changes.

To narrow down the chart data, apply filters in the filter panel.

Filters.

You can filter views based on Reservations, Folders, Projects, Users, Priorities, Job Types, Errors and Jobs. Some filters are only available for certain charts. Reservations, Folders, Projects, and Users filters are populated with the respective resources that have consumed slots in the selected time frame. For example, if a project hasn't been used in the last 30 days, it does not appear in the project filter list.

The chart refreshes after you apply filters to show data within the selected parameters.

Table component

The table displays metrics that are relevant to the time period and dimension that you selected in the chart:

  • With the Admin Resource Chart grouped by reservation, the table component shows the top reservations in the organization.

  • With the Admin Resource Chart grouped by job, the table component displays the jobs that used the most slots in the organization for the selected time period.

For the Slot Usage chart, the table component displays the following data:

  • Average slot usage

    • Jobs view: The average slot usage of the job over its duration. Only usage and time within the selected time period are included in the calculation.

    • All other views: The average slot usage across all jobs over the selected time period. Jobs that didn't finish within the selected time period include only slots used within the time period.

For the Job Performance chart, the table component displays the following data:

  • Number of jobs: The number of completed jobs ran or partially ran within the selected time period.

  • Average job duration: The average job duration for completed jobs that at least partially ran within the selected time period.

  • Median job duration: The median job duration for completed jobs that at least partially ran within the selected time period.

  • Min job duration: The minimum job duration for completed jobs that at least partially ran within the selected time period. This column is hidden by default, but you can enable it in Column display options.

  • Max job duration: The maximum job duration for completed jobs that at least partially ran within the selected time period. This column is hidden by default, but you can enable it in Column display options.

For the Failed Jobs chart, the table component displays the following data:

  • Average slot usage: Same as the Slot Usage chart, but only for failed jobs.

  • Total failed jobs: The number of jobs that failed within the selected time period.

Billing

Admin Resource Charts is available at no additional cost. Queries used to populate these charts are not billed and do not use slots in user-owned reservations. Queries that process too much data will time out.