View application telemetry

This document describes how to view the dashboards that Application Monitoring creates for your App Hub applications, services, and workloads. These dashboards can help you understand how your application's resources are performing and to diagnose issues.

The Application Monitoring dashboards display the following:

To determine what data to display, Application Monitoring uses the application-specific labels and attributes that supported infrastructure and instrumented applications attach to the telemetry they generate.

Application-specific labels and attributes

This section lists the App Hub-specific labels and attributes that might be attached to your log, metric, and trace data. These labels let Google Cloud Observability construct the hierarchy of dashboards, filter the displayed telemetry, and generate the topology map.

In this section, interpret a.b.{x,y} as a.b.x and a.b.y.

Log labels

Log entries from Google Cloud infrastructure contain an apphub field that lists the labels for the application and for a service or workload. The following illustrates the format of this data:

apphub: {
  application: {
    container: "projects/my-project"
    id: "my-app"
    location: "my-app-location"
  }
  workload: {
    criticalityType: "MEDIUM"
    environmentType: "STAGING"
    id: "my-workload-id"
  }
}

Metric labels

Metric data from Google Cloud infrastructure includes the following metadata labels:

  • metadata.system_labels.apphub_application_{container,id,location}
  • metadata.system_labels.apphub_{workload,service}_{criticality_type,environment_type,id}

Metric data from instrumentation you added to your applications can include the following metric labels:

  • metric.labels.apphub_application_{container,id,location}
  • metric.labels.apphub_workload_{criticality_type,environment_type,id}

Trace resource attributes

Trace spans generated by instrumentation you added to your applications can include the following resource attributes:

  • gcp.apphub.application.{container,id,location}
  • gcp.apphub.{workload,service}.{criticality_type,environment_type,id}

Alerting policy labels

Service and workload labels aren't automatically attached to alerting policies. However, you can associate an alerting policy with a specific workload or service by attaching all of the following labels with the following keys to the alerting policy:

  • apphub_application_location
  • apphub_application_id
  • apphub_service_id or apphub_workload_id

When an alerting policy includes the previous keys, incidents for the policy are shown on the Application Monitoring dashboards.

For more information, see Associate an alerting policy with a service or workload.

App Hub supports two configurations: app-enabled folder and App Hub host projects. This section applies only when you use an app-enabled folder.

When you use an app-enabled folder, the Google Cloud console automatically resets the resource picker from an app-enabled folder to the folder's management project when you navigate to an Google Cloud Observability page. If you navigate to another page in the Google Cloud console, you the resource picker is reset to the app-enabled folder.

Cloud Logging supports folders as a resource. For example, folders store system-created log sinks and log buckets named _Default and _Required. To access the folder resource, use the resource picker and manually select the app-enabled folder.

Before you begin

Before you can use Application Monitoring, make sure that you configure your App Hub host project or management project . This is the project that you use to view your application's telemetry data:

  • To get the permissions that you need to view log and metric data, and to create alerting policies and customize dashboards, ask your administrator to grant you the following IAM roles:

    • Logs Viewer (roles/logging.viewer) on your App Hub host project or management project and any other projects listed its default log scope. This role grants access to most log entries in the _Default log bucket. For more information, see Logging roles.
    • Logs View Accessor (roles/logging.viewAccessor) on the log views listed in the default log scope of your App Hub host project or management project . To learn more about granting access to a log view, see Control access to a log view.
    • Monitoring Editor (roles/monitoring.editor) on your App Hub host project or management project . For principals who don't need to create or modify alerting policies or dashboards, consider granting the Monitoring Viewer role (roles/monitoring.viewer).
    • Cloud Trace User (roles/cloudtrace.user) on your App Hub host project or management project , and the projects in the default trace scope.
    • App Hub viewer (roles/apphub.viewer) on your App Hub host project or management project . For more information, see App Hub roles and permissions.

    For more information about granting roles, see Manage access to projects, folders, and organizations.

    You might also be able to get the required permissions through custom roles or other predefined roles.

  • Make sure that you've configured the observability scope of your App Hub host project or management project . For more information, see Set up Application Monitoring.

    The observability scope controls which data is available to your App Hub host project or management project . The default settings for these scopes might not let you view all log or metric data.

  • Update the alerting policies that monitor your application data to include application-specific labels. For more information, see Associate an alerting policy with a service or workload.

View summary of all applications

The top-level Application Monitoring dashboard provides a concise summary of the state of your registered applications. For each application, this page lists the number of open incidents and information about the location, criticality, and environment.

The links on the top-level Application Monitoring dashboard let you open the hierarchy of dashboards that Google Cloud Observability creates for each application. These application-specific dashboards let you explore the telemetry of your application, services, and workloads. They also let you explore the topology of your application.

The following screenshot illustrates the summary dashboard:

Dashboard that lists the registered applications.

To open the top-level dashboard, which lists summary information for your applications registered with App Hub, do the following:

  1. In the Google Cloud console, go to the Application monitoring page:

    Go to Application monitoring

    If you use the search bar to find this page, then select the result whose subheading is Monitoring.

  2. In the toolbar of the Google Cloud console, select your App Hub host project or management project .

  3. Optional: To open an application in App Hub, click More, and select Manage in AppHub.

  4. To explore the telemetry or the topology for a specific application, select the application.

    The detailed view for the application opens. This page contains three tabs that display different types of information:

    • The Overview tab displays summary information about your App Hub application. For example, it lists the application's services and workloads, the number of open incidents, and some golden signals.

    • The Dashboard tab displays telemetry data like golden signals, log data, trace data, and open incidents. Each section on this tab provides options to let you explore the data in more detail.

    • The Topology tab (preview) displays a dynamic view of your application's services and dashboards, incidents, and traffic. To learn about the Topology tab, see View application topology.

View an application's telemetry or topology

To open an application's dashboard, do the following:

  1. In the Google Cloud console, go to the Application monitoring page:

    Go to Application monitoring

    If you use the search bar to find this page, then select the result whose subheading is Monitoring.

  2. In the toolbar of the Google Cloud console, select your App Hub host project or management project .

  3. On the Application monitoring page, select the name of an application.

    The dashboard for the application opens with the Overview tab selected. This tab lists information about the application, each service and workload, the number of open incidents, and values for the golden signals:

    • Traffic: Incoming request rates on the service or workload over the selected time period.
    • Server error rate: Average percentage of incoming requests that generate or map to 5xx HTTP response codes over the selected time period.
    • P95 latency: 95th percentile of latency for a request served over the selected time period, in milliseconds.
    • Saturation: Measures how full your service or workload is. For example, for managed instance groups (MIGs), Cloud Run, and Google Kubernetes Engine deployments, this field shows the CPU utilization.

    An overview that lists the services and workloads in an application.

    Options on the Overview tab let you do the following:

    • Open the application, a service, or a workload in App Hub.
    • Open the detailed dashboard for a service or workload.
    • Reset the time range. When you open the dashboard, the time range is set to the most recent hour.

  4. To view telemetry data and information about open incidents, select the Dashboard tab.

    Dashboard that lists the golden signals and logs for an application.

    The Dashboard tab displays log, metric, trace, and incident information. This tab is also configured to overlay event annotations on your charts. Annotations can help you correlate data from different sources when you are troubleshooting an issue. To view which annotations are enabled, in the toolbar, click Annotations.

    The tab also contains filters, which you can use to restrict what data the tab displays. For more information about how to explore the data shown on this tab, see the following sections of this document:

  5. To view the topology of your application, select the Topology tab (preview).

    To learn about the topology map and how to configure your application so that Google Cloud Observability has the information needed to construct this map, see View application topology.

Filter a dashboard

The Dashboard tab displays data for services and workloads in an application. However, you can filter this dashboard to a specific service or to a set of services.

For example, the toolbar on the Dashboard tab contains a filter like apphub_service_id: *. If you want to view data for specific services, then click Menu, and then select those services from the list.

Explore log data

If you are viewing the Dashboard tab of an application dashboard, then your log data is shown in the Application Logs section. Otherwise, your data data is shown in the Logs section.

The Application Logs and Logs sections of a dashboard display log entries that match a filter and are stored by the resources listed in the default log scope of your App Hub host project or management project . For applications, the filter is similar to the following:

-- Application logs
(apphub_application_location=global)
(apphub_application_id=my-app)

For services and workloads, the filter contains additional clauses. To view the filter applied to your log data, click Show query.

If you want to write queries, then either open a flyout or open the Logs Explorer page. Both options let you write and run queries, and to change scope. However, the flyout retains your current context and lets you save your exploration results to a custom dashboard:

  • To open a flyout, click Explore data.

    When your exploration is complete, click Cancel to close the flyout and discard your changes. If you want to save your changes to a custom dashboard, then click Save to dashboard and complete the dialog. You can't save your changes to the dashboard you were viewing when you opened the flyout.

    The flyout doesn't list the filters shown by the Show query button. Those filters are applied at the dashboard level and can't be removed.

  • To open the Logs Explorer page in a new browser tab, click View in Logs Explorer.

    The query is preconfigured to show log entries that match specific application labels and that are in the time range specified by the dashboard. When you close the browser tab, your changes are discarded.

For more information about how to explore your logs, see the following documents:

Explore metric data

The charts and tables on your dashboard display golden signals and other metric data that is relevant for the resources used by your application, service, or workload. The displayed data can be stored in any project listed by the metrics scope of the your App Hub host project or management project .

View chart configuration

To view details like the metric type and the aggregation options for a chart that displays metric data, go to the toolbar of a chart or table, and then click Information.

Explore data

To explore a chart or table in more detail, either open a flyout or open the Metrics Explorer page. Both options let you try different chart settings and save the results to a custom dashboard. However, the flyout retains your current context:

  • To open a flyout, click Explore data.

    When your exploration is complete, click Cancel to close the flyout and discard your changes. If you want to save your changes to a custom dashboard, then click Save to dashboard and complete the dialog. You can't save your changes to the dashboard you were viewing when you opened the flyout.

  • To view the chart or table in the Metrics Explorer, go to the toolbar, click More, and then select View in Metrics Explorer.

    You now have a temporary chart that you can modify and explore. For example, you might change the aggregation options or change the display style. To save your chart to a custom dashboard, click Save to dashboard and complete the

Correlate metric and log data

When you view a chart, you might want to explore log data that was written against the same resource as the charted metric data. For example, if you see a spike or a dip in a chart, then you might want to explore log entries so that you can get additional information about the system.

One way you can correlate your metric and log data, is to use chart options to open a flyout that displays both the chart and selected log entries. You can then use the toolbars and menus in the flyout to explore your metric and log data. For example, you might add filters to the metric query, modify the logging query, or expand and explore log entries.

The following screenshot illustrates the flyout:

Example of metric and log correlation.

The flyout doesn't display all log entries that match the dashboard's time-range selector. Instead, the system creates a logging query that is based on the metric query. The logging query includes any applicable variables, but it doesn't include dashboard-level filters.

View metric and log data

You can correlate metric and log data when a chart displays any metric type except a log-based metric or a Prometheus metric. When a chart displays a log-based metric, a Prometheus metric, or when it doesn't display metric data, you can't correlate metric and log data.

To open the flyout that displays both metric and log data, do one of the following:

  • To filter log data only by the resource type and applicable variables, in the chart's toolbar, click More chart options, and then select Inspect related logs.

  • To show log entries that correlate with a specific time series, in the chart, use your pointer to select a point on the time series, and then go to the tooltip and select Inspect related logs.

    In this mode, the flyout modifies the time-range selector to be five minutes before and after the timestamp of the selected point.

If your chart doesn't display an Inspect related logs option, then it doesn't support metric and log correlation.

Save your changes

You can save the chart or logs panel shown in the flyout, including any changes that you make, to a custom dashboard. If you save the chart to the same dashboard from which you opened the flyout, then a new chart is created on the dashboard.

To save the chart or logs panel to a custom dashboard, open the widget's Actions menu, select Save to dashboard, and then complete the dialog. You can select an existing custom dashboard or you can create a dashboard and then save the widget.

Close the flyout

To close the flyout, in the toolbar, click Cancel.

Explore incidents

The Incidents section of a dashboard lists recent incidents for alerting policies that are associated with your application. If you are viewing a dashboard for a service or workload, then this section displays incidents for that service or workload.

To get more information about your incidents and alerting policies, do the following:

  • To find details about the alerting policy that caused the incident, click the policy name. The dashboard that opens displays detailed information about the alerting policy, including a chart that displays the monitored data and the labels.

  • To view all alerting policies associated with the application, go to the toolbar and click Explore data. A flyout opens that lists all policies. To return to the dashboard, click Cancel.

If you don't see any entries in the Incidents section, then one of the following is true:

  • There aren't any open incidents.

    To view closed incident and those that opened outside the dashboard's time range setting, in the toolbar of the Incidents section, disable Grouped by policy, and then enable Show closed incidents.

    If the table remains empty, then either you don't have any associated alerting policies or none of the associated policies have reported an incident.

  • None of the alerting policies stored in your App Hub host project or management project are associated with a service or workload. To make this association, add labels to the alerting policy. For more information, see Associate an alerting policy with a service or workload.

Explore trace data

The Traces section of a dashboard lists information about spans generated by your registered services and workloads when those spans contain application-specific resource attributes. The required resource attributes are available when you instrument your application. Only spans that satisfy the dashboard-level filters are shown.

For trace data, the application-level dashboard groups spans by their name and by the service or workload name, and then displays the following information in a tabular format:

  • Span name.
  • Service or workload name.
  • The number of spans in the grouping.
  • The 50th, 90th, and 95th latency percentiles of the spans in the grouping.
  • The error rate.

To get more information about a grouping, select it. An interactive flyout opens:

Example of the trace flyout.

The interactive flyout displays the following:

  • Individual spans in the group. The first entry is automatically selected.

  • Information about the parent trace for the selected span. This information includes the following:

    • Statistical information, like the total duration.
    • The call hierarchy.
    • The service or workload associated with each call. App Hub-registered services are shown with the service icon, . Similarly, workloads are shown with the workload icon, .
    • Latency of each call.
    • Success status of each, which is represented by blue or red.

    The tabular widget lets you explore attributes attached to a span, or to explore other data, like related log entries. This widget displays data for the selected span.

To view your application spans on the Trace Explorer page, toolbar of the Traces section, click View in Trace Explorer.

  • Application-specific filters are automatically applied.
  • You can add, remove, or modify any filters. However, changes that you make are discarded when you close the Trace Explorer page.

To learn more about exploring your trace data, see Find and explore traces.

Explore service or workload telemetry and incidents

Application Monitoring creates a dashboard for each service and workload. These dashboards display log, metric, and incident information, and they can help you understand the health of a specific service or workload. You can explore the data on these dashboards similarly to exploring the data shown on an application's Dashboard tab.

The following screenshot, illustrates the dashboard for a workload:

Detailed dashboard for a workload.

To open the Application Monitoring dashboard for a service or workload, do the following:

  1. In the Google Cloud console, go to the Application monitoring page:

    Go to Application monitoring

    If you use the search bar to find this page, then select the result whose subheading is Monitoring.

  2. In the toolbar of the Google Cloud console, select your App Hub host project or management project .

  3. On the Application monitoring page, select the name of an application, and then select a service or workload.

    A dashboard displays the log, metric, and incident information for the service or workload you selected. For workloads, the dashboard is configured to overlay event annotations on your charts. Annotations can help you correlate data from different sources when you are troubleshooting an issue. To view which annotations are enabled, in the toolbar, click Annotations.

    The summary card displays general information and the infrastructure type. When the infrastructure type can't be identified, this field displays generic text like App Hub Service. For more information, see Application Monitoring supported infrastructure.

    For information about how to explore the data on a service or workload dashboard, see the following sections of this document:

Create a custom dashboard

Google Cloud Observability supports the creation of custom dashboards, which let you have full control over the content of the dashboard. Custom dashboards are listed in the Dashboards page of the Google Cloud console, and they can be copied, edited, and deleted. They can also be the destination where you save a dashboard widget that you've configured. For example, if you are exploring a chart on an application dashboard, then you can save that chart to a custom dashboard by going to the toolbar and selecting Save to dashboard.

If you plan to create a custom dashboard for an application, service, or workload, then the following might be helpful:

Associate an alerting policy with an App Hub application

You associate an alerting policy with a service or workload by adding application-specific labels to the alerting policy. When you add these labels, they are listed in the alerting policy and also included in any incidents. To learn more about labels, see Annotate incidents with labels.

To associate an alerting policy with a workload or service by using the Google Cloud console, do the following:

  1. In the Google Cloud console, go to the  Alerting page:

    Go to Alerting

    If you use the search bar to find this page, then select the result whose subheading is Monitoring.

  2. In the toolbar of the Google Cloud console, select your App Hub host project or management project .
  3. Find the alerting policy, click View more, select Edit, and then go to the Notifications and name section.
  4. In the Application labels section, select your application and then select your workload or service.
  5. Click Save policy.

After you complete these steps, labels with the following keys are attached to your alerting policy. These labels identify your application and your service or workload:

  • apphub_application_location
  • apphub_application_id
  • apphub_service_id or apphub_workload_id

You can also add user labels to an alerting policy by using the Google Cloud CLI, Terraform, or the Cloud Monitoring API. However, you must use the label keys shown in the previous example. For more information, see the following:

Troubleshoot

For information that might help you understand what data you are viewing, or why a dashboard doesn't display data, see Troubleshoot Application Monitoring.

What's next