Getting started with Google Cloud Monitoring APIs — Part 2
So you have a project on GCP. You have set up your monitoring dashboards in the console, all the graphs are making sense and through just one single UI, you are able to keep a check on your whole GCP environment. Hi, you monitoring expert! Thank you for managing everything so well.
Now, what if you want these metric graphs in some part of your application? Maybe you want to format the graphs with your brand colors, pull out real-time data and present it on your website.
‘Our APIs have an impressive response time 24x7! — Here, have a look at this graph on our website itself.’
In my previous blog in this series, I wrote about how you can create custom monitoring metrics using the Python client library for Google Cloud monitoring APIs, and thereby automate this process. In this blog, I will be explaining how to use the list_time_series
method to read metric data and fetch it for our use. Quick reminder, metric data is referred to as ‘time-series data’.
Setting the scene
I highly recommend going through the previous blog to get an understanding of metrics, how they are stored and how you should authenticate your service account to use these APIs directly or through the client library.
I have tried this in a Jupyter notebook on GCP, but you can do this in either a local notebook or use the methods explained here, in your application code. To install google-cloud-monitoring
in your environment —
pip install google-cloud-monitoring
You can fetch metric data for any monitored resource on GCP. You can find a list of metric names for reference in your application in the GCP documentation, or experiment with the metric explorer under the monitoring section on your GCP console. The metrics explorer lets you filter out metrics based on resource types, resource IDs, etc and also helps you build queries for these filters which can be used in your application. If filters don’t make sense here, don’t worry we will be discussing them in the next section.
About the ‘list_time_series’ method
The list_time_series
is the method in the client library that we will use to fetch our time series data.
results = client.list_time_series(
request={
"name": project_name,
"filter": 'metric.type = "compute.googleapis.com/instance/cpu/usage_time"',
"interval": interval,
"view": monitoring_v3.ListTimeSeriesRequest.TimeSeriesView.FULL,
}
)
The request sent using this method is an abstraction over the API to call the list
method, as it might be understood from the name and can be compared to the request body in a similar API call. Here we have four parameters in the example ( order not important ) —
name
: The name of the project from where you want to fetch the monitoring metric data.filter
: This is where you specify the metric you want the data of. In this example, we only have a type but it can also have other filters as I mentioned earlier. I mostly use the metric explorer to build my filtering query. Essentially we want the output from the monitoring UI to be the one we see in our application, so validating your query using the metric explorer is a good idea.view
: There can be two responses possible from this method —
i. FULL view: The response contains the metadata of your metric and the time series data aspoints
ii. HEADER view: The response only contains the meta data of your metricinterval
: The time window from which you want the data. You have to specify the parametersstart_time
andend_time
where the time input is in unix epoch. It is a notion to donate your time value as the number of seconds that have elapsed since January 1, 1970 at midnight UTC time.
Note that by default, data is stored only for 6 weeks. If you wish to persist you monitoring data for longer, consider moving it to a database like cloud storage or BigQuery.
interval = monitoring_v3.TimeInterval(
{
"end_time": {"seconds": start_time, "nanos": nanos},
"start_time": {"seconds": (start_time - seconds_in_a_month ), "nanos": nanos},
}
)
Fetching and plotting the metric data
I have used the list_time_series
in the code snippet below and used matplotlib
(plotting library in Python) to plot the points
returned as response by the API.
Note: In the above code snippet,
results
store the API response.results
is an object you can iterate over to get metrics data. Each object is the collection ofpoints
for one resource/metric kind depending on the filter you specified. Time series data i.e. metric data is stored inpoints
and can be accessed by iterating overresult.points
whereresult
is an object inresults
.
Unlike me, please learn more about Matplotlib and make better use of the metric data. This can be a good starting point, but way to go! Here’s the output plot from the code snippet above:
If you have more than 10,000 data points in your response, you might not be able to see all the data points in the response. This is because, for a large response, the response is automatically paginated by the API and what is returned is the first page of the whole output, containing 10,000 points. I have discussed this in detail in this blog below. :)
Note: This blog post is written keeping in mind the product’s and the client library’s current latest version. For more information on monitoring on GCP, refer to https://cloud.google.com/monitoring/docs.