Getting started with Google Cloud Monitoring APIs — Part 3

Iterators simplify the process of paging through API responses. Learn to use page iterators to get all data points from the `list_time_series` call.

Missing some data? [ Photo by Franki Chamaki on Unsplash ]

Enterprises running on GCP often have multiple instances of a resource in different regions/zones for HA (High Availability) i.e. for providing a consistent level of uptime for prolonged periods.

A GCP customer I worked with, had 9 routers set up in their cloud environment. These routers generated a data point with value as either 0 or 1 every minute, where ‘1’ denoted downtime of the router. Can you picture the number of data points in your head?

9 * 30 * 24 * 60 = something! Phew.

My last blog post discusses how you can fetch these metric data points. When the API response returns more than 100,000 data points, like in the case above —388800 points, the response only contains the first 100,000 points and a nextPageToken which can be used to get the next set of points. This can be a little confusing. How can you use nextPageToken? This blog suggests two ways to which you can use to get all your data points.

Where are all my data points?

As a beginner using the monitoring APIs, it was quite difficult to understand why the response did not contain all the data points, especially when you did not pass a pageSize in the request along with parameters like filter or interval.
The answer is that, if thepageSize passed in the request is empty (not passed) or more than 100,000 results, the effective pageSize is 100,000 results. If view is set to FULL, this is the maximum number of Points returned.

How can I get the remaining data points?

Refer to the API response in detail here.

{
  "timeSeries": [
    {
      object (TimeSeries)
    }
  ],
  "nextPageToken": string,
  "executionErrors": [
    {
      object (Status)
    }
  ],
  "unit": string
}

Using nextPageToken directly

As the documentation says, if there are more results than have been returned, then the nextPageTokenis set to a non-empty value.

To see the additional results, pass the nextPageTokenas the pageToken in the request for next call to this method. The response is a list of results/data points with a token for the next page, until you’ve reach a call where the nextPageToken is empty, implying all values are returned.

You are wondering isn’t this quite a long process? Well yes, making the calls yourselves and adding the token value yourself in every value is. That is why we are using pages.

Using page iterators

Iterators simplify the process of paging through API responses. API clients that have methods that follow the list pagination pattern can return an Iterator. You can use this iterator to get all of the results across all pages.

results = client.list_time_series(request={ ... })

Take a case where results contains the response from your last API call with appropriate parameters in the request. Now, we know that if the response contains more than 100,000 points the response will be paginated. Simply use the below code snippet to iterate over the pages and store all your points in a separate list. This list stays in memory. Later, we can use the points from the list instead of making calls to the API for each page.

all_points = []
for page in results.pages:
    for series in page.time_series:
        for point in series.points:
             all_points.append(point)

Since we have not mutated any points or changed them to list objects, we can still use the native methods that points have to get the time and corresponding values (metric value) at that time.

Some recommendations straight from the developers of the monitoring APIs —

Pagination as we can imagine, is making one server call per page, and hence is quite slow. A better approach when you have a large number of points is to structure your filters better so that the number of points in the response becomes lesser than 100,000. Explaining it using the router example from the beginning of the blog —

Retrieved all the data now :) [ Photo by Franki Chamaki on Unsplash ]

If all the routers have about the same number of points, you could try sending 9 queries that each have a filter with “router = x” to just get each router’s data individually. Each of the responses can then have values fitting in one page. You can use list_time_series with view set to HEADERS to get just the values of x (headers gives you just which timeseries exist without any of the actual data).

Understand `list pagination` in detail

Since these methods follow a common scheme for all APIs with paginated response, you can check list pagination in the Google API Style Guide for more details.
I actually found the solution for the monitoring API problem by referring to a lot of answers on the google APIs for calendar, because, yes, they follow the same style guide. This also makes a case for how it is important to follow best practices while writing your code. Patterns make it easy for people to assimilate information, and use it further.

Hope you found this useful!

[Photo by Author — Ending the monitoring series here with a little digital art I tried ! 👋🏼]