Using FieldMask for Google Cloud Client Libraries

2021-12-17

FieldMask can be used to return a subset of data from an API call.

For example, a GCS ListBuckets API call will return a full buckets with all its included fields.

However, if you are only interested in a partial set of data, you can specify a FieldMask in the request itself.

For more information, see Partial responses


The snippets below requests partial responses from GCS (REST) and PubSub (gRPC)

NOTE: Some service do not expose the FieldMask capability in the Cloud Client Libraries. Though not recommended, you may need to use Google API Library set if this feature is important (see Client Libraries Explained))

curl -s -H "Authorization: Bearer `gcloud auth print-access-token`"  \
     "https://storage.googleapis.com/storage/v1/b?project=mineral-minutia-820&projection=noAcl&fields=items(projectNumber)&prettyPrint=false" | jq '.'

Set thefields= parameter for REST based apis like GCS. For gRPC based apis, set the value as metadata.

NOTE: x-goog-fieldmask header needs to be in lowercase (at least in python)

project='mineral-minutia-820'

from google.cloud import storage
client = storage.Client(project=project)
for b in client.list_buckets(fields=["items(projectNumber)"]):
   print(b.project_number)

from google.cloud import pubsub_v1
publisher = pubsub_v1.PublisherClient()
project_path = f"projects/{project}"
for topic in publisher.list_topics(request={"project": project_path}, metadata = [("x-goog-fieldmask", "topics(messageStoragePolicy)")] ):
  print(topic)

TODO

NOTE: GCS Cloud Client Library does not support Partial responses so the sample below is using Google API Library

package main

import (
	"fmt"

	"golang.org/x/net/context"
	googleapiPubsub "google.golang.org/api/pubsub/v1"
	googleapiStorage "google.golang.org/api/storage/v1"
)

const (
	projectID = "mineral-minutia-820"
)

func main() {

	ctx := context.Background()

	storageService, err := googleapiStorage.NewService(ctx)
	if err != nil {
		panic(err)
	}

	pageToken := ""
	for {
		bucketsListReq := storageService.Buckets.List(projectID).Fields("items/projectNumber")
		if pageToken != "" {
			bucketsListReq.PageToken(pageToken)
		}
		r, err := bucketsListReq.Do()
		if err != nil {
			fmt.Printf("Unable to retrieve buckets: %v", err)
			break
		}
		for _, bkt := range r.Items {
			fmt.Printf("Bucket ProjectNumber: %d\n", bkt.ProjectNumber)
		}

		if r.NextPageToken == "" {
			break
		}
		pageToken = r.NextPageToken
	}

	// *******************************************

	pubsubService, err := googleapiPubsub.NewService(ctx)
	if err != nil {
		panic(err)
	}

	pageToken = ""
	for {
		topicsListReq := pubsubService.Projects.Topics.List(fmt.Sprintf("projects/%s", projectID)).Fields("topics/messageStoragePolicy")
		if pageToken != "" {
			topicsListReq.PageToken(pageToken)
		}
		r, err := topicsListReq.Do()
		if err != nil {
			fmt.Printf("Unable to retrieve topics: %v", err)
			break
		}
		for _, t := range r.Topics {
			fmt.Printf("Topics MessagePolicy: %v\n", t.MessageStoragePolicy)
		}

		if r.NextPageToken == "" {
			break
		}
		pageToken = r.NextPageToken
	}

}

TODO: java pubub does not seem to have a way to ask for partial response

package com.test;

import com.google.cloud.pubsub.v1.TopicAdminClient;
import com.google.cloud.pubsub.v1.TopicAdminClient.ListTopicsPagedResponse;
import com.google.cloud.pubsub.v1.TopicAdminSettings;
import com.google.cloud.storage.Bucket;
import com.google.cloud.storage.Storage.BucketField;
import com.google.cloud.storage.Storage.BucketListOption;
import com.google.cloud.storage.Storage;
import com.google.cloud.storage.StorageOptions;
import com.google.pubsub.v1.ListTopicsRequest;
import com.google.pubsub.v1.ProjectName;
import com.google.pubsub.v1.Topic;
import com.google.protobuf.Descriptors.FieldDescriptor;

public class TestApp {
	public static void main(String[] args) {
		TestApp tc = new TestApp();
	}

	public TestApp() {
		try {

			//itms(projectNumber)
			Storage storage_service = StorageOptions.newBuilder().build().getService();
			for (Bucket b : storage_service.list(BucketListOption.fields(BucketField.NAME)).iterateAll()) {
				System.out.println(b);
			}

			TopicAdminClient topicClient = TopicAdminClient.create(TopicAdminSettings.newBuilder().build());

			// https://cloud.google.com/java/docs/reference/protobuf/latest/com.google.protobuf.FieldMask.Builder
			//FieldDescriptor fieldDescriptor = Topic.getDescriptor().findFieldByNumber(Topic.MESSAGE_STORAGE_POLICY_FIELD_NUMBER);

			// PubSubListTopics does not seem to have a way to list fields...
			ListTopicsRequest listTopicsRequest = ListTopicsRequest.newBuilder()
					.setProject(ProjectName.format("mineral-minutia-820")) //.setField(fieldDescriptor, value)
					.build();
				
			ListTopicsPagedResponse response = topicClient.listTopics(listTopicsRequest);
			Iterable<Topic> topics = response.iterateAll();
			for (Topic topic : topics)
				System.out.println(topic);

		} catch (Exception ex) {
			System.out.println("Error:  " + ex);
		}
	}

}

TODO. Node GCS and PubSub do not seem to have a way to override it with the Cloud API library set

using System;

using Google.Cloud.Storage.V1;
using Google.Cloud.PubSub.V1;
using System.Threading.Tasks;

using Google.Api.Gax.ResourceNames;
using Google.Api.Gax.Grpc;

namespace main
{
    class Program
    {
        const string projectID = "mineral-minutia-820";
        [STAThread]
        static void Main(string[] args)
        {
            new Program().Run().Wait();
        }

        private Task Run()
        {
            var client = StorageClient.Create();
            var options = new ListBucketsOptions { Fields = "items(projectNumber),nextPageToken" };
            foreach (var b in client.ListBuckets(projectID, options))
                Console.WriteLine(b.ProjectNumber);

            PublisherServiceApiClient publisher = PublisherServiceApiClient.Create();
            ProjectName projectName = ProjectName.FromProject(projectID);
            CallSettings callSettings = CallSettings.FromFieldMask("topics/messageStoragePolicy");
            foreach (Topic t in publisher.ListTopics(projectName, null, null, callSettings))
                Console.WriteLine(t.MessageStoragePolicy);
            return Task.CompletedTask;
        }
    }
}

This site supports webmentions. Send me a mention via this form.



comments powered by Disqus