Protocol Buffer import resolution

Question

Before reading through this rather long question, I've raised a bug https://github.com/GoogleCloudPlatform/python-docs-samples/issues/1103.

The documentation for Proto Packages and Name Resolution states

You can use definitions from other .proto files by importing them. To import another .proto's definitions, you add an import statement to the top of your file.

My example.proto depends on annotations.proto to transcode HTTP/JSON to gRPC. This is a trivial example, but note I use the import path from the googleapis/google/api Git repo (i.e. google/api/annotations.proto):

syntax = "proto3";
import "google/api/annotations.proto";

message MyExample {
  // Message definition here.
}

Note, annotations.proto depends on http.proto - they are siblings in the same package (googleapis/google/api)

My local project directory contains three .proto files:

1. example.proto
2. google/api/annotations.proto
3. google/api/http.proto

...or as a tree:

|____google
| |____api
| | |____annotations.proto
| | |____http.proto
|____example.proto

The target (or, 'out') directories are added too, ready to receive the generated python files:

|____generated_pb2
| |____google
| | |____api

There for my complete project directory structure is:

• example.proto
• google/api/annotations.proto
• google/api/http.proto
• generated_pb2/google/api

...or as a tree:

|____example.proto
|____google
| |____api
| | |____annotations.proto
| | |____http.proto
|____generated_pb2
| |____google
| | |____api

With this in place I can compile my protos (formatting added for readability):

python -m grpc_tools.protoc
  --python_out=generated_pb2
  --grpc_python_out=generated_pb2
  -I ~/protoc/include/google/protobuf
  -I /google/api
  example.proto

Breaking this down:

• generated_pb2 - Destination for generated python files and gprc files.
• ~/protoc/include/google/protobuf - Location of common protos shipped with protoc binary, needed since annotations.proto depends on google/protobuf/descriptor.proto.
• google/api - Location of annotations.proto and http.proto

This compiles example.proto giving:

• generated_pb2/example_pb2.py
• generated_pb2/example_pb2_gprc.py

However the first line of generated_pb2/example_pb2.py imports the generated files for annotations.proto:

from google.api import annotations_pb2 as google_dot_api_dot_annotations__pb2

This file doesn't exist. No problem, I'll compile annotations.proto separately:

python -m grpc_tools.protoc
  --python_out=generated_pb2/google/api
  --grpc_python_out=generated_pb2/google/api
  -I ~/protoc/include/google/protobuf
  -I google/api annotations.proto

Breaking this down:

• generated_pb2/google/api - Destination for generated python files and gprc files.
• ~/protoc/include/google/protobuf - Location of common protos shipped with protoc binary, needed since annotations.proto depends on google/protobuf/descriptor.proto.
• google/api - Location of http.proto upon which annotations.proto depends.

Unfortunately I get an error at this point:

google/api/http.proto: File not found.
annotations.proto: Import "google/api/http.proto" was not found or had errors.
annotations.proto:30:3: "HttpRule" is not defined.

I guess this is because annotations.proto looks for http.proto in google/api:

syntax = "proto3";
package google.api;

import "google/api/http.proto";
import "google/protobuf/descriptor.proto";

However it's unclear how this dependency is resolved. protoc --help documents the -I flag:

-IPATH, --proto_path=PATH   Specify the directory in which to search for
                            imports.  May be specified multiple times;
                            directories will be searched in order.  If not
                            given, the current working directory is used.

How can http.proto upon which annotations.proto depends, be resolved?

Answer 1

as i was trying the same thing as you do, i came up with a possible solution using a Makefile to create the appropriate files. Because i was testing with python, i installed the grpc python package and used protoc through python instead of using it directly, but the input and outcome should be the same though.

General protobuf flags used in every protoc call:

GRPC_FLAGS := \
    -I. \
    -I/usr/local/include \
    -I$(GOPATH)/src \
    -I$(GOPATH)/src/github.com/grpc-ecosystem/grpc-gateway/third_party/googleapis

Source generation

Source code specific flags:

CLIENT_FLAGS := \
    --proto_path=./protos \       <-- This is where my *.proto live
    --python_out=grpctest/client \
    --grpc_python_out=grpctest/client

Call protoc to generate project specific protocol from your *.proto

python3 -m grpc_tools.protoc $(CLIENT_FLAGS) $(GRPC_FLAGS) protos/*.proto

Annotation generation

Annotation specific flags:

CLIENT_GW_FLAGS := \
    --python_out=grpctest/client \
    --grpc_python_out=grpctest/client

Call protoc to generate annotation specific files:

python3 -m grpc_tools.protoc $(CLIENT_GW_FLAGS) $(GRPC_FLAGS) google/api/annotations.proto
python3 -m grpc_tools.protoc $(CLIENT_GW_FLAGS) $(GRPC_FLAGS) google/api/http.proto

Final Filesystem Structure

├── client.py
├── config.yml
├── file
├── google
│   └── api
│       ├── __pycache__
│       ├── annotations_pb2.py
│       ├── annotations_pb2_grpc.py
│       ├── http_pb2.py
│       └── http_pb2_grpc.py
├── grpctest_pb2.py
└── grpctest_pb2_grpc.py

Answer 2

Try this : pip install googleapis-common-protos. I encountered the same error and solved it using this method.