gRPC/protobuf generated Python files use incorrect import statements

Question

I have a grpc repo included in a python project repo as a submodule at src/grpc/protobuf with the following example structure:

protobuf/
  |
  |-data_structs/
  |   |-example_structA.proto
  |   |-example_structB.proto
  |   
  |-messages/
  |   |-example_messageA.proto
  |   |-example_messageB.proto
  |
  |-services/
      |-example_service.proto

With simplified example implementations:

//example_structA.proto

syntax = "proto3"

message example_structA {
    int id = 1;
}

//example_structB.proto

syntax = "proto3"

import "data_structs/example_structA.proto";

message example_structB {
     int id = 1;
     repeated example_structA items = 2;
}

//example_messageA.proto

syntax = "proto3"

import "data_structs/example_structB.proto"

message example_messageA {
    string info = 1;
    example_structB data = 2;
}

//example_messageB.proto

syntax = "proto3"

message example_messageB {
    string status = 1;
}

//example_service.proto

syntax = "proto3"

import "messages/example_messageA.proto";
import "messages/example_messageB.proto";

service example_service {
    rpc SendMessage (example_messageA) returns (example_messageB) {}
}

I am generating the python code with the following command, run from the python projects root dir:

find src/grpc/protobuf -name "*.proto" | xargs python -m grpc_tools.protoc =I=./src/grpc/protobuf --python_out=./src/grpc/generated --grpc_python_out=./src/grpc/generated

This correctly generates .py files in the src/grpc/generated directory, and places them into subdirectories that mimic the proto repo structure. e.g. the files for example_structB are located at src/grpc/generated/data_structs/example_structB_pb2.py and src/grpc/generated/data_structs/example_structB_pb2_grpc.py messages are in the src/grpc/generated/messages directory, etc.

These generated files have faulty imports generated though.

example_structB_pb2.py has from data_structs import example_structA_pb2 This should be from . import example_structA_pb2 or simply import example_structA_pb2

example_messageA_pb2.py has from data_structs import example_structB_pb2 This should be from ..data_structs import example_structB_pb2

So on and so forth. Services importing messages should be from ..messages import instead of just from messages import

I have not had any issues using this directory/proto structure and generation in java or c++. Is there any way to resolve this for python generated code?

Answer 1

It is not a bug it is working as intended, pyi_generator.cc:168 add's the path from the --python_out as the argument to the from and it does not checking if it is accesible by the root (if you make the entry point the same as the out path with python_out=./, it should work without the export) and it's also disallowed the use of .. in a path (importer.cc:341) so your solution is the valid one but i would have preferred to add the path from inside python, like in here:

# this line is somewhere before the import of the proto-stuff
sys.path.append(os.path.join(os.path.dirname(__file__), 'src', 'grpc', 'generated'))

Basic example (edit 1)

If you have a tree like this, with the source folder containing an exact sub-tree as if it is under ./, the topest folder will be in the entry dir, this means that it will compile with the correct import just because its the same structure.

root
│  <your main>.py
│  proto_compile.py
└─ proto_src
   └─ proto_py
        │  base.proto
        └─ structs
            ├─ message
            │    type_blob.proto
            │    type_text.proto
            └─ utils
                 blob.proto

In python, the imports are "in sys.path" finds, the entry point is added to that list at the beginning of the execution and the . is independent from that, it is relative to that file even if that file is not in a known dir. Then if you have a dir that cant follow a "branching from root" import style you just add that path to the sys.path (but this could lead to a circular import problem if you add paths from inside the project).

These are the files to test:

# proto_compile.py

import subprocess
import os

def get_proto_files(path):
    proto_files = []
    for root, dirs, files in os.walk(path):
        for name in files:
            if name.rsplit('.')[-1] == 'proto':
                proto_files.append(
                    # path is relative !
                    os.path.join(root, name)
                )
    return proto_files

def run_protoc(exe_path, src, dst):
    popen = subprocess.Popen(
        [
            exe_path,
            f'--proto_path={src}',
            f'--python_out={dst}'
        ] + get_proto_files(src),
        # could olso use os.getcwd()
        cwd=os.path.dirname(__file__),
        stdout=subprocess.PIPE,
        stderr=subprocess.STDOUT
    )
    while popen.returncode is None:
        stdout, stderr = popen.communicate()
        if stdout is not None:
            print(stdout.decode())
        if stderr is not None:
            print(stderr.decode())

if __name__ == '__main__':
    run_protoc(
        './protobuff/bin/protoc.exe',
        './proto_src',
        './'
    )

// base.proto

syntax = "proto3";

import "proto_py/structs/message/type_blob.proto";
import "proto_py/structs/message/type_text.proto";

message user_message {
    int32 id = 1;
    oneof content {
        content_text  text  = 2;
        content_image image = 3;
    }
}

// type_text.proto

syntax = "proto3";

message content_text {
    int32 id = 1;
}

// type_blob.proto

syntax = "proto3";

import "proto_py/structs/utils/blob.proto";

message content_image {
    int32 id = 1;
    blob_check check = 2;
}

// blob.proto

syntax = "proto3";

message blob_check {
    string sha256 = 1;
}

Answer 2

I resolved this by exporting src/grpc/generated to my PYTHONPATH

export PYTHONPATH=${PYTHONPATH}:/path/to/project/root/src/grpc/generated