protocol buffers in python: no classes generated

Question

My proto file is as follows:

syntax = "proto3";
option csharp_namespace = "Proto";

message FileListRequest {
    repeated File Files = 1;
}

message File {
    string Path = 1;
}

message ImageFile {
    File File = 1;
    Size Size = 2;
    bytes Content = 3;
}

message Size {
    int32 Width = 1;
    int32 Height = 2;
}

message SendNextFile {
    
}

I compile it with the following command:

protoc --proto_path=. -I . --python_out=..\..\python\Modules\PreloadingIteratorWrapper\ .\filelist.proto

This creates the following file:

# -*- coding: utf-8 -*-
# Generated by the protocol buffer compiler.  DO NOT EDIT!
# source: filelist.proto
"""Generated protocol buffer code."""
from google.protobuf.internal import builder as _builder
from google.protobuf import descriptor as _descriptor
from google.protobuf import descriptor_pool as _descriptor_pool
from google.protobuf import symbol_database as _symbol_database
# @@protoc_insertion_point(imports)

_sym_db = _symbol_database.Default()




DESCRIPTOR = _descriptor_pool.Default().AddSerializedFile(b'\n\x0e\x66ilelist.proto\"\'\n\x0f\x46ileListRequest\x12\x14\n\x05\x46iles\x18\x01 \x03(\x0b\x32\x05.File\"\x14\n\x04\x46ile\x12\x0c\n\x04Path\x18\x01 \x01(\t\"F\n\tImageFile\x12\x13\n\x04\x46ile\x18\x01 \x01(\x0b\x32\x05.File\x12\x13\n\x04Size\x18\x02 \x01(\x0b\x32\x05.Size\x12\x0f\n\x07\x43ontent\x18\x03 \x01(\x0c\"%\n\x04Size\x12\r\n\x05Width\x18\x01 \x01(\x05\x12\x0e\n\x06Height\x18\x02 \x01(\x05\"\x0e\n\x0cSendNextFileB\x08\xaa\x02\x05Protob\x06proto3')

_builder.BuildMessageAndEnumDescriptors(DESCRIPTOR, globals())
_builder.BuildTopDescriptorsAndMessages(DESCRIPTOR, 'filelist_pb2', globals())
if _descriptor._USE_C_DESCRIPTORS == False:

  DESCRIPTOR._options = None
  DESCRIPTOR._serialized_options = b'\252\002\005Proto'
  _FILELISTREQUEST._serialized_start=18
  _FILELISTREQUEST._serialized_end=57
  _FILE._serialized_start=59
  _FILE._serialized_end=79
  _IMAGEFILE._serialized_start=81
  _IMAGEFILE._serialized_end=151
  _SIZE._serialized_start=153
  _SIZE._serialized_end=190
  _SENDNEXTFILE._serialized_start=192
  _SENDNEXTFILE._serialized_end=206
# @@protoc_insertion_point(module_scope)

According to the documentation this file should contain a class for each message type, but it doesn't. Why?

@Kurt ['DESCRIPTOR', 'File', 'FileListRequest', 'ImageFile', 'SendNextFile', 'Size', '_FILE', '_FILELISTREQUEST', '_IMAGEFILE', '_SENDNEXTFILE', '_SIZE', '__builtins__', '__cached__', '__doc__', '__file__', '__loader__', '__name__', '__package__', '__spec__', '_builder', '_descriptor', '_descriptor_pool', '_sym_db', '_symbol_database']. It's looking like the tutorial is incorrect and the classes are generated at runtime: github.com/protocolbuffers/protobuf/issues/2150 — Ian Newson, Commented Apr 21, 2022 at 20:21
ya that was my guess, you can see bits and pieces of your classes in that serialized string passed to DESCRIPTOR — Kurt, Commented Apr 21, 2022 at 20:43
@DazWilkin I understand that, but the documentation also specifically states: "and some mysteriously empty classes, one for each message type". This is absent in my generated code. — Ian Newson, Commented Apr 22, 2022 at 13:27
The documentation is incorrect|outdated. Your code should work with that generated code. Interestingly, I used the grpcio-tools to compile a repro of you question because this included (a version of) protoc. That generated class does not include "mysteriously empty classes" either but it is different again. — DazWilkin, Commented Apr 22, 2022 at 15:40

Krzysiek · Accepted Answer · 2023-03-09 22:39:41Z

32

I have the same issue. Experimenting with grpc-tools (I found answer in another thread suggesting usage of this tool) I finally found a solution.

Just add to protoc command this arg: --pyi_out like:

protoc --proto_path=. -I . --python_out=..\..\python\Modules\PreloadingIteratorWrapper\ --pyi_out=..\..\python\Modules\PreloadingIteratorWrapper\ .\filelist.proto

It will generate for you _bp2.py as well as corresponding _bp2.pyi (stub file). After that your IDE will see the class names, intellisense will work etc.

edited Mar 9, 2023 at 22:39

answered Nov 25, 2022 at 16:50

Krzysiek

79866 silver badges1414 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

Poonam Anthony Over a year ago

Thanks, this helped me. A small correction:

protoc --proto_path=. -I . --python_out=..\..\python\Modules\PreloadingIteratorWrapper\ --pyi_out=..\..\python\Modules\PreloadingIteratorWrapper\  .\filelist.proto

Gabor Somogyi Over a year ago

Ah man, saved me quite some time. I was struggling with it and exactly this helped :)

lbt Over a year ago

This answer is misleading :( The part of the google documentation that says "it generates special descriptors for all your messages, enums, and fields, and some mysteriously empty classes, one for each message type" is wrong. These mysterious classes are now even more mysteriously dynamically generated at runtime! (see comment by mic on dimankiev's answer which defends the documentation as correct - it really isn't!) This answer helpfully produces a .pyi stub file (see peps.python.org/pep-0561 ) but that just has type hints for us/editors and is not needed to run the code (try it).

Dmytro N. · Accepted Answer · 2022-08-25 00:54:32Z

Documentation says:

Unlike when you generate Java and C++ protocol buffer code, the Python protocol buffer compiler doesn't generate your data access code for you directly.

It means, that your .proto files won't be converted into some familiar accessors (no classes, no methods and no properties defined)

Then, in docs, you see this:

Instead (as you'll see if you look at addressbook_pb2.py) it generates special descriptors for all your messages, enums, and fields, and some mysteriously empty classes, one for each message type

Which means:

You'll see descriptors, parsed from serialized .proto file
You'll see variables, named exactly the same as in .proto file

But these "variables" are nothing but GeneratedProtocolMessageType class, which is a metaclass, but for protocol messages

Then, you should look into docs for GeneratedProtocolMessageType from package google.protobuf.internal.python_message, where you should see this line:

Metaclass for protocol message classes created at runtime from Descriptors.

And this line means that you won't see any expected properties or methods while you code. Because these variables will become metaclasses for your protocol messages only at runtime! These metaclasses, at the time you look at the lines with their instantiation, are factories for your protocol messages

Moreover, this behavior is mentioned in the middle of the docs for that class:

The protocol compiler currently uses this metaclass to create protocol message classes at runtime.

It works only that way:

We add implementations for all methods described in the Message class. We also create properties to allow getting/setting all fields in the protocol message. Finally, we create slots to prevent users from accidentally "setting" nonexistent fields in the protocol message, which then wouldn't get serialized / deserialized properly.

This metaclass generates classes for your protocol messages, described in .proto files, using descriptors from generated files. Only at runtime (not at coding time)
And only at runtime you'll be able to use them as factories for your classes (those, which you expect to see in code) and, also, as a types for method parameters

So there is nothing wrong with Google Protobuf Documentation, it is not outdated

Collectives™ on Stack Overflow

protocol buffers in python: no classes generated

2 Answers 2

3 Comments

3 Comments

Your Answer

Post as a guest

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

3 Comments

3 Comments

Your Answer

Sign up or log in

Post as a guest

Post as a guest

Linked

Related