Serialize and deserialize protocol buffers

[kyleconroy]

As outlined in #447, TinyGo should support common serialization formats. While most people may think of JSON, I personally want TinyGo to support protocol buffers. This issue is meant to track my progress on adding Protocol Buffer support to TinyGo.

Design

The default Protocol Buffer package (https://pkg.go.dev/google.golang.org/protobuf) makes heavy use of reflection. TinyGo only supports a small amount of the reflect package. Heavy use of reflection leads to slow code, so the folks at PlanetScale wrote vtprotobuf, which generates explicit serialization / deserialization code. This code works today with TinyGo. However, it requires the existing, slow code to compile.

So the plan is simple: Support compiling the code generated by protoc-gen-go and running and of the code contained in init() functions. Use the code generated by vtprotobuf to serialize and deserialize the data.

Roadmap

• Compile code generated by https://github.com/planetscale/vtprotobuf
• Compile code generated by protoc-gen-go
  • feat: Add stubs for more of the reflect package #2624
  • (constant -4 of type uintptr) overflows uintptr #2806
• Import code generated by protoc-gen-go
  • unknown import: env::sync/atomic.StoreUint32 has not been defined #2652
  • Implement the PkgPath() method on reflect.Type #2668

[paralin]

@kyleconroy This is a great idea, I'm using vtprotobuf a lot currently and it works great. Having some kind of auto-transpiler that automatically converts the reflection-based protobuf-go to use vtprotobuf would be great.

Quick question: as of March 2023, it seems like most of the reflection features are supported - is there anything missing for the existing reflection version to work currently?

[dgryski]

And even more reflect is working, including the last bug on you roadmap.

[gitslav]

@kyleconroy @dgryski looking at the roadmap is it time for the confetti emoji?

[rockwotj]

So it seems that the default protobuf compiler is not supported still even with the improvements to reflection. I have yet to test vtprotobuf

% ./wasmtime-v14.0.4-x86_64-linux/wasmtime run protobuf-example.wasm
panic: unimplemented: (reflect.Type).MethodByName()
Error: failed to run main module `protobuf-example.wasm`

Caused by:
    0: failed to invoke command default
    1: error while executing at wasm backtrace:
           0: 0x5a15 - <unknown>!runtime._panic
           1: 0x5939 - <unknown>!(reflect.rawType).MethodByName
           2: 0x6484 - <unknown>!(*reflect.rawType).MethodByName
           3: 0x103253 - <unknown>!(*google.golang.org/protobuf/internal/impl.MessageInfo).makeStructInfo
           4: 0xb95e6 - <unknown>!(*google.golang.org/protobuf/internal/impl.MessageInfo).init
           5: 0xb8e02 - <unknown>!google.golang.org/protobuf/proto.protoMethods
           6: 0x34c67 - <unknown>!(google.golang.org/protobuf/proto.MarshalOptions).marshal
           7: 0x2d45a - <unknown>!runtime.run$1
           8: 0x2cf76 - <unknown>!runtime.run$1$gowrapper
           9:  0xdc1 - <unknown>!tinygo_launch
          10: 0x2ce48 - <unknown>!_start
       note: using the `WASMTIME_BACKTRACE_DETAILS=1` environment variable may show more debugging information
    2: wasm trap: wasm `unreachable` instruction executed

[rockwotj]

FWIW I have successfully used https://github.com/planetscale/vtprotobuf to marshal/unmarshal protobuf in tinygo -target=wasi

(this can be closed then?)

[wrnlb666]

How are you able to use vtprotobuf with tinygo? Isn't the generated code from vtprotobuf depends on the result of protobuf-go? I am still getting the same error with my _vtproto.pb.go file.

[rockwotj]

How are you able to use vtprotobuf with tinygo? Isn't the generated code from vtprotobuf depends on the result of protobuf-go?

The reflection methods are stubbed out, so it still compiles and the vtprotobuf version doesn't invoke the reflection methods, so it works fine for me.

I am still getting the same error with my _vtproto.pb.go file.

Post the error and make sure you're on the latest tinygo version

[wrnlb666]

I am on the latest tinygo v0.31.1, and I am also using it with wasmtime. The error I got is exactly the same with the one you posted before without using vtprotobuf. Did you change any of the code generated by protoc --go_out ?