Tinygo need serialization tools

[fractus-chain]

We really need serialization tools for tinygo, because tiny go can not use serialization tools just like coding/json encoding/gob in official golang.

[m-chichikalov]

Agree, having Google ProtoBuf is must have from the embeded world point of view... I didn't try to compile ProtoBuf with TinyGo so far.

@aykevl, maybe mark with issue as docs and help wanted and maybe someone will take over this?

[PaluMacil]

From the list of things that work and what they are blocked on, it looks like to enable a lot of them we need to get base64 working first before moving on to solve gob, etc. The error for base64 regarding branching on a const expression might be addressed in dev, according to a comment on #437 two days ago, so this is a good place to look. If this fixed base64, it probably unblocks work on other packages. Gob and json might benefit from the recent updates to reflection, so it's certainly something worth checking on dev so that we can get going with more encoders.

[deadprogram]

As of TinyGo v0.8.0 the encoding/base64 package still cannot be imported due to #437 so that is still not fixed.

[rubenv]

As of TinyGo v0.8.0 the encoding/base64 package still cannot be imported due to #437 so that is still not fixed.

encoding/base64 seems to be supported now, but that's still not sufficient for ProtoBuf support. The culprit now being encoding/json:

# encoding/json                                                                                                                                                                                                                                                                                     
../../../../../../../../usr/lib/golang/src/encoding/json/encode.go:346:23: Map not declared by package sync                                       
../../../../../../../../usr/lib/golang/src/encoding/json/encode.go:1264:21: Map not declared by package sync                                      
../../../../../../../../usr/lib/golang/src/encoding/json/encode.go:1117:19: sf.Tag.Get undefined (type string has no field or method Get)         
../../../../../../../../usr/lib/golang/src/encoding/json/encode.go:1130:11: ft.Name undefined (type reflect.Type has no field or method Name)     
../../../../../../../../usr/lib/golang/src/encoding/json/encode.go:1187:41: ft.Name undefined (type reflect.Type has no field or method Name)     
../../../../../../../../usr/lib/golang/src/encoding/json/encode.go:767:16: PtrTo not declared by package reflect                                  
../../../../../../../../usr/lib/golang/src/encoding/json/encode.go:708:11: t.Key undefined (type reflect.Type has no field or method Key)         
../../../../../../../../usr/lib/golang/src/encoding/json/encode.go:713:9: t.Key undefined (type reflect.Type has no field or method Key)          
../../../../../../../../usr/lib/golang/src/encoding/json/encode.go:392:7: t.Implements undefined (type reflect.Type has no field or method Implements)
../../../../../../../../usr/lib/golang/src/encoding/json/encode.go:396:14: PtrTo not declared by package reflect                                  
../../../../../../../../usr/lib/golang/src/encoding/json/encode.go:401:7: t.Implements undefined (type reflect.Type has no field or method Implements)                                                                                                                                              
../../../../../../../../usr/lib/golang/src/encoding/json/encode.go:405:14: PtrTo not declared by package reflect                                  
../../../../../../../../usr/lib/golang/src/encoding/json/encode.go:365:11: WaitGroup not declared by package sync                                                                                                                                                                                   
../../../../../../../../usr/lib/golang/src/encoding/json/decode.go:916:9: v.NumMethod undefined (type reflect.Value has no field or method NumMethod)
../../../../../../../../usr/lib/golang/src/encoding/json/decode.go:945:6: v.SetBytes undefined (type reflect.Value has no field or method SetBytes)
../../../../../../../../usr/lib/golang/src/encoding/json/decode.go:949:9: v.NumMethod undefined (type reflect.Value has no field or method NumMethod)                                                                                                                                               
../../../../../../../../usr/lib/golang/src/encoding/json/decode.go:983:9: v.NumMethod undefined (type reflect.Value has no field or method NumMethod)                                                                                                                                               
../../../../../../../../usr/lib/golang/src/encoding/json/decode.go:991:23: v.OverflowInt undefined (type reflect.Value has no field or method OverflowInt)                                                                                                                                          
../../../../../../../../usr/lib/golang/src/encoding/json/decode.go:999:23: v.OverflowUint undefined (type reflect.Value has no field or method OverflowUint)                                                                                                                                        
../../../../../../../../usr/lib/golang/src/encoding/json/decode.go:1007:23: v.OverflowFloat undefined (type reflect.Value has no field or method OverflowFloat)                                                                                                                                     
../../../../../../../../usr/lib/golang/src/encoding/json/decode.go:622:40: v.NumMethod undefined (type reflect.Value has no field or method NumMethod)                                                                                                                                              
../../../../../../../../usr/lib/golang/src/encoding/json/decode.go:638:12: t.Key undefined (type reflect.Type has no field or method Key)                                                                                                                                                           
../../../../../../../../usr/lib/golang/src/encoding/json/decode.go:643:16: PtrTo not declared by package reflect                                                                                                                                                                                    
../../../../../../../../usr/lib/golang/src/encoding/json/decode.go:643:24: t.Key undefined (type reflect.Type has no field or method Key)         
../../../../../../../../usr/lib/golang/src/encoding/json/decode.go:650:18: MakeMap not declared by package reflect                                
../../../../../../../../usr/lib/golang/src/encoding/json/decode.go:771:12: t.Key undefined (type reflect.Type has no field or method Key)         
../../../../../../../../usr/lib/golang/src/encoding/json/decode.go:775:31: reflect.ValueOf(key).Convert undefined (type reflect.Value has no field or method Convert)
../../../../../../../../usr/lib/golang/src/encoding/json/decode.go:776:17: PtrTo not declared by package reflect                                  
../../../../../../../../usr/lib/golang/src/encoding/json/decode.go:787:40: reflect.Zero(kt).OverflowInt undefined (type reflect.Value has no field or method OverflowInt)                                                                                                                           
../../../../../../../../usr/lib/golang/src/encoding/json/decode.go:791:30: reflect.ValueOf(n).Convert undefined (type reflect.Value has no field or method Convert)                                                                                                                                 
../../../../../../../../usr/lib/golang/src/encoding/json/decode.go:795:40: reflect.Zero(kt).OverflowUint undefined (type reflect.Value has no field or method OverflowUint)                                                                                                                         
../../../../../../../../usr/lib/golang/src/encoding/json/decode.go:799:30: reflect.ValueOf(n).Convert undefined (type reflect.Value has no field or method Convert)                                                                                                                                 
../../../../../../../../usr/lib/golang/src/encoding/json/decode.go:805:7: v.SetMapIndex undefined (type reflect.Value has no field or method SetMapIndex)                                                                                                                                           
../../../../../../../../usr/lib/golang/src/encoding/json/decode.go:517:8: v.NumMethod undefined (type reflect.Value has no field or method NumMethod)                                                                                                                                               
../../../../../../../../usr/lib/golang/src/encoding/json/decode.go:550:13: Copy not declared by package reflect                                                                                                                                                                                     
../../../../../../../../usr/lib/golang/src/encoding/json/decode.go:554:7: v.SetLen undefined (type reflect.Value has no field or method SetLen)                                                                                                                                                     
../../../../../../../../usr/lib/golang/src/encoding/json/decode.go:591:6: v.SetLen undefined (type reflect.Value has no field or method SetLen)                                                                                                                                                     
../../../../../../../../usr/lib/golang/src/encoding/json/decode.go:450:41: v.Type().Name undefined (type reflect.Type has no field or method Name) 
../../../../../../../../usr/lib/golang/src/encoding/json/decode.go:476:15: v.Type().NumMethod undefined (type reflect.Type has no field or method NumMethod)
../../../../../../../../usr/lib/golang/src/encoding/json/decode.go:306:30: cannot convert nil (untyped nil value) to reflect.Type                                                                                                                                                                   
../../../../../../../../usr/lib/golang/src/encoding/json/decode.go:309:39: d.errorContext.Struct.Name undefined (type reflect.Type has no field or method Name)                                                                                                                                     
../../../../../../../../usr/lib/golang/src/encoding/json/decode.go:291:26: cannot convert nil (untyped nil value) to reflect.Type        
../../../../../../../../usr/lib/golang/src/encoding/json/decode.go:159:15: cannot convert nil (untyped nil value) to reflect.Type   

Still, progress, thanks to everyone involved!

[PaluMacil]

The 0.9 release gave us base32, base64, and csv. We'll probably need additional reflection support before json, xml, and gob. I expect ProtoBuf will have other things it needs. Great progress!

[Gregory-Ledray]

It's not necessary to use encoding/json to parse and encode JSON in Go. This library compiles on tinygo with -target wasm: https://github.com/buger/jsonparser

This is possible because it import graph is really small - the only thing that imports 'reflect' is its fmt dependency: https://godoc.org/github.com/buger/jsonparser?import-graph

I wasn't able to test jsonparser because I'd like to use it to handle a HTTP request response, but alas I don't see any examples for passing values from JS to exported WASM functions, even though I can call them in the JS callback.

[PaluMacil]

This is a good tip. While most people using JSON probably don't want the tradeoffs in jsonparser, I'd be willing to bet most users of tinygo would want those tradeoffs, at least if they are using tinygo for tiny machines.

[benma]

for golang/protobuf, apart from the json issue mention above, there are the following other issues:

../../vendor/github.com/golang/protobuf/proto/extensions.go:62:46: Locker not declared by package sync
../../vendor/github.com/golang/protobuf/proto/extensions.go:82:71: Locker not declared by package sync
../../vendor/github.com/golang/protobuf/proto/extensions.go:155:78: Locker not declared by package sync
../../vendor/github.com/golang/protobuf/proto/text_parser.go:765:20: Append not declared by package reflect
../../vendor/github.com/golang/protobuf/proto/text_parser.go:785:18: Append not declared by package reflect
../../vendor/github.com/golang/protobuf/proto/text_parser.go:469:14: cannot convert nil (untyped nil value) to reflect.Type
../../vendor/github.com/golang/protobuf/proto/text_parser.go:506:8: sv.FieldByName undefined (type reflect.Value has no field or method FieldByName)
../../vendor/github.com/golang/protobuf/proto/text_parser.go:507:8: sv.FieldByName undefined (type reflect.Value has no field or method FieldByName)
../../vendor/github.com/golang/protobuf/proto/text_parser.go:561:18: Append not declared by package reflect
../../vendor/github.com/golang/protobuf/proto/text_parser.go:599:21: MakeMap not declared by package reflect
../../vendor/github.com/golang/protobuf/proto/text_parser.go:601:34: dst.Type().Key undefined (type reflect.Type has no field or method Key)
../../vendor/github.com/golang/protobuf/proto/text_parser.go:655:8: dst.SetMapIndex undefined (type reflect.Value has no field or method SetMapIndex)
../../vendor/github.com/golang/protobuf/proto/text.go:404:23: st.Field(i).Tag.Get undefined (type string has no field or method Get)
../../vendor/github.com/golang/protobuf/proto/text.go:412:38: inner.Type().Field(0).Tag.Get undefined (type string has no field or method Get)
../../vendor/github.com/golang/protobuf/proto/text.go:208:13: sv.FieldByName undefined (type reflect.Value has no field or method FieldByName)
../../vendor/github.com/golang/protobuf/proto/text.go:209:12: sv.FieldByName undefined (type reflect.Value has no field or method FieldByName)
../../vendor/github.com/golang/protobuf/proto/text.go:221:11: cannot convert nil (untyped nil value) to reflect.Type
../../vendor/github.com/golang/protobuf/proto/table_unmarshal.go:1736:10: t.Key undefined (type reflect.Type has no field or method Key)
../../vendor/github.com/golang/protobuf/proto/table_unmarshal.go:1738:44: f.Tag.Get undefined (type string has no field or method Get)
../../vendor/github.com/golang/protobuf/proto/table_unmarshal.go:1739:44: f.Tag.Get undefined (type string has no field or method Get)
../../vendor/github.com/golang/protobuf/proto/table_unmarshal.go:1799:18: MakeMap not declared by package reflect
../../vendor/github.com/golang/protobuf/proto/table_unmarshal.go:1803:5: m.SetMapIndex undefined (type reflect.Value has no field or method SetMapIndex)
../../vendor/github.com/golang/protobuf/proto/table_unmarshal.go:488:57: t.Name undefined (type reflect.Type has no field or method Name)
../../vendor/github.com/golang/protobuf/proto/table_unmarshal.go:610:47: t.Name undefined (type reflect.Type has no field or method Name)
../../vendor/github.com/golang/protobuf/proto/table_unmarshal.go:613:46: t.Name undefined (type reflect.Type has no field or method Name)
../../vendor/github.com/golang/protobuf/proto/table_unmarshal.go:454:39: f.Tag.Get undefined (type string has no field or method Get)
../../vendor/github.com/golang/protobuf/proto/table_unmarshal.go:294:60: f.Type.Name undefined (type reflect.Type has no field or method Name)
../../vendor/github.com/golang/protobuf/proto/table_unmarshal.go:302:66: f.Type.Name undefined (type reflect.Type has no field or method Name)
../../vendor/github.com/golang/protobuf/proto/table_unmarshal.go:305:13: f.Tag.Get undefined (type string has no field or method Get)
../../vendor/github.com/golang/protobuf/proto/table_unmarshal.go:313:58: f.Type.Name undefined (type reflect.Type has no field or method Name)
../../vendor/github.com/golang/protobuf/proto/table_unmarshal.go:322:18: f.Tag.Get undefined (type string has no field or method Get)
../../vendor/github.com/golang/protobuf/proto/table_unmarshal.go:329:17: f.Tag.Get undefined (type string has no field or method Get)
../../vendor/github.com/golang/protobuf/proto/table_unmarshal.go:332:51: t.Name undefined (type reflect.Type has no field or method Name)
../../vendor/github.com/golang/protobuf/proto/table_unmarshal.go:366:35: PtrTo not declared by package reflect
../../vendor/github.com/golang/protobuf/proto/table_unmarshal.go:378:31: f.Tag.Get undefined (type string has no field or method Get)
../../vendor/github.com/golang/protobuf/proto/table_unmarshal.go:406:29: PtrTo not declared by package reflect
../../vendor/github.com/golang/protobuf/proto/table_unmarshal.go:406:39: reflect.Zero(reflect.PtrTo(t)).MethodByName undefined (type reflect.Value has no field or method MethodByName)
../../vendor/github.com/golang/protobuf/proto/table_unmarshal.go:409:71: t.Name undefined (type reflect.Type has no field or method Name)
../../vendor/github.com/golang/protobuf/proto/table_unmarshal.go:190:41: PtrTo not declared by package reflect
../../vendor/github.com/golang/protobuf/proto/table_merge.go:212:59: tf.Name undefined (type reflect.Type has no field or method Name)
../../vendor/github.com/golang/protobuf/proto/table_merge.go:500:53: tf.Name undefined (type reflect.Type has no field or method Name)
../../vendor/github.com/golang/protobuf/proto/table_merge.go:502:58: tf.Name undefined (type reflect.Type has no field or method Name)
../../vendor/github.com/golang/protobuf/proto/table_merge.go:572:55: tf.Name undefined (type reflect.Type has no field or method Name)
../../vendor/github.com/golang/protobuf/proto/table_merge.go:581:22: MakeMap not declared by package reflect
../../vendor/github.com/golang/protobuf/proto/table_merge.go:589:11: dm.SetMapIndex undefined (type reflect.Value has no field or method SetMapIndex)
../../vendor/github.com/golang/protobuf/proto/table_merge.go:595:11: dm.SetMapIndex undefined (type reflect.Value has no field or method SetMapIndex)
../../vendor/github.com/golang/protobuf/proto/table_merge.go:600:11: dm.SetMapIndex undefined (type reflect.Value has no field or method SetMapIndex)
../../vendor/github.com/golang/protobuf/proto/table_merge.go:609:61: tf.Name undefined (type reflect.Type has no field or method Name)
../../vendor/github.com/golang/protobuf/proto/table_merge.go:646:16: t.FieldByName undefined (type reflect.Type has no field or method FieldByName)
../../vendor/github.com/golang/protobuf/proto/table_marshal.go:2282:15: t.Key undefined (type reflect.Type has no field or method Key)
../../vendor/github.com/golang/protobuf/proto/table_marshal.go:2284:33: f.Tag.Get undefined (type string has no field or method Get)
../../vendor/github.com/golang/protobuf/proto/table_marshal.go:2285:33: f.Tag.Get undefined (type string has no field or method Get)
../../vendor/github.com/golang/protobuf/proto/table_marshal.go:476:32: sf.Tag.Get undefined (type string has no field or method Get)
../../vendor/github.com/golang/protobuf/proto/table_marshal.go:446:30: f.Tag.Get undefined (type string has no field or method Get)
../../vendor/github.com/golang/protobuf/proto/table_marshal.go:420:15: PtrTo not declared by package reflect
../../vendor/github.com/golang/protobuf/proto/table_marshal.go:316:13: PtrTo not declared by package reflect
../../vendor/github.com/golang/protobuf/proto/table_marshal.go:324:35: PtrTo not declared by package reflect
../../vendor/github.com/golang/protobuf/proto/table_marshal.go:346:25: f.Tag.Get undefined (type string has no field or method Get)
../../vendor/github.com/golang/protobuf/proto/table_marshal.go:370:12: f.Tag.Get undefined (type string has no field or method Get)
../../vendor/github.com/golang/protobuf/proto/table_marshal.go:374:12: f.Tag.Get undefined (type string has no field or method Get)
../../vendor/github.com/golang/protobuf/proto/table_marshal.go:285:40: PtrTo not declared by package reflect
../../vendor/github.com/golang/protobuf/proto/properties.go:372:30: f.Tag.Get undefined (type string has no field or method Get)
../../vendor/github.com/golang/protobuf/proto/properties.go:374:18: f.Tag.Get undefined (type string has no field or method Get)
../../vendor/github.com/golang/protobuf/proto/properties.go:394:35: PtrTo not declared by package reflect
../../vendor/github.com/golang/protobuf/proto/properties.go:410:27: sft.Tag.Get undefined (type string has no field or method Get)
../../vendor/github.com/golang/protobuf/proto/properties.go:278:29: PtrTo not declared by package reflect
../../vendor/github.com/golang/protobuf/proto/properties.go:278:43: p.mtype.Key undefined (type reflect.Type has no field or method Key)
../../vendor/github.com/golang/protobuf/proto/properties.go:278:64: f.Tag.Get undefined (type string has no field or method Get)
../../vendor/github.com/golang/protobuf/proto/properties.go:284:20: PtrTo not declared by package reflect
../../vendor/github.com/golang/protobuf/proto/properties.go:286:43: f.Tag.Get undefined (type string has no field or method Get)
../../vendor/github.com/golang/protobuf/proto/properties.go:289:16: cannot convert nil (untyped nil value) to reflect.Type
../../vendor/github.com/golang/protobuf/proto/pointer_reflect.go:300:15: Append not declared by package reflect
../../vendor/github.com/golang/protobuf/proto/pointer_reflect.go:159:37: reflect.ValueOf(&v).Convert undefined (type reflect.Value has no field or method Convert)
../../vendor/github.com/golang/protobuf/proto/pointer_reflect.go:152:20: p.v.Elem().Convert undefined (type reflect.Value has no field or method Convert)
../../vendor/github.com/golang/protobuf/proto/pointer_reflect.go:133:13: p.v.Convert undefined (type reflect.Value has no field or method Convert)
../../vendor/github.com/golang/protobuf/proto/pointer_reflect.go:113:5: s.SetLen undefined (type reflect.Value has no field or method SetLen)
../../vendor/github.com/golang/protobuf/proto/pointer_reflect.go:115:17: Append not declared by package reflect
../../vendor/github.com/golang/protobuf/proto/pointer_reflect.go:100:31: p.v.Elem().FieldByIndex undefined (type reflect.Value has no field or method FieldByIndex)
../../vendor/github.com/golang/protobuf/proto/pointer_reflect.go:54:11: f.Index undefined (type *reflect.StructField has no field or method Index)
../../vendor/github.com/golang/protobuf/proto/equal.go:119:15: v1.FieldByName undefined (type reflect.Value has no field or method FieldByName)
../../vendor/github.com/golang/protobuf/proto/equal.go:120:13: v2.FieldByName undefined (type reflect.Value has no field or method FieldByName)
../../vendor/github.com/golang/protobuf/proto/equal.go:126:15: v1.FieldByName undefined (type reflect.Value has no field or method FieldByName)
../../vendor/github.com/golang/protobuf/proto/equal.go:127:13: v2.FieldByName undefined (type reflect.Value has no field or method FieldByName)
../../vendor/github.com/golang/protobuf/proto/equal.go:133:11: v1.FieldByName undefined (type reflect.Value has no field or method FieldByName)
../../vendor/github.com/golang/protobuf/proto/equal.go:139:11: v2.FieldByName undefined (type reflect.Value has no field or method FieldByName)
../../vendor/github.com/golang/protobuf/proto/discard.go:318:29: f.Tag.Get undefined (type string has no field or method Get)
../../vendor/github.com/golang/protobuf/proto/discard.go:332:13: v.FieldByName undefined (type reflect.Value has no field or method FieldByName)
../../vendor/github.com/golang/protobuf/proto/discard.go:244:16: t.FieldByName undefined (type reflect.Type has no field or method FieldByName)
../../vendor/github.com/golang/protobuf/proto/clone.go:169:20: MakeMap not declared by package reflect
../../vendor/github.com/golang/protobuf/proto/clone.go:185:8: out.SetMapIndex undefined (type reflect.Value has no field or method SetMapIndex)
../../vendor/github.com/golang/protobuf/proto/clone.go:212:8: out.SetBytes undefined (type reflect.Value has no field or method SetBytes)
../../vendor/github.com/golang/protobuf/proto/clone.go:222:20: AppendSlice not declared by package reflect
../../vendor/github.com/golang/protobuf/proto/clone.go:227:21: Append not declared by package reflect
../../vendor/github.com/golang/protobuf/proto/clone.go:122:11: in.FieldByName undefined (type reflect.Value has no field or method FieldByName)
../../vendor/github.com/golang/protobuf/proto/clone.go:128:7: out.FieldByName undefined (type reflect.Value has no field or method FieldByName)

[elliots]

For anyone else trying to get protobuf to work... it can. Sort of.

If you use gogoproto, you can have it generate marshal and unmarshal functions that work on the bytes, without reflection.

protoc --gogofaster_out=. *.proto

Then, you "just" need to stub the proto package with something like this

package proto

import "errors"

const Marshal = 0
const GoGoProtoPackageIsVersion3 = 0

type Message interface {
}

type InternalMessageInfo interface {
	DiscardUnknown(interface{})
	Merge(interface{}, interface{})
	Marshal(interface{}, interface{}, interface{}) ([]byte, error)
}

func RegisterFile(string, []byte) {}

func RegisterType(interface{}, string) {}

func CompactTextString(interface{}) string {
	return "nope"
}

You can then call Marshal and Unmarshal directly.

[jsimnz]

Just wanted to give a heads up. My tests have found that https://github.com/valyala/fastjson compiles successfully and runs with the TinyGo compiler targeting WASM/WASI.