field "cc.arduino.cli.commands.Library.Maintainer" contains invalid UTF-8

[JAndrassy]

Describe the problem

Library Manager doesn't show libraries. Output console shows [INFO] ERROR: 2019/10/21 21:46:10 grpc: server failed to encode response: rpc error: code = Internal desc = grpc: error while marshaling: proto: field "cc.arduino.cli.commands.Library.Maintainer" contains invalid UTF-8

Operating system

• Linux
• Windows

Operating system version

• Linux Mint 18.3
• Windows 11

Additional context

Additional reports

• field "cc.arduino.cli.commands.Library.Maintainer" contains invalid UTF-8 #1907 (comment)
• https://forum.arduino.cc/t/timerfive-library-breaks-library-manager-of-ardiuno-ide-2-0/859764
• https://forum.arduino.cc/t/library-manager-is-empty/1297756
• https://forum.arduino.cc/t/library-manager-is-empty/1002124
• https://forum.arduino.cc/t/library-manager-is-empty2/1297757
• https://forum.arduino.cc/t/ide-2-0-0-libraries-problem/1042463
• https://forum.arduino.cc/t/libraries-dont-appear-in-the-library-manager/1050903

[JAndrassy]

can this issue be migrated to the arduino-cli repository?

[spoenemann]

Do you have the arduino-cli already installed? If yes, which version? (Try arduino-cli version in a terminal)

[JAndrassy]

I didn't install arduino-cli. Doesn't v0.0.3-alpha.preview have cli bundled?

[spoenemann]

Yes, but if it's found in the global PATH, that version is taken (see arduino/arduino-pro-ide#142).

[JAndrassy]

duro@nuc ~ $ arduino-cli version
arduino-cli: command not found

duro@nuc ~/Arduino Pro IDE-v0.0.3-linux $ ./arduino-pro-ide 
root WARN please install @theia/electron@0.14.0-next.0159cd5b as a runtime dependency
root INFO Theia app listening on http://localhost:42569.
daemon INFO >>> Starting arduino-cli version: 0.6.0 commit: 2049a7a daemon from /home/duro/Arduino Pro IDE-v0.0.3-linux/resources/app/node_modules/arduino-ide-extension/build/arduino-cli...

[per1234]

Hi @JAndrassy. Thanks so much for taking the time to report this!

Would you mind downloading the new 0.1.4 release of Arduino Pro IDE or and letting us know whether the bug still occurs?

From the message, I suspect that the issue is specific to the contents of one of the package index files provided by the Boards Manager URLs. Do you remember if you were using any custom URLs, or did you just have the default configuration, which only provides Boards Manager with the official Arduino package index?

[JAndrassy]

Hi @JAndrassy. Thanks so much for taking the time to report this!

Would you mind downloading the new 0.1.4 release of Arduino Pro IDE or and letting us know whether the bug still occurs?

From the message, I suspect that the issue is specific to the contents of one of the package index files provided by the Boards Manager URLs. Do you remember if you were using any custom URLs, or did you just have the default configuration, which only provides Boards Manager with the official Arduino package index?

sorry. Per. I don't have time now for this. the problem then was my name in my published library. I modified library.properties for next version, so now library.properties of my libraries doesn't have UTF character

[per1234]

Thats enough for me (and yeah, I'm dumb for saying it was a package index). I found the culprit here (ANSI encoding):
https://github.com/jandrassy/arduino-library-wifilink/blob/45dda6563bcf9d5c35015beef93054cc1bf8af7a/library.properties#L4

maintainer=Juraj Andrássy <juraj.andrassy@gmail.com>

Steps to reproduce:

1. Run this command from your Arduino libraries folder (or download the equivalent zip archive and install if you prefer):

   git clone https://github.com/jandrassy/arduino-library-wifilink && cd arduino-library-wifilink && git checkout 45dda6563bcf9d5c35015beef93054cc1bf8af7a
   

2. Compile a sketch that uses the library:

   #include <WiFiLink.h>
   void setup() {}
   void loop() {}

The compilation process fails:

Sketch uses 2182 bytes (0%) of program storage space. Maximum is 253952 bytes.
Global variables use 226 bytes (2%) of dynamic memory, leaving 7966 bytes for local variables. Maximum is 8192 bytes.
Compilation error: Error: 13 INTERNAL: grpc: error while marshaling: string field contains invalid UTF-8

You can see that the code actually compiled just fine, but this error will block uploading.

The issue does not occur when doing the same with the classic Arduino IDE or Arduino CLI.

[ubidefeo]

@per1234 what's your advice on this?
should we force CLI to accept other encodings?
If so, let's turn it into a task

[per1234]

In order to fully form an opinion, I would need to understand two things:

• How much complexity would need to be added to the code to support them
• What are the use cases where non-UTF-8 encodings are beneficial

If it can be done without significantly harming the maintainability of the code, I'm all for it. Even if we document a UTF-8 requirement in the specifications, people are still going to end up inadvertently using other encodings, so there will always be a benefit to supporting them. But if this is something difficult to support, then I would want to be able to understand better the cost vs benefit balance.

Maybe @silvanocerza or @kittaakos would be able to provide some insight into what would we needed to support other encodings.

Maybe @JAndrassy would be able to provide some insight into the use case for non-UTF-8 encoding.

[JAndrassy]

Normally a .properties file should be ISO 8859-1 encoding (Latin-1) and non-Latin-1 characters should be entered by using Unicode escape characters. I use Eclipse IDE and it automatically uses 8859-1 for .properties files.

[kittaakos]

People suggest using bytes instead of string if the data may contain non-UTF-8 characters: protocolbuffers/protobuf#4970 (comment)

What do you think @cmaglie?