SemanticHighlighting encodeBase64 corrupts data

[alexzielenski]

Hey! I was tinkering around with clangd to use with a few modifications locally and found that the data transmitted via base64 is getting corrupted.

llvm-project/clang-tools-extra/clangd/SemanticHighlighting.cpp

Lines 289 to 315 in 82879c2

	std::string encodeBase64(const llvm::SmallVectorImpl<char> &Bytes) {
	static const char Table[] = "ABCDEFGHIJKLMNOPQRSTUVWXYZ"
	"abcdefghijklmnopqrstuvwxyz"
	"0123456789+/";
	std::string Res;
	size_t I;
	for (I = 0; I + 2 < Bytes.size(); I += 3) {
	uint32_t X = (Bytes[I] << 16) + (Bytes[I + 1] << 8) + Bytes[I + 2];
	Res += Table[(X >> 18) & 63];
	Res += Table[(X >> 12) & 63];
	Res += Table[(X >> 6) & 63];
	Res += Table[X & 63];
	}
	if (I + 1 == Bytes.size()) {
	uint32_t X = (Bytes[I] << 16);
	Res += Table[(X >> 18) & 63];
	Res += Table[(X >> 12) & 63];
	Res += "==";
	} else if (I + 2 == Bytes.size()) {
	uint32_t X = (Bytes[I] << 16) + (Bytes[I + 1] << 8);
	Res += Table[(X >> 18) & 63];
	Res += Table[(X >> 12) & 63];
	Res += Table[(X >> 6) & 63];
	Res += "=";
	}
	return Res;
	}

I replaced the built-in implementation with another found on GitHub and my issue was alleviated (modified to support llvm vector type):

https://github.com/ReneNyffenegger/cpp-base64/blob/a8aae956a2f07df9aac25b064cf4cd92d56aac45/base64.cpp#L35-L85

static const std::string base64_chars = 
             "ABCDEFGHIJKLMNOPQRSTUVWXYZ"
             "abcdefghijklmnopqrstuvwxyz"
             "0123456789+/";


static inline bool is_base64(unsigned char c) {
  return (isalnum(c) || (c == '+') || (c == '/'));
}

std::string encodeBase64cppBase64(const llvm::SmallVectorImpl<char> &Bytes) {
  std::string ret;
  int i = 0;
  int j = 0;
  unsigned char char_array_3[3];
  unsigned char char_array_4[4];

  unsigned in_len = Bytes.size();
  unsigned idx = 0;
  while (in_len--) {
    char_array_3[i++] = Bytes[idx++];
    if (i == 3) {
      char_array_4[0] = (char_array_3[0] & 0xfc) >> 2;
      char_array_4[1] = ((char_array_3[0] & 0x03) << 4) + ((char_array_3[1] & 0xf0) >> 4);
      char_array_4[2] = ((char_array_3[1] & 0x0f) << 2) + ((char_array_3[2] & 0xc0) >> 6);
      char_array_4[3] = char_array_3[2] & 0x3f;

      for(i = 0; (i <4) ; i++)
        ret += base64_chars[char_array_4[i]];
      i = 0;
    }
  }

  if (i)
  {
    for(j = i; j < 3; j++)
      char_array_3[j] = '\0';

    char_array_4[0] = ( char_array_3[0] & 0xfc) >> 2;
    char_array_4[1] = ((char_array_3[0] & 0x03) << 4) + ((char_array_3[1] & 0xf0) >> 4);
    char_array_4[2] = ((char_array_3[1] & 0x0f) << 2) + ((char_array_3[2] & 0xc0) >> 6);

    for (j = 0; (j < i + 1); j++)
      ret += base64_chars[char_array_4[j]];

    while((i++ < 3))
      ret += '=';

  }

  return ret;

}

Example:
I ran this code:

  llvm::SmallVector<char, 1> data;
  llvm::raw_svector_ostream OS(data);
  write32be(70, OS);
  write16be(8, OS);
  write16be(0x00EE, OS);
  log("encoded cpp-base64: {0}", encodeBase64cppBase64(data));
  log("encoded llvm: {0}", encodeBase64original(data));

output:

encoded cpp-base64: AAAARgAIAO4=
encoded llvm: AAAARgAI/+4=

Indeed, running the strings back through any random online decoder yields different results:

Using https://cryptii.com/pipes/base64-to-hex:

cpp-base64:

llvm encodeBase64:

Encoding also on this random site gives same results as cpp-base64:

[serge-sans-paille]

There's already a base64 implementation in llvm/lib/MC/WinCOFFObjectWriter.cpp too... we should probably syndicate that code in lib/Support, I'll have a look.

[serge-sans-paille]

Fixed by https://reviews.llvm.org/rG5a1958f2673f8c771e406a7e309e160b432c9a79