Revise StringUtils.parseLocale(String) for proper handling of corner cases [SPR-16651]

[spring-projects-issues]

Michael Decker opened SPR-16651 and commented

We are using Spring Boot 2.0.0.RELEASE (jdk1.8.0_144, Windows 10)

We had to realize, that org.springframework.util.StringUtils.parseLocaleString(String) does not return always the same instance, as it was created from:

from '' (und) to 'null (null)'
from 'sr_BA_#Latn' (sr-Latn-BA) to 'sr_BA_#Latn (sr-BA)'
from 'ja_JP_JP_#u-ca-japanese' (ja-JP-u-ca-japanese-x-lvariant-JP) to 'ja_JP_JP_#u-ca-japanese (ja-JP-x-lvariant-JP)'
from 'sr_ME_#Latn' (sr-Latn-ME) to 'sr_ME_#Latn (sr-ME)'
from 'sr__#Latn' (sr-Latn) to 'sr_#LATN (sr)'
from 'th_TH_TH_#u-nu-thai' (th-TH-u-nu-thai-x-lvariant-TH) to 'th_TH_TH_#u-nu-thai (th-TH-x-lvariant-TH)'
from 'sr_RS_#Latn' (sr-Latn-RS) to 'sr_RS_#Latn (sr-RS)'

The test is:

import static org.junit.Assert.assertEquals;
import static org.springframework.util.StringUtils.parseLocale;
import static org.springframework.util.StringUtils.parseLocaleString;

import java.util.List;
import java.util.Locale;
import java.util.stream.Collectors;
import java.util.stream.Stream;

import org.junit.Test;

public class SpringStringUtilsParseLocaleTest {

  @Test
  public void testParseLocale() {
    List<String> failing = Stream.of(Locale.getAvailableLocales()) //
        .filter(locale -> !locale.equals(parseLocale(locale.toString()))) //
        .map(locale -> String.format("from '%s' (%s) to '%s (%s)'", locale, toLanguageTag(locale),
            parseLocale(locale.toString()), toLanguageTag(parseLocale(locale.toString())))) //
        .collect(Collectors.toList());

    // assertEquals(Collections.emptyList(), failing);
    assertEquals("", failing.stream().collect(Collectors.joining("\n")));
  }

  @Test
  public void testParseLocaleString() {
    List<String> failing = Stream.of(Locale.getAvailableLocales()) //
        .filter(locale -> !locale.equals(parseLocaleString(locale.toString()))) //
        .map(locale -> String.format("from '%s' (%s) to '%s (%s)'", locale, toLanguageTag(locale),
            parseLocaleString(locale.toString()), toLanguageTag(parseLocaleString(locale.toString())))) //
        .collect(Collectors.toList());

    // assertEquals(Collections.emptyList(), failing);
    assertEquals("", failing.stream().collect(Collectors.joining("\n")));
  }

  private static String toLanguageTag(Locale locale) {
    return locale == null ? null : locale.toLanguageTag();
  }

}

---

Affects: 5.0.4

Issue Links:

• StringUtils.parseLocaleString(): does not parse locale string in java 7 for Serbian (Latin) [SPR-14718] #19283 StringUtils.parseLocaleString(): does not parse locale string in java 7 for Serbian (Latin)
• StringUtils#parseLocaleString(String) does not work for empty locale [SPR-6562] #11228 StringUtils#parseLocaleString(String) does not work for empty locale
• StringUtils#parseLocaleString(String) with Variant when no Country [SPR-7598] #12254 StringUtils#parseLocaleString(String) with Variant when no Country
• StringToLocaleConverter should allow BCP 47 values [SPR-16188] #20736 StringToLocaleConverter should allow BCP 47 values
• CookieLocaleResolver is not RFC6265 compliant when setting a locale and time zone [SPR-16700] #21241 CookieLocaleResolver is not RFC6265 compliant when setting a locale and time zone

Referenced from: commits cab35aa, 55563c1, 88e4006

[spring-projects-issues]

Juergen Hoeller commented

As far as I can tell, these language tag mismatches are due to how the variant gets parsed: The platform Locale instances as returned from Locale.getAvailableLocales() feature a BCP 47 'script' part for certain variants whereas our parsed Locale instances expose a general variant part there. The public ways of constructing a Locale instance only support general variant parts (you cannot specify a dedicated 'script' part there which is also documented in the Locale javadoc), so it looks like this is a known limitation at the core JDK level and an unavoidable consequence from java.util.Locale's design compromises for BCP 47 support in Java 7+.

If you'd like consistent language tag exposure, you could parse language tags rather than toString output, e.g. switching to locale.toLanguageTag() as input for StringUtils.parseLocale. The previous mismatches disappear then, only a single new one appears for a variant that cannot be expressed in a language tag. Or you could parse toString output but also compare the resulting toString for equality then, not the Locale instance itself... in which case the problems are also reduced to a single failing case (which I'm currently investigating).

[spring-projects-issues]

Michael Decker commented

Okay. The issue is, the most serializer (e.g. jackson) are using just the toString method.

I tried also to use run the test with toLanguageTag and it fails in one case: org.junit.ComparisonFailure: expected:<[]> but was:<[from 'no_NO_NY' (nn-NO) to 'nn_NO (nn-NO)']>
I guess, that is the case, you mentioned:

  @Test
  public void testParseLocale() {
    List<String> failing = Stream.of(Locale.getAvailableLocales()) //
        .filter(locale -> !locale.equals(parseLocale(locale.toLanguageTag()))) //
        .map(locale -> String.format("from '%s' (%s) to '%s (%s)'", locale, toLanguageTag(locale),
            parseLocale(locale.toLanguageTag()), toLanguageTag(parseLocale(locale.toLanguageTag())))) //
        .collect(Collectors.toList());
    // assertEquals(Collections.emptyList(), failing);
    assertEquals("", failing.stream().collect(Collectors.joining("\n")));
  }

[spring-projects-issues]

Juergen Hoeller commented

With #12254 addressed, parseLocaleString can parse all available locales on the JVM now, with the goal of Locale.toString() equality. 5.0.4's parseLocale method can parse all available locales via Locale.toLanguageTag() equality, and it also joins parseLocaleString in its rejection of invalid locales and its handling of the empty locale.