Add MediaType.APPLICATION_JSON_UTF8 and improve documentation [SPR-13600]

[spring-projects-issues]

Manuel Jordan opened SPR-13600 and commented

Hello

I have the following declared:

@JsonPropertyOrder(value={"id","nombre","apellido","fecha"})
public class PersonaJson implements Serializable {
	
	private static final long serialVersionUID = 7212492147213027538L;
	
	private String id;
	private String nombre;
	private String apellido;
	
	private Date fecha;

	public PersonaJson(){

	}

	@NotNull(message="{field.null}")
	@JsonProperty(value="id")
	public String getId() {
		return id;
	}
	public void setId(String id) {
		this.id = id;
	}
	
	@NotNull(message="{field.null}")
	@Size(min=5, max=20, message="{persona.nombre.size}")
	@JsonProperty(value="nombre")
	public String getNombre() {
		return nombre;
	}

With:

@Bean
   public MappingJackson2HttpMessageConverter mappingJackson2HttpMessageConverter(){
            MappingJackson2HttpMessageConverter converter = new MappingJackson2HttpMessageConverter();
                                                                                            converter.setPrettyPrint(true);		
            return converter;
   }

And with

@EnableWebMvc
@Configuration
public class WebMvcConfig extends WebMvcConfigurerAdapter {

       @Autowired
	private HttpMessageConverterConfig httpMessageConverterConfig;
….

        @Override
	public void configureMessageConverters(List<HttpMessageConverter<?>> converters) {
	    converters.add(httpMessageConverterConfig.mappingJackson2HttpMessageConverter());
	}

The data generated from the server is declared in this way (testing purpose)

public static PersonaJson crearPersonaMix(){
		PersonaJson persona = new PersonaJson();
		persona.setId("100");
		persona.setNombre("Jesús Você");
		persona.setApellido("Mão Nuñez");
		persona.setFecha(new Date());
		return persona;
	}

Note: Observe these characters ú ê ã ñ

I have two @Controller's

For One

@Controller
@RequestMapping(value="….", method=RequestMethod.GET)
public class PersonaJsonFindOneController {

@RequestMapping(value="….")
	public @ResponseBody PersonaJson findOneRequestParamById(@RequestParam String id){
		logger.info("findOneRequestParamById called...");
		return personas.get(id);
	}

Observe there is no a produces=MediaType.APPLICATION_JSON_VALUE

When I use the URL in any browser Safari, FF, Chrome, Opera I can see the data and the "weird" characters in peace.
It how 02.png

For Two

@Controller
@RequestMapping(value="...", 
			    method=RequestMethod.GET, 
			    produces=MediaType.APPLICATION_JSON_VALUE)
public class PersonaJsonFindOneWithProducesController

@RequestMapping(value="….")
	public @ResponseBody PersonaJson findOneRequestParamById(@RequestParam String id){
		logger.info("findOneRequestParamById called...");
		return personas.get(id);
	}

Observe now there is a produces=MediaType.APPLICATION_JSON_VALUE

When I use the URL (the other for that controller) in any browser Safari, FF, Chrome, Opera I can't see the data and the "weird" characters in peace. It how 01.png

In FF if I enable JSONView I can see now all ok… It how 02.png

Conclusion: Just thinking I need just install that add-on to each web browser (of course using an equivalent).

For Opera: I just used

• JsonViewer

For Chrome:

• JSONView same creator for FF now for Chrome

or (not both)

• JSON Viewer

The problem is that for that browsers (non FF) I still get the not desired results, it how 01.png

Therefore

• If @Controller does not use produces=MediaType.APPLICATION_JSON_VALUE all works fine
• If @Controller does use produces=MediaType.APPLICATION_JSON_VALUE
  • Only works on FF with JSONView, fails in the others Web Browser, with an equivalent add-on.

Therefore I don't know if :

(A) I have any kind of missing configuration in some @Configuration/@Bean
(B) I am using the wrong add-ons for the non-FF browsers.

Assuming is (B) I think is wise share the solution to the community in the reference documentation.

---

Affects: 4.2 GA

Attachments:

• 01.png (13.76 kB)
• 02.png (15.30 kB)

Issue Links:

• Allow specifying HTTP response Content-Type without losing default charset [SPR-13631] #18209 Allow specifying HTTP response Content-Type without losing default charset
• Document why "charset=UTF-8" is specified for JSON and not for XML [SPR-14715] #19280 Document why "charset=UTF-8" is specified for JSON and not for XML

Referenced from: commits 09cb286, 994a11d, 62cd6ad

[spring-projects-issues]

Sébastien Deleuze commented

Hi Manuel Jordan, MappingJackson2HttpMessageConverter default mediatype is not MediaType.APPLICATION_JSON_VALUE but new MediaType("application", "json", Charset.forName("UTF-8")), that's why you observe this behavior when you override the produces attribute.

If you specify @RequestMapping(value = "...", method = RequestMethod.GET, produces = "application/json; charset=UTF-8") it should work as expected.

I am going to update the Javadoc to document that the default charset is UTF-8.

[spring-projects-issues]

Manuel Jordan commented

Hello Sébastien Deleuze

Thanks a lot by the reply. It works, but I am confused

About this:

Hi Manuel Jordan, MappingJackson2HttpMessageConverter default mediatype is not MediaType.APPLICATION_JSON_VALUE but new MediaType("application", "json", Charset.forName("UTF-8")), that's why you observe this behavior when you override the produces attribute.

With the best intentions, do a detailed expansion (explanation) for above due the following (links to Github code for each class):

MappingJackson2HttpMessageConverter is a subclass of AbstractJackson2HttpMessageConverter

The super class has the following declared:

Line 63:

public static final Charset DEFAULT_CHARSET = Charset.forName("UTF-8");

And in the subclass has the following:

Lines 63-68:

/**
 * Construct a new {@link MappingJackson2HttpMessageConverter} with a custom {@link ObjectMapper}.
 * You can use {@link Jackson2ObjectMapperBuilder} to build it easily.
 * @see Jackson2ObjectMapperBuilder#json()
 */
public MappingJackson2HttpMessageConverter(ObjectMapper objectMapper) {
     super(objectMapper, new MediaType("application", "json", DEFAULT_CHARSET),
               new MediaType("application", "*+json", DEFAULT_CHARSET));
}

Note: unique place where that DEFAULT_CHARSET is used in that class.

And how you can see, practically there is located your

new MediaType("application", "json", Charset.forName("UTF-8")

So me question is (in bold soon)

If I am using in my code the MappingJackson2HttpMessageConverter's no-args constructor

@Bean
public MappingJackson2HttpMessageConverter mappingJackson2HttpMessageConverter(){
         MappingJackson2HttpMessageConverter converter = new MappingJackson2HttpMessageConverter();
                                                                                         converter.setPrettyPrint(true);		
         return converter;
}

where in the MappingJackson2HttpMessageConverter's no-args constructor source code we have

Lines 52-58:

/**
 * Construct a new {@link MappingJackson2HttpMessageConverter} using default configuration
 * provided by {@link Jackson2ObjectMapperBuilder}.
 */
public MappingJackson2HttpMessageConverter() {
     this(Jackson2ObjectMapperBuilder.json().build());
}

which calls the other MappingJackson2HttpMessageConverter's args constructor (this()), where DEFAULT_CHARSET is used, why fails my code?.

I am very confused…

Pls expand about this too:

that's why you observe this behavior when you override the produces attribute

Now, like I said, your solution works:

@Controller
@RequestMapping(value=PersonaUrlJsonHelperComplement.ROOT_URL_PRODUCES_JSON, 
			    method=RequestMethod.GET, 
			    //produces=MediaType.APPLICATION_JSON_VALUE
			    produces = "application/json; charset=UTF-8")

Now, even when it works (thanks again by your valuable support) I don't want use static or raw String.

Is possible have something like this:

@Controller
@RequestMapping(value=PersonaUrlJsonHelperComplement.ROOT_URL_PRODUCES_JSON, 
			    method=RequestMethod.GET, 
			    produces=MediaType.APPLICATION_JSON_VALUE
			    charset=SomethingNewClass.UTF-8)

Thanks a lot in advance.

For me I would be happy to work with the current:

@Controller
@RequestMapping(value=PersonaUrlJsonHelperComplement.ROOT_URL_PRODUCES_JSON, 
			    method=RequestMethod.GET, 
			    produces=MediaType.APPLICATION_JSON_VALUE
)

and have the charset=UTF-8" already working by default behind the scenes…

With the best intentions: please share your best explanation and share it in the Reference documentation. Because It would be problematic for other users/developers.

Thanks again

[spring-projects-issues]

Sébastien Deleuze commented

You code is "failing" because your are overriding the default "application/json; charset=UTF-8" media type at http message converter level by the "application/json" one specified at @RequestMapping level.

Since the charset information is already included in the MediaType, you can specify it with:

@RequestMapping(value = "...", method = RequestMethod.GET, produces = MyClass.APPLICATION_JSON_UTF_8)

With

public class MyClass {
    public static final String APPLICATION_JSON_UTF_8 = "application/json; charset=UTF-8";
}

I agree that it is easy to override unexpectedly the media type charset. That said, I think changing the current behavior even slightly could cause some regression, so I would like to avoid it especially in a minor revision. In addition to the Javadoc improvements already commited, my proposal would be to improve the reference documentation "Producible Media Types" section by using "application/json; charset=UTF-8" instead of "application/json" in the example, and add a new explicit warning that explains the MediaType should include the character set if needed (see this commit).

Does it sound like a reasonable improvement?

[spring-projects-issues]

Manuel Jordan commented

Hello

Again thanks by your support.

You code is "failing" because your are overriding the default "application/json; charset=UTF-8" media type at http message converter level

it about

public MappingJackson2HttpMessageConverter(ObjectMapper objectMapper) {
		super(objectMapper, new MediaType("application", "json", DEFAULT_CHARSET),
				new MediaType("application", "*+json", DEFAULT_CHARSET));
}

Am I correct?

by the "application/json" one specified at @RequestMapping level.

Yes, it about

@RequestMapping(value="/something", 
			        method=RequestMethod.GET, 
			        produces=MediaType.APPLICATION_JSON_VALUE
)

Until here: Alpha: - that situation would be added to the reference documentation?
The point here is share the idea or indicate for the developer the warning and consequences about this overriding through produces (where it is not intentional and is problematic to find the reason when that behaviour arises)

About

Since the charset information is already included in the MediaType

What do you mean with that? It has been overridden … I am confused...

About

public class MyClass {
    public static final String APPLICATION_JSON_UTF_8 = "application/json; charset=UTF-8";
}

Yes has sense. That's what I thought. But why not handle from the source (Spring) see below.

Beta
Just checking the class MediaType on Github, I can see in lines 86-89

/**
 * A String equivalent of {@link MediaType#APPLICATION_JSON}.
 */
public final static String APPLICATION_JSON_VALUE = "application/json";

so, why not change to

public final static String APPLICATION_JSON_VALUE = "application/json; charset=UTF-8";

With that the overriding effect disappears.

Delta:

Even if Beta is not viable for some reason (give me details of that reason please) why not create a new attribute named charset for the @RequestMapping?

@Controller
@RequestMapping(value=PersonaUrlJsonHelperComplement.ROOT_URL_PRODUCES_JSON, 
			    method=RequestMethod.GET, 
			    produces=MediaType.APPLICATION_JSON_VALUE
			    charset=SomethingNewClass.UTF-8)

According with my understanding UTF-8 is the best recommended value about encoding. So it should be used always…
correct me if I am wrong please. For some reason it is declared how default in the converter. Am I correct?

I agree that it is easy to override unexpectedly the media type charset.

Yes, I did realize about this "weird" behaviour just doing my own experiments.

That said, I think changing the current behavior even slightly could cause some regression, so I would like to avoid it especially in a minor revision

(1) What do you mean with regression? What could be the problem?
(2) Even if is dangerous that regression. Beta has sense?

In addition to the Javadoc improvements already committed

Pls, could you share that link? I want to see it.

my proposal would be to improve the reference documentation "Producible Media Types" section by using "application/json; charset=UTF-8" instead of "application/json" in the example

Yes, has sense. I am agree. I think is wise explain why that "charset=UTF-8" is added. Perfect place to share the Alpha behaviour. Just curious if Beta fits how the solution here. If Beta is approved then Alpha should be not exists in the reference documentation.

and add a new explicit warning that explains the MediaType should include the character set if needed (see this commit).

Your note is good, but does not indicate the problem about Alpha (the overriding effect).

Omicron

So thinking like reader (according with your current note)
"why I should declare:

produces="application/json; charset=UTF-8"

if MappingJackson2HttpMessageConverter already does that
"

I hope you see my point here.

Does it sound like a reasonable improvement?

Yes, has sense. But consider Omicron.

Kind Regards

-Manuel

[spring-projects-issues]

Sébastien Deleuze commented

Alpha What I mean is that a MediaType can be created for a specific Charset, that's what is done when parsing "application/json; charset=UTF-8", see MediaType(String type, String subtype, Charset charset) constructor for more details.

Beta That would be a breaking change, and we want to avoid it.

Delta I understand the idea behind adding a charset attribute, and the related advantages (redefine the media type without "reseting" the charset unexpectedly), but since this information can and is already specified in the produces one, I am wondering about the backward compatibility and if this is really needed. I will discuss your proposal with the team and send you a feedback.

Omicron You are likely to declare produces="application/json; charset=UTF-8" when you want to restrict a handler method or a controller class to only JSON when your converters support other media types like XML in addition to JSON.

[spring-projects-issues]

Sébastien Deleuze commented

Rossen Stoyanchev Brian Clozel Are you ok with the documentation improvements contained in this commit for our upcoming 4.2.3 release? Do you think it is worth creating another issue in 4.3 or 5.0 backlog to discuss what we could change to make it more easy to specify a produces @RequestMapping attribute value without reseting the charset?

[spring-projects-issues]

Juergen Hoeller commented

Yep, makes sense. Let's document this for the time being and create a follow-up issue for 4.3.

Juergen

[spring-projects-issues]

Manuel Jordan commented

Thanks a lot by your explanation. Now has more sense. I am going to wait the feedback in peace.

BTW: This overriding behaviour does not happen with XML (MediaType.APPLICATION_XML_VALUE).
Just with JSON (MediaType.APPLICATION_JSON_VALUE)

Therefore same data using the weird characters are represented through XML in peace.
(Using other project with the same entity and working with JAXB 2 annotations)

-Manuel

[spring-projects-issues]

Sébastien Deleuze commented

Manuel Jordan Does your XML document contains a prolog with the encoding specified like <?xml version="1.0" encoding="UTF-8"?>?

[spring-projects-issues]

Sébastien Deleuze commented

Documentation has been improved in 4.2.x, and I have created #18209 to try to provide a "better" default behavior in 4.3.x if possible.

[spring-projects-issues]

Manuel Jordan commented

Hello Sébastien Deleuze

Does your XML document contains a prolog with the encoding specified like <?xml version="1.0" encoding="UTF-8"?>?

No…

When I startup the app through Tomcat I can see the following for example in the Console/Terminal

INFO  o.s.w.s.m.m.a.RequestMappingHandlerMapping - 
Mapped "{[/personas/{id}],
methods=[GET],produces=[application/xml || application/json;charset=UTF-8]}" 
onto public com.manuel.jordan.domain.Persona 
com.manuel.jordan.controller.generic.PersonaFindOneController.findOnePathVariableById(java.lang.String,java.lang.String)

How you can see "application/xml"

In the browser I get the following:

This XML file does not appear to have any style information associated with it. The document tree is shown below.
<persona>
<id>100</id>
<nombre>Jesús Você</nombre>
<apellido>Mão Nuñez</apellido>
<fecha>2015-11-02T07:53:53.048-05:00</fecha>
</persona>

Therefore the <?xml version="1.0" encoding="UTF-8"?> line does not appear. Let me know If I should look that line in other place.

Thanks by create the other Jira Post

-Manuel

[spring-projects-issues]

Sébastien Deleuze commented

Some additional information about this behavior.

application/json entry in IANA registry does not define a charset parameter ("No charset parameter is defined for this registration. Adding one really has no effect on compliant recipients."), and JSON encoding can be UTF-8, UTF-16, or UTF-32 (these can be auto-detected) with UTF-8 likely to be the default in most implementations. And I have found this Chrome bug that seems to match with the behavior describe in this issue.

For XML, I think UTF-8 is just the default encoding.

So the fact that this "charset=UTF-8" parameter is needed may come from some browsers not implementing correctly the spec ...

[spring-projects-issues]

Manuel Jordan commented

Thanks by the valuable information

About XML, I have the following:

 @Bean
public MarshallingHttpMessageConverter marshallingMessageConverter() {
    MarshallingHttpMessageConverter converter = new MarshallingHttpMessageConverter();
    converter.setMarshaller(jaxbMarshaller());
    converter.setUnmarshaller(jaxbMarshaller());
    return converter;
}
 @Bean
public Jaxb2Marshaller jaxbMarshaller() {
    Jaxb2Marshaller jaxbMarshaller = new Jaxb2Marshaller();
    jaxbMarshaller.setClassesToBeBound(PersonaXml.class, 
                                                                                                                               PersonaCollection.class, 
                                                                                                                               GenericCollection.class,
                                                                                                                               PersonaHibrid.class);
           Map<String, Object> props = new HashMap<>();
    props.put(Marshaller.JAXB_ENCODING,"UTF-8");
    props.put(Marshaller.JAXB_FORMATTED_OUTPUT, true);
    jaxbMarshaller.setMarshallerProperties(props);
        return jaxbMarshaller;
}

Where we have "props.put(Marshaller.JAXB_ENCODING,"UTF-8");"

In the @Controller I use produces=MediaType.APPLICATION_XML_VALUE. And all work fine. Weird characters appears without any problem in xml format.

Now thinking, wondered by there does not appears the overrding effect for XML how JSON has.

-Manuel

[spring-projects-issues]

Sébastien Deleuze commented

I guess the different behaviors between XML and JSON regarding encoding are coming from this "strange" browser Webkit/Blink behavior regarding JSON encoding described here.

In any case, we will have to keep using this default application/json;charset=UTF-8 content type for a long time. After discussing with Rossen Stoyanchev, and as you suggested in one of your comments, I have added MediaType.APPLICATION_JSON_UTF8 and MediaType.APPLICATION_JSON_UTF8_VALUE constants to make it even easier to keep the encoding while overriding RequestMapping "produces" attribute, see this commit that will be part of Spring Framework 4.2.3.RELEASE.

[spring-projects-issues]

Manuel Jordan commented

Hello

I guess the different behaviors between XML and JSON regarding encoding are coming from this "strange" browser Webkit/Blink behavior regarding JSON encoding described here.

Yes, has sense. Even when Jackson reuses the JAXB 2 annotations to work around JSON. Of course, other history….

I have added MediaType.APPLICATION_JSON_UTF8 and MediaType.APPLICATION_JSON_UTF8_VALUE constants to make it even easier to keep the encoding while overriding RequestMapping "produces" attribute

Excellent, I've checked the commits.. Thanks a lot!, more easier for us (developers)

Being polite: Do not forget add a special note in the Reference Documentation about this situation. I think is wise advice this useful feature for the community…

-Manuel

[spring-projects-issues]

Sébastien Deleuze commented

You're welcome. I have already updated the reference documentation accordingly.