Re-package WebClient so it isn't in the same jar as the server-side pieces [SPR-16760]

[spring-projects-issues]

Dave Syer opened SPR-16760 and commented

If WebClient is a general purpose HTTP client (which I take to be the case, and seems to fit with the messaging from the Framework team) then I feel quite strongly that it should be packaged separately from the server-side components. If all I want is a client, I shouldn't have the server stuff on the classpath. I don't think I'm the only one to be bitten by this (e.g. https://github.com/rabbitmq/hop/issues/122 It also leads to Spring Boot making a false assumption about my intention to start a server.

---

Affects: 5.0.5

0 votes, 9 watchers

[spring-projects-issues]

Rossen Stoyanchev commented

WebClient was in spring-web initially following the example of the RestTemplate. However unlike the RestTemplate which shares nothing with spring-webmvc, the WebClient is more aligned with WebFlux APIs and today shares a common base package + some infrastructure (e.g. BodyInserters, BodyExtractors) with server-side functional endpoints.

The argument can be taken further to say that the RestTemplate and org.springframework.http should be available independent of spring-web, spring-core, etc. and it's a reasonable argument, but it's not a goal of ours. The expectation is that RestTemplate and WebClient are used in scenarios where Spring is already on the classpath.

Can you elaborate more on your specific case?

I did comment on the HOP ticket. I think it's more an indication that the presence of a module alone is not enough to interpret the intent. Netty and Reactor Netty, for what it's worth, also do not split out client support.

 

[spring-projects-issues]

Dave Syer commented

I'm not sure that netty and reactor are in the same category - I don't use those APIs, and I expect nice wrappers to be provided by Spring and Spring Boot. The issue really comes down to the behaviour of Spring Boot when WebClient is on the classpath, but the user didn't want to start a web server. Spring Boot has implemented a workaround which is a flag to switch off the web server, but that seems wrong to me. It's too surprising to see an app suddenly start listening on port 8080 when all you wanted was a client. Even worse, the client behaviour might not even be something that your own app uses or provides (see the HOP and spring-rabbit example), and suddenly you are seeing behaviour and using resources that you don't want.

 

I get that breaking this up means 1 jar might become 2 or 3. I can see why you'd rather not do it. But I think Spring Framework owes this as a courtesy to Spring Boot and Spring Boot users, where there really isn't an acceptable way to implement sensible autoconfig for a webflux server without the split.

[spring-projects-issues]

Brian Clozel commented

HOP is a bad example here, because moving to reactor-netty is a better choice for them as a lightweight library that doesn't need a full blown web framework just to send HTTP requests and decode JSON. The cf-java-client team made the same choice and I believe it's the right one.

We did talk numerous times about that possible split before 5.0, and if I remember correctly the rationale for keeping everything together was:

• the WebClient is not meant to be distributed as an independent library; it is meant to be an HTTP client within a Spring WebFlux application. It can be used without a WebFlux server running, but it's clearly aligned with a "HTTP client in a server app" use case
• the code naturally aligns with that decision and making things independent would potentially break many features like sharing codecs and resources, automatic infrastructure setup, testing features, and more
• we knew at that point that this would differ from what we've got with Spring MVC and RestTemplate and that this would force auto-configuration choices in Spring Boot

I think that splitting the current arrangement in several jars would require moving public classes to different packages, breaking backwards compatibility. The alternative, using split packages, is definitely not a good idea.

Now I don't consider the Spring Boot configuration flag as a workaround, as it's been there (under a different name) for ages, switching on/off the web server auto-configuration. If the WebClient is mostly expected to be used within a server application, Spring Boot is here reflecting that opinion in its auto-configuration.

If anything, we could make that a bit more obvious in the Spring Framework reference documentation (what WebClient is meant for), and in Spring Boot (auto-configuration choices).

[spring-projects-issues]

Dave Syer commented

 If the {{WebClient}} is mostly expected to be used within a server application

 

That's the assumption that I think is invalid. Which reactive HTTP client would we recommend for use not in a server application, if not this one? What part of it makes it more useful in a server than out? It doesn't make any sense still to me, but I'm happy to learn that I am wrong. If I am wrong then I think we need another HTTP client.

[spring-projects-issues]

Arnaud Cogoluègnes commented

What would be the recommendation for a batch application using Spring Boot batch starter and in need of a reactive HTTP client? I admit it's a bit far-fetched, but using WebClient and switching off the web server looks like the most reasonable way by now. It sounds odd to be told "don't use the WebClient because this isn't a web application", especially when there's a bunch of Spring dependencies on the classpath already.

[spring-projects-issues]

Rossen Stoyanchev commented

In addition to the RestTemplate, the spring-websocket module also includes client and server, spring-messaging has STOMP client and server, and not to forget spring-webflux has WebSocket client and server support too. We would have a lot more than 2-3 jars if we followed through with the idea.

It's not just about the extra jars though. It's about how package structure and module boundary decisions are made. I have a hard time accepting that autoconfig should dictate such decisions even in cases where we do have control. The current structure under o.s.w.r.function makes sense, it's what we arrived at, and is the preferred structure.

As for the docs, what would we actually say? I find "WebClient is not meant to be distributed as an independent client library" rather vague. Using only WebClient or WebSocketClient from spring-webflux, or even distributing spring-webflux is not a problem by itself, assuming you're okay with the extra dependencies. This is no different from the RestTemplate, to which the WebClient is positioned as a better alternative. There is also nothing inherently wrong with using the WebClient in a client application, and the presence of server support again is not a problem by itself. The only way to properly understand that statement is in the context of Spring Boot autoconfig.

On the HOP ticket I asked about the possibility to detect if spring-boot-starter-webflux was declared. I'm quoting the response in order to continue that conversation:

A Spring Boot starter is nothing but a convenient list of dependencies. In this case, spring-boot-starter-webflux is roughly spring-webflux + reactor-netty, which both ship client and server implementations in a single jar.

Enforcing different behaviors between using starters and raw dependencies would be inconsistent with the whole Spring Boot ecosystem.

Thanks and good to know. Let me expand on the idea rather than prescribing the solution. Currently a WebFlux server is started when spring-webflux is present. That assumption is turned off implicitly if spring-webmvc is also declared, or explicitly if spring.main.web-application-type=none is added. This model takes into account a couple of scenarios but I think it needs evolving.

What if the model was reversed? Make no assumptions from the presence of spring-webflux and instead look for a more explicit indication of the intent to start a WebFlux server. A webflux starter is just such an indication and hence my question. Is there no feasible way to detect what starters were declared, or is the objection purely about making such a check in the first place? Arguably in this case, it would be the very intent to differentiate between the declaration of a starter and the mere presence of a dependency.

Some such change, e.g. a property, manifest attribute, or any marker in the webflux starter, that can then be detected should not disrupt existing applications since they already declare the webflux starter. I understand such a change goes against current practice, but may be there is some generally useful feature that can come out of this? As for applications that intentionally don't declare the webflux starter but want a WebFlux server, they would have to use something like the spring.main.web-application-type property and that's a reasonble expectation.

The alternative of breaking up spring-webflux, this far into 5.0.x, when it's already used in applications and frameworks, is going to cause enough pain and is practically a non-starter. The docs could be updated to explain about the use of WebClient, spring-webflux, and Spring Boot autoconfig, but it doesn't change the fact that we'd have to also qualify when the WebClient can or cannot replace the RestTemplate and that's a complex explanation if all cases are covered.

[spring-projects-issues]

Rossen Stoyanchev commented

Arnaud Cogoluègnes, the spring-batch example might not be very common indeed, I can't say, but the Spring Integration HTTP outbound adapter is currently using the RestTemplate and it could reach a similar dilemma at some point, except in that case the use of Spring dependencies is not at all an issue.

[spring-projects-issues]

Phil Webb commented

If you consider the design of spring-webflux.jar to be closer to spring-web than spring-webmvc, then the current granularity doesn't seem bad. I guess it might be possible to split up spring-webflux some more, but I don't think we should use the Spring Boot's auto-configuration problems as a driving factor in that decision. I've raised #12973 to see if there's a better way for Spring Boot to deduce if an embedded server should be started or not.

[spring-projects-issues]

Stéphane Nicoll commented

What if the model was reversed?

That would be a breaking change (and not a small one). There are several ways to see the current big picture but I think we haven't discussed much about what Spring Boot is doing. Being opinionated at the right level proved to be very popular IMO. Adding spring-boot-starter-web brings the necessary dependencies to start a server so that's what we do by default. They were a few support issues where folks would add that starter only to use RestTemplate and were confused that a web server was started. We could easily explain what to do, the idea of a "rest starter" that would only bring spring-web was also requested (and declined).

In Spring Boot 2, our goal was to offer the same developer experience and a consistent set of use cases regardless of the kind of app you want to build. So, naturally, there is a webflux starter that is the mirror of the web starter, and it also starts a web server by default.

 I have a hard time accepting that autoconfig should dictate such decisions 

IMO, that is not what's happening here. If you look at the features parity I mentioned earlier, the http client is a separate unit in the MVC world and isnt' in the reactive world. The auto-config doesn't dictate anything, it tries to adapt to this new world with similar concepts.

 

[spring-projects-issues]

Rossen Stoyanchev commented

I'm not questioning the decisions made and I agree completely that Boot 2 autoconfig does not dictate anything but rather adapts and evolves the model with Spring MVC. I am pointing out the need to evolve autoconfig rather than expect spring-webflux to be broken up.

I am not suggesting a breaking change. You only quote the first sentence of that paragraph. I can't say what the solution is but I guess the conversation on that continues under the Boot ticket Phil created, and I'm learning myself from the discussions on that so far.

To elaborate on Phil's comment about spring-webflux being closer to spring-web than spring-webmvc, the root package of spring-webmvc, o.s.web.servlet, makes it the home for Servlet based support and naturally neither the RestTemplate, nor the spring-websocket module fit there. That makes it very convenient for autoconfig. The base package in spring-webflux o.s.web.reactive makes it the right home for a wider range of things including client, server programming models, and also WebSocket support. So just like the presence of spring-web doesn't tell much, same is true for spring-webflux.

 

[spring-projects-issues]

Rossen Stoyanchev commented

Resolving for now reflecting the discussion. Depending on the outcome of #12973, if there is no change, then we could consider some doc update.

For example if there was one place that explains thoroughly how Spring Boot autoconfig works with regards to spring-webflux, then we could link to that from the Spring Framework reference, and that I think would be all the information we need to provide.

From what I could find Web Environment talks about Spring MVC + spring-webflux, there is also the WebClient section but nothing to put the full picture together -- i.e. that spring-webflux implies a web environment and that will start a WebFlux server, unless Spring MVC is there too, so if you're using the WebClient in any other scenario (neither Spring MVC, not WebFlux server), then you have to turn off the web environment property too.