Spring Boot 3.3.5 webflux dependency causing threads to block with state of TIMED_WAITING

[vahidmah]

Running on: openjdk 22.0.2, Kotlin: 2.0.21, Spring Boot: 3.3.5, Spring Cloud: 2023.0.3.

My Project is a REST API using Hikari for connection pooling. When I run many concurrent requests against my app, I can see that some connections never return to HikariPool

HikariPool-1 - Pool stats (total=10, active=3, idle=7, waiting=0)

I got a thread dump and can see many threads are in TIMED_WAITING state.

http-nio-7900-exec-18 (waiting on condition)

http-nio-7900-exec-18" prio=0 tid=0x0 nid=0x0 waiting on condition
     java.lang.Thread.State: TIMED_WAITING
 on kotlinx.coroutines.BlockingCoroutine@70dc10b
	at [email protected]/jdk.internal.misc.Unsafe.park(Native Method)

All the waiting on condition threads are running Kotlin runBloking code in which I'm using org.springframework.web.reactive.function.client.WebClient.

As an example,

<code-removed-for-brevity>

val result: Map<String, Any> = runBlocking {
  getFileDetails("user1", fileId)
}

private fun createWebClient(username: String, logging: Boolean = true): WebClient {
    return WebClient.builder()
        .uriBuilderFactory(factory)
        .clientConnector(if (logging) loggingConnector else nonLoggingConnector)
        .filters { exchangeFilterFunctions -> filters.forEach { exchangeFilterFunctions.add(it) } }
        .codecs { it.defaultCodecs().maxInMemorySize(MAX_MEMORY_SIZE) }
        .build()
}

private suspend fun getFileDetails(
    username: String,
    fileId: Long,
): Map<String, Any> {
    return createWebClient(username)
      .get()
      .uri { uriBuilder ->
        uriBuilder
            .path("/files/$fileId")
            .build()
      }
      .retrieve()
      .awaitBody()
}

Note that:

1. Spring Boot 3.3.5 brings in spring-webflux version 6.1.14.
2. Same code works in Spring Boot 3.3.4 which brings in spring-webflux version 6.1.13.
3. I use Spring mvc in controller and not reactive controller.

If I override spring-webflux in my pom back to 6.1.13 everything works just fine. There are no threads in TIMED_WAITING and HiKari pool will show active connection of 0 at the end of my concurrent test.

Is this a bug related to spring-webflux version 6.1.14?

[sdeleuze]

Hey, sorry for the delay, if you still observe this issue with Spring Framework 6.2, could you please provide a reproducer that can allow us to reproduce the working version and the broken one?

[spring-projects-issues]

If you would like us to look at this issue, please provide the requested information. If the information is not provided within the next 7 days this issue will be closed.

[spring-projects-issues]

Closing due to lack of requested feedback. If you would like us to look at this issue, please provide the requested information and we will re-open the issue.

[vahidmah]

Hi @sdeleuze,

I managed to fix my issue. I'm just sharing in case this helps other.
As its shown above, my webclient has an ExchangeFilterFunction. This filter is used get a valid oauth2 token before an actual call. The problem was, I was not fully consuming the response body.

override fun filter(request
                    : clientRequest, next
                    : ExchangeFunction)
    : Mono<ClientResponse> {
  return Mono.just(request)
      .filter{it.attribute(OAUTH2_AUTH_CLIENT_ATTR_NAME).isPresent}
      .flatMap{reauthorizeClient(getOauthAuthClient(it.attributes()))}
      .switchIfEmpty(Mono.defer{
          Mono.just(request).flatMap{authorizeClient(this.clientRegistrationId)}

      })
      .map{bearer(request.requireNotNull(it))}
      .flatMap{next.exchange(it)
      // This is the fix. Consuming the response fully
      .flatMap{response
          -> response.bodyToMono(String::class.java)
          .thenReturn(Response)
      }}
      .switchIfEmpty(Mono.dfer{next.exchange(request)})
}

I'm not quite sure why this was not an issue in webflux-6.1.13, because the same code without the fix was working. I'm guessing the issue in webflux-6.1.14 is related to #33559 and Commit 776811b.