[CI] FieldExtractorIT class failing

[elasticsearchmachine]

Build Scans:

• elasticsearch-periodic #8030 / openjdk21_8.19.0_java-fips-matrix-bwc
• elasticsearch-pull-request #70533 / 8.19.0_bwc-snapshots
• elasticsearch-pull-request #70456 / 8.19.0_bwc-snapshots
• elasticsearch-periodic #8000 / openjdk21_8.19.0_java-fips-matrix-bwc
• elasticsearch-pull-request #70356 / 8.19.0_bwc-snapshots
• elasticsearch-pull-request #70349 / 8.19.0_bwc
• elasticsearch-pull-request #70378 / 8.19.0_bwc-snapshots
• elasticsearch-pull-request #70385 / 8.19.0_bwc-snapshots
• elasticsearch-pull-request #70377 / 8.19.0_bwc-snapshots
• elasticsearch-intake #22574 / 8.19.0_bwc-snapshots

Reproduction Line:

undefined

Applicable branches:
main

Reproduces locally?:
N/A

Failure History:
See dashboard

Failure Message:

undefined

Issue Reasons:

• [main] 11 failures in class org.elasticsearch.xpack.esql.qa.mixed.FieldExtractorIT (3.1% fail rate in 350 executions)
• [main] 2 failures in step openjdk21_8.19.0_java-fips-matrix-bwc (20.0% fail rate in 10 executions)
• [main] 8 failures in step 8.19.0_bwc-snapshots (6.2% fail rate in 130 executions)
• [main] 2 failures in pipeline elasticsearch-periodic (20.0% fail rate in 10 executions)
• [main] 7 failures in pipeline elasticsearch-pull-request (6.3% fail rate in 111 executions)
• [main] 2 failures in pipeline elasticsearch-intake (5.9% fail rate in 34 executions)

Note:
This issue was created using new test triage automation. Please report issues or feedback to es-delivery.

[elasticsearchmachine]

Pinging @elastic/es-analytical-engine (Team:Analytics)

[alex-spies]

It looks to me like we had trouble during cluster orchestration. Doesn't seem related to FieldExtractorIT itself.

First non-PR buildscan has

Caused by: java.lang.RuntimeException: Timed out after PT3M waiting for ports files for: { cluster: 'test-cluster', node: 'test-cluster-1' }	

and I found

[2025-05-05T21:07:53,129][ERROR][o.e.x.c.s.SSLService     ] [test-cluster-0] unsupported ciphers [[TLS_CHACHA20_POLY1305_SHA256, TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256, TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256]] were requested but cannot be used in this JVM, however there are supported ciphers that will be used [[TLS_AES_256_GCM_SHA384, TLS_AES_128_GCM_SHA256, TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA]]. If you are trying to use ciphers with a key length greater than 128 bits on an Oracle JVM, you will need to install the unlimited strength JCE policy files.

Second non-PR buildscan looks just like that.

Intake buildscan has:

[2025-05-02T19:50:36,749][WARN ][o.e.c.c.PublicationTransportHandler] [test-cluster-2] unexpected error while deserializing an incoming cluster state java.time.DateTimeException: Invalid ID for ZoneOffset, invalid format: 

	at java.base/java.time.ZoneOffset.of(ZoneOffset.java:250)
	at java.base/java.time.ZoneId.of(ZoneId.java:405)
	at java.base/java.time.ZoneId.of(ZoneId.java:359)
	at org.elasticsearch.server@8.19.0-SNAPSHOT/org.elasticsearch.common.io.stream.StreamInput.readZonedDateTime(StreamInput.java:916)
	at org.elasticsearch.server@8.19.0-SNAPSHOT/org.elasticsearch.common.io.stream.StreamInput.readGenericValue(StreamInput.java:869)
	at org.elasticsearch.server@8.19.0-SNAPSHOT/org.elasticsearch.common.io.stream.StreamInput.readGenericMap(StreamInput.java:789)
	at org.elasticsearch.server@8.19.0-SNAPSHOT/org.elasticsearch.cluster.metadata.ComponentTemplate.<init>(ComponentTemplate.java:92)
	at org.elasticsearch.server@8.19.0-SNAPSHOT/org.elasticsearch.common.io.stream.StreamInput.readMap(StreamInput.java:742)
	at org.elasticsearch.server@8.19.0-SNAPSHOT/org.elasticsearch.common.io.stream.StreamInput.readMap(StreamInput.java:719)
	at org.elasticsearch.server@8.19.0-SNAPSHOT/org.elasticsearch.cluster.metadata.ComponentTemplateMetadata.<init>(ComponentTemplateMetadata.java:64)
	at org.elasticsearch.server@8.19.0-SNAPSHOT/org.elasticsearch.common.io.stream.NamedWriteableAwareStreamInput.readNamedWriteable(NamedWriteableAwareStreamInput.java:57)
	at org.elasticsearch.server@8.19.0-SNAPSHOT/org.elasticsearch.cluster.NamedDiffableValueSerializer.read(NamedDiffableValueSerializer.java:30)
	at org.elasticsearch.server@8.19.0-SNAPSHOT/org.elasticsearch.cluster.NamedDiffableValueSerializer.read(NamedDiffableValueSerializer.java:20)
	at org.elasticsearch.server@8.19.0-SNAPSHOT/org.elasticsearch.cluster.DiffableUtils$MapDiff.<init>(DiffableUtils.java:320)
	at org.elasticsearch.server@8.19.0-SNAPSHOT/org.elasticsearch.cluster.DiffableUtils.readImmutableOpenMapDiff(DiffableUtils.java:95)
	at org.elasticsearch.server@8.19.0-SNAPSHOT/org.elasticsearch.cluster.metadata.Metadata$MetadataDiff.<init>(Metadata.java:1624)
	at org.elasticsearch.server@8.19.0-SNAPSHOT/org.elasticsearch.cluster.metadata.Metadata.readDiffFrom(Metadata.java:1495)
	at org.elasticsearch.server@8.19.0-SNAPSHOT/org.elasticsearch.cluster.ClusterState$ClusterStateDiff.<init>(ClusterState.java:1181)
	at org.elasticsearch.server@8.19.0-SNAPSHOT/org.elasticsearch.cluster.ClusterState.readDiffFrom(ClusterState.java:1039)
	at org.elasticsearch.server@8.19.0-SNAPSHOT/org.elasticsearch.cluster.coordination.PublicationTransportHandler.deserializeAndApplyDiff(PublicationTransportHandler.java:196)
	at org.elasticsearch.server@8.19.0-SNAPSHOT/org.elasticsearch.cluster.coordination.PublicationTransportHandler.handleIncomingPublishRequest(PublicationTransportHandler.java:166)
	at org.elasticsearch.server@8.19.0-SNAPSHOT/org.elasticsearch.cluster.coordination.PublicationTransportHandler.lambda$new$0(PublicationTransportHandler.java:115)
	at org.elasticsearch.server@8.19.0-SNAPSHOT/org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:76)
	at org.elasticsearch.server@8.19.0-SNAPSHOT/org.elasticsearch.transport.InboundHandler.doHandleRequest(InboundHandler.java:289)
	at org.elasticsearch.server@8.19.0-SNAPSHOT/org.elasticsearch.transport.InboundHandler$1.doRun(InboundHandler.java:302)
	at org.elasticsearch.server@8.19.0-SNAPSHOT/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:1044)
	at org.elasticsearch.server@8.19.0-SNAPSHOT/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:27)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1095)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:619)
	at java.base/java.lang.Thread.run(Thread.java:1447)

[2025-05-02T19:50:36,776][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [test-cluster-2] fatal error in thread [elasticsearch[test-cluster-2][generic][T#4]], exiting java.lang.AssertionError: java.time.DateTimeException: Invalid ID for ZoneOffset, invalid format: 

	at org.elasticsearch.server@8.19.0-SNAPSHOT/org.elasticsearch.cluster.coordination.PublicationTransportHandler.deserializeAndApplyDiff(PublicationTransportHandler.java:219)
	at org.elasticsearch.server@8.19.0-SNAPSHOT/org.elasticsearch.cluster.coordination.PublicationTransportHandler.handleIncomingPublishRequest(PublicationTransportHandler.java:166)
	at org.elasticsearch.server@8.19.0-SNAPSHOT/org.elasticsearch.cluster.coordination.PublicationTransportHandler.lambda$new$0(PublicationTransportHandler.java:115)
	at org.elasticsearch.server@8.19.0-SNAPSHOT/org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:76)
	at org.elasticsearch.server@8.19.0-SNAPSHOT/org.elasticsearch.transport.InboundHandler.doHandleRequest(InboundHandler.java:289)
	at org.elasticsearch.server@8.19.0-SNAPSHOT/org.elasticsearch.transport.InboundHandler$1.doRun(InboundHandler.java:302)
	at org.elasticsearch.server@8.19.0-SNAPSHOT/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:1044)
	at org.elasticsearch.server@8.19.0-SNAPSHOT/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:27)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1095)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:619)
	at java.base/java.lang.Thread.run(Thread.java:1447)
Caused by: java.time.DateTimeException: Invalid ID for ZoneOffset, invalid format: 

	at java.base/java.time.ZoneOffset.of(ZoneOffset.java:250)
	at java.base/java.time.ZoneId.of(ZoneId.java:405)
	at java.base/java.time.ZoneId.of(ZoneId.java:359)
	at org.elasticsearch.server@8.19.0-SNAPSHOT/org.elasticsearch.common.io.stream.StreamInput.readZonedDateTime(StreamInput.java:916)
	at org.elasticsearch.server@8.19.0-SNAPSHOT/org.elasticsearch.common.io.stream.StreamInput.readGenericValue(StreamInput.java:869)
	at org.elasticsearch.server@8.19.0-SNAPSHOT/org.elasticsearch.common.io.stream.StreamInput.readGenericMap(StreamInput.java:789)
	at org.elasticsearch.server@8.19.0-SNAPSHOT/org.elasticsearch.cluster.metadata.ComponentTemplate.<init>(ComponentTemplate.java:92)
	at org.elasticsearch.server@8.19.0-SNAPSHOT/org.elasticsearch.common.io.stream.StreamInput.readMap(StreamInput.java:742)
	at org.elasticsearch.server@8.19.0-SNAPSHOT/org.elasticsearch.common.io.stream.StreamInput.readMap(StreamInput.java:719)
	at org.elasticsearch.server@8.19.0-SNAPSHOT/org.elasticsearch.cluster.metadata.ComponentTemplateMetadata.<init>(ComponentTemplateMetadata.java:64)
	at org.elasticsearch.server@8.19.0-SNAPSHOT/org.elasticsearch.common.io.stream.NamedWriteableAwareStreamInput.readNamedWriteable(NamedWriteableAwareStreamInput.java:57)
	at org.elasticsearch.server@8.19.0-SNAPSHOT/org.elasticsearch.cluster.NamedDiffableValueSerializer.read(NamedDiffableValueSerializer.java:30)
	at org.elasticsearch.server@8.19.0-SNAPSHOT/org.elasticsearch.cluster.NamedDiffableValueSerializer.read(NamedDiffableValueSerializer.java:20)
	at org.elasticsearch.server@8.19.0-SNAPSHOT/org.elasticsearch.cluster.DiffableUtils$MapDiff.<init>(DiffableUtils.java:320)
	at org.elasticsearch.server@8.19.0-SNAPSHOT/org.elasticsearch.cluster.DiffableUtils.readImmutableOpenMapDiff(DiffableUtils.java:95)
	at org.elasticsearch.server@8.19.0-SNAPSHOT/org.elasticsearch.cluster.metadata.Metadata$MetadataDiff.<init>(Metadata.java:1624)
	at org.elasticsearch.server@8.19.0-SNAPSHOT/org.elasticsearch.cluster.metadata.Metadata.readDiffFrom(Metadata.java:1495)
	at org.elasticsearch.server@8.19.0-SNAPSHOT/org.elasticsearch.cluster.ClusterState$ClusterStateDiff.<init>(ClusterState.java:1181)
	at org.elasticsearch.server@8.19.0-SNAPSHOT/org.elasticsearch.cluster.ClusterState.readDiffFrom(ClusterState.java:1039)
	at org.elasticsearch.server@8.19.0-SNAPSHOT/org.elasticsearch.cluster.coordination.PublicationTransportHandler.deserializeAndApplyDiff(PublicationTransportHandler.java:196)
	... 10 more

[alex-spies]

At least the SSLService errors have been observed before in #87106.

Back then we said it's temporary flakiness causing timeouts.

But the assertion error in PublicationTransportHandler.deserializeAndApplyDiff looks interesting and I can see fresh changes related to de-/serializing cluster state diffs in #127404, so it is probably best to double check this.

@original-brownbear , do you think you could take a look and assess the failure in the intake buildscan?

[elasticsearchmachine]

Pinging @elastic/es-distributed-coordination (Team:Distributed Coordination)