Delegated authorization using Microsoft Graph (SDK) #128396

richard-dennehy · 2025-05-23T15:17:07Z

Initial implementation of authorizing Entra ID users with the Microsoft Graph API via the official Microsoft Graph SDK for Java.

Getting the SDK integrated with Elasticsearch has proven challenging, and requires workarounds for various issues, including:

Merging the Kiota libs together, to work around microsoft-kiota-** dependencies have conflicting Automatic-Module-Name microsoft/kiota-java#1108
Bytecode patching azure-core to delete a forbidden call to addShutdownHook
A few strange class path issues (that is, classes failing to load at runtime despite the relevant libraries being declared as dependencies)

This works when tested against a real Azure account, and has a functioning Integration test.

...src/main/java/org/elasticsearch/xpack/security/authz/microsoft/MicrosoftGraphAuthzRealm.java

richard-dennehy · 2025-05-23T15:34:06Z

x-pack/plugin/security/build.gradle

@@ -80,6 +80,7 @@ dependencies {

  // Dependencies for oidc
  api "com.nimbusds:oauth2-oidc-sdk:11.22.2"
+  runtimeOnly "com.nimbusds:content-type:2.3"


if I add this as a dependency to the plugin, where it should be, there's a NoClassDefFoundError at runtime

I've spent some time looking into this. It seems like the wrong class loader is used or the ExtendedPluginsClassLoader is not working correctly for some reason. It seem to break when the SAML realm calls out to the delegated authorization realm for some reason.

If I move runtimeOnly "com.nimbusds:content-type:2.3" to the plugin build.gradle as implementation "com.nimbusds:content-type:2.3" it's included in the distribution archive.

If I then explicitly add this call to the lookupUser method it's able to load the class:

ClassLoader extensionLoader = MicrosoftGraphAuthzRealm.class.getClassLoader(); Class<?> type = Class.forName("com.nimbusds.common.contenttype.ContentType", true, extensionLoader);

But then it still fails with

An error occurred while attempting to authenticate [<unauthenticated-saml-user>] against realm [saml1] java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.NoClassDefFoundError: com/nimbusds/common/contenttype/ContentType at [email protected]/com.azure.identity.implementation.IdentitySyncClient.authenticateWithConfidentialClient(IdentitySyncClient.java:142) at [email protected]/com.azure.identity.ClientSecretCredential.getTokenSync(ClientSecretCredential.java:133) at [email protected]/com.microsoft.kiota.authentication.AzureIdentityAccessTokenProvider.getAuthorizationToken(AzureIdentityAccessTokenProvider.java:167) at [email protected]/com.microsoft.kiota.authentication.BaseBearerTokenAuthenticationProvider.authenticateRequest(BaseBearerTokenAuthenticationProvider.java:46) at [email protected]/com.microsoft.kiota.http.OkHttpRequestAdapter.getHttpResponseMessage(OkHttpRequestAdapter.java:741) at [email protected]/com.microsoft.kiota.http.OkHttpRequestAdapter.send(OkHttpRequestAdapter.java:302) at [email protected]/com.microsoft.graph.users.item.UserItemRequestBuilder.get(UserItemRequestBuilder.java:761) at [email protected]/org.elasticsearch.xpack.security.authz.microsoft.MicrosoftGraphAuthzRealm.sdkFetchUserProperties(MicrosoftGraphAuthzRealm.java:144) at [email protected]/org.elasticsearch.xpack.security.authz.microsoft.MicrosoftGraphAuthzRealm.lookupUser(MicrosoftGraphAuthzRealm.java:97) at [email protected]/org.elasticsearch.xpack.security.authc.support.RealmUserLookup.lambda$lookup$1(RealmUserLookup.java:48) at [email protected]/org.elasticsearch.xpack.core.common.IteratingActionListener.run(IteratingActionListener.java:117) at [email protected]/org.elasticsearch.xpack.security.authc.support.RealmUserLookup.lookup(RealmUserLookup.java:59) at [email protected]/org.elasticsearch.xpack.security.authc.support.DelegatedAuthorizationSupport.resolve(DelegatedAuthorizationSupport.java:132) at [email protected]/org.elasticsearch.xpack.security.authc.saml.SamlRealm.buildUser(SamlRealm.java:588) at [email protected]/org.elasticsearch.xpack.security.authc.saml.SamlRealm.authenticate(SamlRealm.java:550) at [email protected]/org.elasticsearch.xpack.security.authc.RealmsAuthenticator.lambda$consumeToken$4(RealmsAuthenticator.java:170) at [email protected]/org.elasticsearch.xpack.core.common.IteratingActionListener.onResponse(IteratingActionListener.java:135) at [email protected]/org.elasticsearch.xpack.security.authc.RealmsAuthenticator.lambda$consumeToken$4(RealmsAuthenticator.java:234) at [email protected]/org.elasticsearch.xpack.core.common.IteratingActionListener.onResponse(IteratingActionListener.java:135) at [email protected]/org.elasticsearch.xpack.security.authc.RealmsAuthenticator.lambda$consumeToken$4(RealmsAuthenticator.java:234) at [email protected]/org.elasticsearch.xpack.core.common.IteratingActionListener.onResponse(IteratingActionListener.java:135) at [email protected]/org.elasticsearch.xpack.security.authc.RealmsAuthenticator.lambda$consumeToken$4(RealmsAuthenticator.java:234) at [email protected]/org.elasticsearch.xpack.core.common.IteratingActionListener.run(IteratingActionListener.java:117) at [email protected]/org.elasticsearch.xpack.security.authc.RealmsAuthenticator.consumeToken(RealmsAuthenticator.java:271) at [email protected]/org.elasticsearch.xpack.security.authc.RealmsAuthenticator.authenticate(RealmsAuthenticator.java:106) at [email protected]/org.elasticsearch.xpack.security.authc.AuthenticatorChain.lambda$getAuthenticatorConsumer$5(AuthenticatorChain.java:167) at [email protected]/org.elasticsearch.xpack.core.common.IteratingActionListener.onResponse(IteratingActionListener.java:135) at [email protected]/org.elasticsearch.xpack.security.authc.AuthenticatorChain.lambda$getAuthenticatorConsumer$4(AuthenticatorChain.java:175) at [email protected]/org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:258) at [email protected]/org.elasticsearch.xpack.security.authc.ApiKeyAuthenticator.authenticate(ApiKeyAuthenticator.java:73) at [email protected]/org.elasticsearch.xpack.security.authc.AuthenticatorChain.lambda$getAuthenticatorConsumer$5(AuthenticatorChain.java:167) at [email protected]/org.elasticsearch.xpack.core.common.IteratingActionListener.onResponse(IteratingActionListener.java:135) at [email protected]/org.elasticsearch.xpack.security.authc.AuthenticatorChain.lambda$getAuthenticatorConsumer$4(AuthenticatorChain.java:175) at [email protected]/org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:258) at [email protected]/org.elasticsearch.xpack.security.authc.OAuth2TokenAuthenticator.authenticate(OAuth2TokenAuthenticator.java:65) at [email protected]/org.elasticsearch.xpack.security.authc.AuthenticatorChain.lambda$getAuthenticatorConsumer$5(AuthenticatorChain.java:167) at [email protected]/org.elasticsearch.xpack.core.common.IteratingActionListener.onResponse(IteratingActionListener.java:135) at [email protected]/org.elasticsearch.xpack.security.authc.AuthenticatorChain.lambda$getAuthenticatorConsumer$4(AuthenticatorChain.java:175) at [email protected]/org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:258) at [email protected]/org.elasticsearch.xpack.security.authc.ServiceAccountAuthenticator.authenticate(ServiceAccountAuthenticator.java:78) at [email protected]/org.elasticsearch.xpack.security.authc.AuthenticatorChain.lambda$getAuthenticatorConsumer$5(AuthenticatorChain.java:167) at [email protected]/org.elasticsearch.xpack.core.common.IteratingActionListener.run(IteratingActionListener.java:117) at [email protected]/org.elasticsearch.xpack.security.authc.AuthenticatorChain.doAuthenticate(AuthenticatorChain.java:125) at [email protected]/org.elasticsearch.xpack.security.authc.AuthenticatorChain.authenticate(AuthenticatorChain.java:81) at [email protected]/org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:225) at [email protected]/org.elasticsearch.xpack.security.action.saml.TransportSamlAuthenticateAction.doExecuteForked(TransportSamlAuthenticateAction.java:83) at [email protected]/org.elasticsearch.xpack.security.action.saml.TransportSamlAuthenticateAction.lambda$doExecute$0(TransportSamlAuthenticateAction.java:73) at [email protected]/org.elasticsearch.action.ActionRunnable$4.doRun(ActionRunnable.java:101) at [email protected]/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:1044) at [email protected]/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:27) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1095) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:619) at java.base/java.lang.Thread.run(Thread.java:1447) Caused by: java.util.concurrent.ExecutionException: java.lang.NoClassDefFoundError: com/nimbusds/common/contenttype/ContentType at java.base/java.util.concurrent.CompletableFuture.wrapInExecutionException(CompletableFuture.java:347) at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:442) at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2119) at [email protected]/com.azure.identity.implementation.IdentitySyncClient.authenticateWithConfidentialClient(IdentitySyncClient.java:140) ... 52 more Caused by: java.lang.NoClassDefFoundError: com/nimbusds/common/contenttype/ContentType at [email protected]/com.nimbusds.oauth2.sdk.http.HTTPMessage.setContentType(HTTPMessage.java:104) at [email protected]/com.nimbusds.oauth2.sdk.http.HTTPRequest.setContentType(HTTPRequest.java:74) at [email protected]/com.microsoft.aad.msal4j.TokenRequestExecutor.createOauthHttpRequest(TokenRequestExecutor.java:45) at [email protected]/com.microsoft.aad.msal4j.TokenRequestExecutor.executeTokenRequest(TokenRequestExecutor.java:35) at [email protected]/com.microsoft.aad.msal4j.AbstractApplicationBase.acquireTokenCommon(AbstractApplicationBase.java:56) at [email protected]/com.microsoft.aad.msal4j.AcquireTokenByAuthorizationGrantSupplier.execute(AcquireTokenByAuthorizationGrantSupplier.java:63) at [email protected]/com.microsoft.aad.msal4j.AcquireTokenByClientCredentialSupplier.acquireTokenByClientCredential(AcquireTokenByClientCredentialSupplier.java:87) at [email protected]/com.microsoft.aad.msal4j.AcquireTokenByClientCredentialSupplier.execute(AcquireTokenByClientCredentialSupplier.java:50) at [email protected]/com.microsoft.aad.msal4j.AuthenticationResultSupplier.get(AuthenticationResultSupplier.java:69) at [email protected]/com.microsoft.aad.msal4j.AuthenticationResultSupplier.get(AuthenticationResultSupplier.java:18) at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1814) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:545) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:328) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:309) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1095) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:619) at java.base/java.lang.VirtualThread.run(VirtualThread.java:466) Caused by: java.lang.ClassNotFoundException: com.nimbusds.common.contenttype.ContentType at [email protected]/org.elasticsearch.plugins.ExtendedPluginsClassLoader.findClass(ExtendedPluginsClassLoader.java:37) at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:557) at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:490) at java.base/jdk.internal.loader.Loader.loadClass(Loader.java:503) at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:490) ... 17 more

I wonder if it has something to do with the context we're in when we're doing delegated authorization. We're in the context of the Security plugin classloader that doesn't seem to know about the dependencies of the microsoft graph plugin.

@tvernum do you know of any limitations like the one above? Can be reproduced by moving the dependency to the plugin and running the MicrosoftGraphAuthzPluginIT.

I think I understand what happens now:

The msal4j library calls -> com.nimbusds.oauth2.sdk.http.HTTPRequest.setContentType in the com.nimbusds:oauth2-oidc-sdk library that uses -> com/nimbusds/common/contenttype/ContentType from the com.nimbusds:content-type library.

Because com.nimbusds:oauth2-oidc-sdk:11.22.2 is a dependency of the security plugin it has the security plugin classloader, not the extension class loader, that's why it can't find ContentType when it's declared in the microsoft graph plugin.

For this to work com.nimbusds:oauth2-oidc-sdk would have to be declared in the microsoft graph plugin, to force it to use the correct classloader. Unfortunately that won't work becuase of jarhell so then our options are either shadowing com.nimbusds:oauth2-oidc-sdk or accepting that the microsoft graph plugin isn't standalone (it requires dependencies from the security plugin).

Since we will release a patch version of ES that the customer will have to upgrade to before installing the new plugin, it should be acceptable to have the content type dependency in the security plugin.

~~I did however find this: #76288 which sounds like something we might be able to use if we want to work around it.~~

jfreden · 2025-05-27T10:00:52Z

plugins/microsoft-graph-authz/src/main/java/module-info.java

+    requires com.microsoft.graph;
+    requires com.azure.identity;
+    requires com.microsoft.graph.core;
+    // FIXME both of these module includes are load bearing, because this project is cursed


The reason not including kotlin fails compile is because the compiler logs a warning:

warning: unknown enum constant DeprecationLevel.ERROR reason: class file for kotlin.DeprecationLevel not found

So if kotlin.stdlib is not included the annotations can't be resolved, which happens at compile time.

com.google.gson is only needed at runtime (no annotation that need to be resolved). Gson needs to be explicitly required because it's not a transitive dependency module of any of the microsoft libraries (they all get default JPMS automatic module so they do not declare any transitive dependencies).

jfreden · 2025-05-28T14:32:18Z

...src/main/java/org/elasticsearch/xpack/security/authz/microsoft/MicrosoftGraphAuthzRealm.java

+    private List<String> sdkFetchGroupMembership(GraphServiceClient client, String userId) throws ReflectiveOperationException {
+        List<String> groups = new ArrayList<>();
+
+        // TODO figure out exactly what we need to fetch here - we may need to fetch transitive groups as well, and may need to remove


The group attribute in SAML and OIDC includes the nested groups. See this

Group claims in tokens include nested groups, except when you're using the option to restrict the group claims to groups that are assigned to the application. If a user is a member of GroupB, and GroupB is a member of GroupA, then the group claims for the user will contain both GroupA and GroupB. When an organization's users have large numbers of group memberships, the number of groups listed in the token can grow the token size. Microsoft Entra ID limits the number of groups that it will emit in a token to 150 for SAML assertions and 200 for JWT. If a user is a member of a larger number of groups, the groups are omitted. A link to the Microsoft Graph endpoint to obtain group information is included instead.

This means that we should be using transitiveMemberOf rather than memberOf.

When using the user/transitiveMemberOf the API returns:

groups

directory roles

administrative units

While in the SAML/OIDC attribute you can choose:

all groups

security groups

directory roles

groups assigned to the application (a subset of the groups assigned to the user)

Default is all groups, so I think we should be using all groups. In the graph api that means groups, directory roles and we should filter out administrative units (probably by type).

In the SAML and OIDC settings source attribute to use as group ID can also be configured:

Group ID

sAMAccountName

NetBIOSDomain\sAMAccountName

DNSDomain\sAMAccountName

On Premises Group Security Identifier

Which one to use should probably be configurable. Let's confirm with the customer to see what the expectations are here.

Some additional information here:

I think it's safe to drop directory roles here too since they're administrator roles within Entra, not relevant to the customer application. This API call gives us only the group id for all groups (filtering out directory roles and administrative units):

<base_url>/users/<user>/transitiveMemberOf/microsoft.graph.group?$select=id

Still TBD if a separate config for group id attribute source is required or not.

…lementation-sdk

breskeby · 2025-06-11T11:16:16Z

x-pack/extras/plugins/build.gradle

+ */
+
+configurations {
+  allPlugins


We need to reference this configuration in our cloud-ess docker image build. Next to

elasticsearch/distribution/docker/build.gradle

Line 119 in 38c90ca

allPlugins project(path: ':plugins', configuration: 'allPlugins')

you need to add allPlugins project(path: ':x-pack:extras:plugins', configuration: 'allPlugins') so we take those plugins into account when packaging the cloud images

Thanks Rene! I've updated the build script.

…lementation-sdk

elasticsearchmachine · 2025-06-12T09:04:52Z

💔 Backport failed

Status	Branch	Result
❌	8.19	Commit could not be cherrypicked due to conflicts

You can use sqren/backport to manually backport by running backport --upstream elastic/elasticsearch --pr 128396

jfreden · 2025-06-12T11:37:39Z

💚 All backports created successfully

Status	Branch	Result
✅	8.19

Questions ?

Please refer to the Backport tool documentation

* Delegated authorization using Microsoft Graph (SDK) --------- Co-authored-by: elasticsearchmachine <[email protected]> Co-authored-by: Johannes Freden Jansson <[email protected]> Co-authored-by: Johannes Fredén <[email protected]> (cherry picked from commit 63da93d) # Conflicts: # gradle/verification-metadata.xml # libs/native/jna/src/main/java/module-info.java # plugins/microsoft-graph-authz/build.gradle # server/src/main/java/org/elasticsearch/search/aggregations/HasAggregations.java # settings.gradle # test/framework/src/main/java/org/elasticsearch/KnownTransportVersions.java # x-pack/extras/build.gradle # x-pack/extras/plugins/microsoft-graph-authz/src/main/java/org/elasticsearch/xpack/security/authz/microsoft/MicrosoftGraphAuthzPlugin.java # x-pack/extras/plugins/microsoft-graph-authz/src/main/resources/META-INF/services/org.elasticsearch.xpack.core.security.SecurityExtension # x-pack/plugin/security/qa/microsoft-graph-authz-tests/build.gradle # x-pack/plugin/security/qa/microsoft-graph-authz-tests/src/javaRestTest/java/org/elasticsearch/xpack/security/authz/microsoft/MicrosoftGraphAuthzPluginIT.java

* Delegated authorization using Microsoft Graph (SDK) --------- Co-authored-by: elasticsearchmachine <[email protected]> Co-authored-by: Johannes Freden Jansson <[email protected]> Co-authored-by: Johannes Fredén <[email protected]>

* propgating retrievers to inner retrievers * test feature taken care of * Small changes in concurrent multipart upload interfaces (#128977) Small changes in BlobContainer interface and wrapper. Relates ES-11815 * Unmute FollowingEngineTests#testProcessOnceOnPrimary() test (#129054) The reason the test fails is that operations contained _seq_no field with different doc value types (with no skippers and with skippers) and this isn't allowed, since field types need to be consistent in a Lucene index. The initial operations were generated not knowing about the fact the index mode was set to logsdb or time_series. Causing the operations to not have doc value skippers. However when replaying the operations via following engine, the operations did have doc value skippers. The fix is to set `index.seq_no.index_options` to `points_and_doc_values`, so that the initial operations are indexed without doc value skippers. This test doesn't gain anything from storing seqno with doc value skippers, so there is no loss of testing coverage. Closes #128541 * [Build] Add support for publishing to maven central (#128659) This ensures we package an aggregation zip with all artifacts we want to publish to maven central as part of a release. Running zipAggregation will produce a zip file in the build/nmcp/zip folder. The content of this zip is meant to match the maven artifacts we have currently declared as dra maven artifacts. * ESQL: Check for errors while loading blocks (#129016) Runs a sanity check after loading a block of values. Previously we were doing a quick check if assertions were enabled. Now we do two quick checks all the time. Better - we attach information about how a block was loaded when there's a problem. Relates to #128959 * Make `PhaseCacheManagementTests` project-aware (#129047) The functionality in `PhaseCacheManagement` was already project-aware, but these tests were still using deprecated methods. * Vector test tools (#128934) This adds some testing tools for verifying vector recall and latency directly without having to spin up an entire ES node and running a rally track. Its pretty barebones and takes inspiration from lucene-util, but I wanted access to our own formats and tooling to make our lives easier. Here is an example config file. This will build the initial index, run queries at num_candidates: 50, then again at num_candidates 100 (without reindexing, and re-using the cached nearest neighbors). ``` [{ "doc_vectors" : "path", "query_vectors" : "path", "num_docs" : 10000, "num_queries" : 10, "index_type" : "hnsw", "num_candidates" : 50, "k" : 10, "hnsw_m" : 16, "hnsw_ef_construction" : 200, "index_threads" : 4, "reindex" : true, "force_merge" : false, "vector_space" : "maximum_inner_product", "dimensions" : 768 }, { "doc_vectors" : "path", "query_vectors" : "path", "num_docs" : 10000, "num_queries" : 10, "index_type" : "hnsw", "num_candidates" : 100, "k" : 10, "hnsw_m" : 16, "hnsw_ef_construction" : 200, "vector_space" : "maximum_inner_product", "dimensions" : 768 } ] ``` To execute: ``` ./gradlew :qa:vector:checkVec --args="/Path/to/knn_tester_config.json" ``` Calling `./gradlew :qa:vector:checkVecHelp` gives some guidance on how to use it, additionally providing a way to run it via java directly (useful to bypass gradlew guff). * ES|QL: refactor generative tests (#129028) * Add a test of LOOKUP JOIN against a time series index (#129007) Add a spec test of `LOOKUP JOIN` against a time series index. * Make ILM `ClusterStateWaitStep` project-aware (#129042) This is part of an iterative process to make ILM project-aware. * Mute org.elasticsearch.xpack.esql.qa.mixed.MixedClusterEsqlSpecIT test {lookup-join.LookupJoinOnTimeSeriesIndex ASYNC} #129078 * Remove `ClusterState` param from ILM `AsyncBranchingStep` (#129076) The `ClusterState` parameter of the `asyncPredicate` is not used anywhere. * Mute org.elasticsearch.xpack.esql.qa.mixed.MixedClusterEsqlSpecIT test {lookup-join.LookupJoinOnTimeSeriesIndex SYNC} #129082 * Mute org.elasticsearch.upgrades.UpgradeClusterClientYamlTestSuiteIT test {p0=upgraded_cluster/70_ilm/Test Lifecycle Still There And Indices Are Still Managed} #129097 * Mute org.elasticsearch.upgrades.UpgradeClusterClientYamlTestSuiteIT test {p0=upgraded_cluster/90_ml_data_frame_analytics_crud/Get mixed cluster outlier_detection job} #129098 * Mute org.elasticsearch.packaging.test.DockerTests test081SymlinksAreFollowedWithEnvironmentVariableFiles #128867 * Threadpool merge executor is aware of available disk space (#127613) This PR introduces 3 new settings: indices.merge.disk.check_interval, indices.merge.disk.watermark.high, and indices.merge.disk.watermark.high.max_headroom that control if the threadpool merge executor starts executing new merges when the disk space is getting low. The intent of this change is to avoid the situation where in-progress merges exhaust the available disk space on the node's local filesystem. To this end, the thread pool merge executor periodically monitors the available disk space, as well as the current disk space estimates required by all in-progress (currently running) merges on the node, and will NOT schedule any new merges if the disk space is getting low (by default below the 5% limit of the total disk space, or 100 GB, whichever is smaller (same as the disk allocation flood stage level)). * Add option to include or exclude vectors from _source retrieval (#128735) This PR introduces a new include_vectors option to the _source retrieval context. When set to false, vectors are excluded from the returned _source. This is especially efficient when used with synthetic source, as it avoids loading vector fields entirely. By default, vectors remain included unless explicitly excluded. * Remove direct minScore propagation to inner retrievers * cleaned up skip * Mute org.elasticsearch.index.engine.ThreadPoolMergeExecutorServiceDiskSpaceTests testAvailableDiskSpaceMonitorWhenFileSystemStatErrors #129149 * Add transport version for ML inference Mistral chat completion (#129033) * Add transport version for ML inference Mistral chat completion * Add changelog for Mistral Chat Completion version fix * Revert "Add changelog for Mistral Chat Completion version fix" This reverts commit 7a57416. * Correct index path validation (#129144) All we care about is if reindex is true or false. We shouldn't worry about force merge. Because if reindex is true, we will create the directory, if its false, we won't. * Mute org.elasticsearch.index.engine.ThreadPoolMergeExecutorServiceDiskSpaceTests testUnavailableBudgetBlocksNewMergeTasksFromStartingExecution #129148 * Implemented completion task for Google VertexAI (#128694) * Google Vertex AI completion model, response entity and tests * Fixed GoogleVertexAiServiceTest for Service configuration * Changelog * Removed downcasting and using `moveToFirstToken` * Create GoogleVertexAiChatCompletionResponseHandler for streaming and non streaming responses * Added unit tests * PR feedback * Removed googlevertexaicompletion model. Using just GoogleVertexAiChatCompletionModel for completion and chat completion * Renamed uri -> nonStreamingUri. Added streamingUri and getters in GoogleVertexAiChatCompletionModel * Moved rateLimitGroupHashing to subclasses of GoogleVertexAiModel * Fixed rate limit has of GoogleVertexAiRerankModel and refactored uri for GoogleVertexAiUnifiedChatCompletionRequest --------- Co-authored-by: lhoet-google <[email protected]> Co-authored-by: Jonathan Buttner <[email protected]> * ES|QL - kNN function initial support (#127322) * Remove optional seed from ES|QL SAMPLE (#128887) * Remove optional seed from ES|QL SAMPLE * make it clear that seed is for testing * [Inference API] Add "rerank" task type to "elastic" provider (#126022) * Rename target destination for microbenchmarks (#128878) * Include direct memory and non-heap memory in ML memory calculations (take #2) (#128742) * Include direct memory and non-heap memory in ML memory calculations. * Reduce ML_ONLY heap size, so that direct memory is accounted for. * [CI] Auto commit changes from spotless * changelog * improve docs * Reuse direct memory to heap factor * feature flag --------- Co-authored-by: elasticsearchmachine <[email protected]> * Throw better exception for unsupported aggregations over shape fields (#129139) * Update Test Framework To Handle Query Rewrites That Rely on Non-Null Searchers (#129160) * Update ReproduceInfoPrinter to correctly print a reproduction line for Lucene & build candidate upgrade tests (#129044) * Increment inference stats counter for shard bulk inference calls (#129140) This change updates the inference stats counter to include chunked inference calls performed by the shard bulk inference filter on all semantic text fields. It ensures that usage of inference on semantic text fields is properly recorded in the stats. * Synthetic source: avoid storing multi fields of type text and match_only_text by default. (#129126) Don't store text and match_only_text field by default when source mode is synthetic and a field is a multi field or when there is a suitable multi field. Without this change, ES would store field otherwise twice in a multi-field configuration. For example: ``` ... "os": { "properties": { "name": { "ignore_above": 1024, "type": "keyword", "fields": { "text": { "type": "match_only_text" } } } ... ``` In this case, two stored fields were added, one in case for the `name` field and one for `name.text` multi-field. This change prevents this, and would never store a stored field when text or match_only_text field is a multi-field. * Adding `scheduled_report_id` field to kibana reporting template (#127827) * Adding scheduled_report_id field to kibana reporting template * Incrementing stack template registry version * ES|QL: Add FORK generative tests (#129135) * ES|QL Completion command syntax change (#129189) * Remove optional seed from ES|QL SAMPLE (#128887) * Remove optional seed from ES|QL SAMPLE * make it clear that seed is for testing * ES|QL Completion command syntax change (#129189) * Remove optional seed from ES|QL SAMPLE (#128887) * Remove optional seed from ES|QL SAMPLE * make it clear that seed is for testing * ES|QL Completion command syntax change (#129189) * Add Cluster Feature for L2 Norm (#129181) * propgating retrievers to inner retrievers * test feature taken care of * Small changes in concurrent multipart upload interfaces (#128977) Small changes in BlobContainer interface and wrapper. Relates ES-11815 * Unmute FollowingEngineTests#testProcessOnceOnPrimary() test (#129054) The reason the test fails is that operations contained _seq_no field with different doc value types (with no skippers and with skippers) and this isn't allowed, since field types need to be consistent in a Lucene index. The initial operations were generated not knowing about the fact the index mode was set to logsdb or time_series. Causing the operations to not have doc value skippers. However when replaying the operations via following engine, the operations did have doc value skippers. The fix is to set `index.seq_no.index_options` to `points_and_doc_values`, so that the initial operations are indexed without doc value skippers. This test doesn't gain anything from storing seqno with doc value skippers, so there is no loss of testing coverage. Closes #128541 * [Build] Add support for publishing to maven central (#128659) This ensures we package an aggregation zip with all artifacts we want to publish to maven central as part of a release. Running zipAggregation will produce a zip file in the build/nmcp/zip folder. The content of this zip is meant to match the maven artifacts we have currently declared as dra maven artifacts. * ESQL: Check for errors while loading blocks (#129016) Runs a sanity check after loading a block of values. Previously we were doing a quick check if assertions were enabled. Now we do two quick checks all the time. Better - we attach information about how a block was loaded when there's a problem. Relates to #128959 * Make `PhaseCacheManagementTests` project-aware (#129047) The functionality in `PhaseCacheManagement` was already project-aware, but these tests were still using deprecated methods. * Vector test tools (#128934) This adds some testing tools for verifying vector recall and latency directly without having to spin up an entire ES node and running a rally track. Its pretty barebones and takes inspiration from lucene-util, but I wanted access to our own formats and tooling to make our lives easier. Here is an example config file. This will build the initial index, run queries at num_candidates: 50, then again at num_candidates 100 (without reindexing, and re-using the cached nearest neighbors). ``` [{ "doc_vectors" : "path", "query_vectors" : "path", "num_docs" : 10000, "num_queries" : 10, "index_type" : "hnsw", "num_candidates" : 50, "k" : 10, "hnsw_m" : 16, "hnsw_ef_construction" : 200, "index_threads" : 4, "reindex" : true, "force_merge" : false, "vector_space" : "maximum_inner_product", "dimensions" : 768 }, { "doc_vectors" : "path", "query_vectors" : "path", "num_docs" : 10000, "num_queries" : 10, "index_type" : "hnsw", "num_candidates" : 100, "k" : 10, "hnsw_m" : 16, "hnsw_ef_construction" : 200, "vector_space" : "maximum_inner_product", "dimensions" : 768 } ] ``` To execute: ``` ./gradlew :qa:vector:checkVec --args="/Path/to/knn_tester_config.json" ``` Calling `./gradlew :qa:vector:checkVecHelp` gives some guidance on how to use it, additionally providing a way to run it via java directly (useful to bypass gradlew guff). * ES|QL: refactor generative tests (#129028) * Add a test of LOOKUP JOIN against a time series index (#129007) Add a spec test of `LOOKUP JOIN` against a time series index. * Make ILM `ClusterStateWaitStep` project-aware (#129042) This is part of an iterative process to make ILM project-aware. * Mute org.elasticsearch.xpack.esql.qa.mixed.MixedClusterEsqlSpecIT test {lookup-join.LookupJoinOnTimeSeriesIndex ASYNC} #129078 * Remove `ClusterState` param from ILM `AsyncBranchingStep` (#129076) The `ClusterState` parameter of the `asyncPredicate` is not used anywhere. * Mute org.elasticsearch.xpack.esql.qa.mixed.MixedClusterEsqlSpecIT test {lookup-join.LookupJoinOnTimeSeriesIndex SYNC} #129082 * Mute org.elasticsearch.upgrades.UpgradeClusterClientYamlTestSuiteIT test {p0=upgraded_cluster/70_ilm/Test Lifecycle Still There And Indices Are Still Managed} #129097 * Mute org.elasticsearch.upgrades.UpgradeClusterClientYamlTestSuiteIT test {p0=upgraded_cluster/90_ml_data_frame_analytics_crud/Get mixed cluster outlier_detection job} #129098 * Mute org.elasticsearch.packaging.test.DockerTests test081SymlinksAreFollowedWithEnvironmentVariableFiles #128867 * Threadpool merge executor is aware of available disk space (#127613) This PR introduces 3 new settings: indices.merge.disk.check_interval, indices.merge.disk.watermark.high, and indices.merge.disk.watermark.high.max_headroom that control if the threadpool merge executor starts executing new merges when the disk space is getting low. The intent of this change is to avoid the situation where in-progress merges exhaust the available disk space on the node's local filesystem. To this end, the thread pool merge executor periodically monitors the available disk space, as well as the current disk space estimates required by all in-progress (currently running) merges on the node, and will NOT schedule any new merges if the disk space is getting low (by default below the 5% limit of the total disk space, or 100 GB, whichever is smaller (same as the disk allocation flood stage level)). * Add option to include or exclude vectors from _source retrieval (#128735) This PR introduces a new include_vectors option to the _source retrieval context. When set to false, vectors are excluded from the returned _source. This is especially efficient when used with synthetic source, as it avoids loading vector fields entirely. By default, vectors remain included unless explicitly excluded. * Remove direct minScore propagation to inner retrievers * cleaned up skip * Mute org.elasticsearch.index.engine.ThreadPoolMergeExecutorServiceDiskSpaceTests testAvailableDiskSpaceMonitorWhenFileSystemStatErrors #129149 * Add transport version for ML inference Mistral chat completion (#129033) * Add transport version for ML inference Mistral chat completion * Add changelog for Mistral Chat Completion version fix * Revert "Add changelog for Mistral Chat Completion version fix" This reverts commit 7a57416. * Correct index path validation (#129144) All we care about is if reindex is true or false. We shouldn't worry about force merge. Because if reindex is true, we will create the directory, if its false, we won't. * Mute org.elasticsearch.index.engine.ThreadPoolMergeExecutorServiceDiskSpaceTests testUnavailableBudgetBlocksNewMergeTasksFromStartingExecution #129148 * Implemented completion task for Google VertexAI (#128694) * Google Vertex AI completion model, response entity and tests * Fixed GoogleVertexAiServiceTest for Service configuration * Changelog * Removed downcasting and using `moveToFirstToken` * Create GoogleVertexAiChatCompletionResponseHandler for streaming and non streaming responses * Added unit tests * PR feedback * Removed googlevertexaicompletion model. Using just GoogleVertexAiChatCompletionModel for completion and chat completion * Renamed uri -> nonStreamingUri. Added streamingUri and getters in GoogleVertexAiChatCompletionModel * Moved rateLimitGroupHashing to subclasses of GoogleVertexAiModel * Fixed rate limit has of GoogleVertexAiRerankModel and refactored uri for GoogleVertexAiUnifiedChatCompletionRequest --------- Co-authored-by: lhoet-google <[email protected]> Co-authored-by: Jonathan Buttner <[email protected]> * Added cluster feature to yaml * Node feature added * Duplicate line - result of merge removed * Update docs/changelog/129181.yaml * Update 129181.yaml --------- Co-authored-by: Tanguy Leroux <[email protected]> Co-authored-by: Martijn van Groningen <[email protected]> Co-authored-by: Rene Groeschke <[email protected]> Co-authored-by: Nik Everett <[email protected]> Co-authored-by: Niels Bauman <[email protected]> Co-authored-by: Benjamin Trent <[email protected]> Co-authored-by: Luigi Dell'Aquila <[email protected]> Co-authored-by: Bogdan Pintea <[email protected]> Co-authored-by: elasticsearchmachine <[email protected]> Co-authored-by: Albert Zaharovits <[email protected]> Co-authored-by: Jim Ferenczi <[email protected]> Co-authored-by: Jan-Kazlouski-elastic <[email protected]> Co-authored-by: Leonardo Hoet <[email protected]> Co-authored-by: lhoet-google <[email protected]> Co-authored-by: Jonathan Buttner <[email protected]> * Fix DRA dependenciesInfo task dependency resolution (#129209) * IVF Hierarchical KMeans Flush & Merge (#128675) added hierarchical kmeans as a clustering algorithm to better partitionin the space when running ivf on flush and merge * Mute org.elasticsearch.xpack.esql.qa.single_node.GenerativeForkIT test {lookup-join.EnrichLookupStatsBug ASYNC} #129228 * Mute org.elasticsearch.xpack.esql.qa.single_node.GenerativeForkIT test {lookup-join.EnrichLookupStatsBug SYNC} #129229 * [ES|QL] Specify population in StdDev docs (#129225) There are 2 types of Standard Deviation: population and sample, this commit clarifies that the existing is population. * Unmute IngestGeoIpClientYamlTestSuiteIT (#129178) * Fix an NPE in the ES|QL completion command. (#129235) * ESQL: fix bwc test by adding min required version (#129204) Closes #129093 Closes #129094 Closes #129095 Closes #129102 Closes #129103 * ESQL: Fix test by add excluding capability (#129202) Closes #129078 Closes #129082 * Fix vault field name (#129184) * Remove all usages of Metadata customs removal methods (#129043) This removes all non-test usage of ``` Metadata.Builder.removeProjectCustom(String) Metadata.Builder.removeProjectCustomIf(BiPredicate) ``` And replaces it with appropriate calls to the equivalent method on `ProjectMetadata.Builder` In most cases this _does not_ make the code project aware, but does reduce the number of deprecated methods in use. * Replace tuple with record (#128976) * improve support for bytecode patching signed jars (#128613) * improve support for bytecode patching signed jars * Update docs/changelog/128613.yaml --------- Co-authored-by: elasticsearchmachine <[email protected]> Co-authored-by: Johannes Freden Jansson <[email protected]> * rename ES|QL sample capability (#129193) * ESQL: Mute GenerativeForkIT for some LOOKUP JOIN tests (#129248) * ESQL: Extend `RENAME` syntax to allow a `new = old` syntax (#129212) This extends RENAME's grammar to allow a new syntax: `| RENAME new_name = old_name` This is supported along the existing `... old_name AS new_name` syntax. Closes #129208 * [DOCS] Adds preview tag to the CHANGE_POINT ES|QL command in the command list. (#129247) * ESQL: Skip unused STATS groups by adding a Top N BlockHash implementation (#127148) - Add a new `LongTopNBlockHash` implementation taking care of skipping unused values. - Add a `TopNUniqueSet` to take care of storing the top N values (without nulls). - Add a `TopNMultivalueDedupeLong` class helping with it (An adaptation of the existing `MultivalueDedupeLong`). - Add some tests to `HashAggregationOperator`. It wasn't changed much, but helps a bit with the E2E. - Add MicroBenchmarks for TopN groupings, to ensure we're actually improving things with this. * Add "Searchable Snapshots" to changelog validation schema (#129180) We created a new ":Distributed Indexing/Searchable Snapshots" label recently on Github, so I think it makes sense to also have a "Searchable Snapshots" label in the changelog. It also makes sense since there is automatic changelog generation based on the pull request label. * ESQL: Fix FieldAttribute name usage in InferNonNullAggConstraint (#128910) * Fix InferNonNullAggConstraint with union types * Begin fixing LucenePushdownPredicates with union types * Introduce a dedicated wrapper record FieldName to be used where field names are really required. The fixes consist of using FieldAttribute.fieldName() instead of .name() or .field().name(). .name() can be some temporary string unrelated to the actual name of the Lucene index field, whereas .field().name() doesn't know about parent fields; .fieldName() gives the full field name (from the root of the document). The biggest offender of such misuse is SearchStats; make this always require a FieldName, not a String - and make FieldAttribute#fieldName handily return an instance of FieldName so users of SearchStats don't accidentally use the return value of FieldAttribute#name. * Remove usages of `Metadata.Builder#indexGraveyard` (#129041) And replace it with appropriate calls to the equivalent method on `ProjectMetadata.Builder`. In most cases this _does not_ make the code project aware, but does reduce the number of deprecated methods in use. Concerns both the getter and the setter. * Mute org.elasticsearch.compute.data.sort.LongTopNSetTests testCrankyBreaker #129257 * Enable Shard-Level Search-load rate metric (#128660) Introduces a new search load metric to the stats infrastructure, measured and tracked on a per-shard basis. The metric represents the Exponentially Weighted Moving Rate (EWMR) of search operations, calculated using the "took" time from each completed search phase. * [ESQL] Fix typo in search-functions.md (#129260) ^^ * ESQL: Log partial failures (#129164) Now that ESQL has `allow_partial_results` we can reply with a `200` even though some nodes failed to run ESQL. This could happen because the node is restarting. Or because of a bug. Or a disconnect. All kinds of things. This logs those partial failures so an operator can look at them and get a sense of why they are happening. * Update Gradle wrapper to 8.14.2 (#129179) * Fix ivf nodestats impl for getOffHeapByteSize (#129259) This fixes a silly bug where we didn't override `OffHeapStats` for IVF. * feat: enable date_detection for all apm data streams (#128913) * feat: enable date_detection for all apm data streams * Update resources.yaml * Create 128913.yml --------- Co-authored-by: Carson Ip <[email protected]> * [BC Upgrage] Fix incorrect version parsing in tests (#129243) This PR introduces several fixes to various IT tests, related to the use and misuse of the version identifier for the start cluster: wherever we can, we replace of versions in test code with features where we can't, we make sure we use the actual stack version (the one provided by -Dtests.bwc.main.version and not the bogus "0.0.0" version string) when requesting the cluster version we make sure we do use the "unresolved" version identifier (the value of the tests.old_cluster_version system property e.g. 0.0.0 ) so we resolve the right distribution These changes enabled the tests to be used in BC upgrade tests (and potentially in serverless upgrade tests too, where they would have also failed) Relates to ES-12010 Precedes #128614, #128823 and #128983 * [Build] Build maven aggregation zip as part of DRA build (#129175) * [Build] Build maven aggregation zip as part of DRA build * Update path for aggregation zip * Throttle indexing when disk IO throttling is disabled (#129245) The threadpool-based merge scheduler triggers indexing throttling if merges are still getting enqueued faster than they're executed, while they are also disk IO unthrottled. This PR fixes the case where indexing throttling was incorrectly NOT triggered when disk IO throttling was disabled via the index settings. * Register match_phrase as a function not a snapshot function (#129255) * Register match_phrase as a function not a snapshot function * Update usage * [Gradle] Spotless plugin update (#115750) - provides better configuration cache support - requires some rework due to changed defaults * Adding support to exclude semantic_text subfields (#127664) * Adding support to exclude semantic_text subfields * Update docs/changelog/127664.yaml * Updating changelog file * remove duplicate test from yaml file * Adding support to exclude semantic_text subfields from mapper builders * Adding support for generic field types * refactoring to use builder and setting exclude value from semantic_text mapper * update in semantic_text mapper and fetcher to incorporate the support functionality * Fix code style issue * adding node feature for yaml tests * Adding more restrictive checks on yaml tests and few refactoring * Returns metadata fields from metadata mappers * returns all source fields for fieldcaps * gather all fields and iterate to process for fieldcaps api * revert back all changes from MappedFieldtype and subclasses * revert back exclude logic from semantic_text mapper * fix lint issues * fix lint issues * Adding runtime fields into fieldCaps * Fix linting issue * removing unused functions that used in previous implementation * fix multifield tests failure * getting alias fields for field caps * adding support for query time runtime fields * [CI] Auto commit changes from spotless * Fix empty mapping fieldCaps call * Address passthrough behavior for mappers * Fix SearchAsYoutype mapper failures * rename abstract method to have more meaningful name * Rename mapper function to match its functionality * Adding filtering for infernece subfields * revert back previous implementation changes * Adding yaml test for field caps not filtering multi-field * Fixing yaml test * Adding comment why .infernece filter is added --------- Co-authored-by: elasticsearchmachine <[email protected]> Co-authored-by: Elastic Machine <[email protected]> * Revert "[Gradle] Spotless plugin update (#115750)" This reverts commit 6370d60. * Switch IVF Writer to ES Logger (#129224) update to use ES logger instead of infostream and fixing native access warnings * Add heap usage estimate to ClusterInfo (#128723) Co-authored-by: ywangd <[email protected]> Co-authored-by: rjernst <[email protected]> Relates: ES-11445 * Revert "Use IndexOrDocValuesQuery in NumberFieldType#termQuery implementations (#128293)" (#129206) This reverts commit de7c91c. * Delegated authorization using Microsoft Graph (SDK) (#128396) * Delegated authorization using Microsoft Graph (SDK) --------- Co-authored-by: elasticsearchmachine <[email protected]> Co-authored-by: Johannes Freden Jansson <[email protected]> Co-authored-by: Johannes Fredén <[email protected]> * Add `none` chunking strategy to disable automatic chunking for inference endpoints (#129150) This introduces a `none` chunking strategy that disables automatic chunking when using an inference endpoint. It enables users to provide pre-chunked input directly to a `semantic_text` field without any additional splitting. The chunking strategy can be configured either on the inference endpoint or directly in the `semantic_text` field definition. **Example:** ```json PUT test-index { "mappings": { "properties": { "my_semantic_field": { "type": "semantic_text", "chunking_settings": { "strategy": "none" <1> } } } } } ``` <1> Disables automatic chunking on `my_semantic_field`. ```json PUT test-index/_doc/1 { "my_semantic_field": ["my first chunk", "my second chunk", ...] <1> ... } ``` <1> Pre-chunked input provided as an array of strings. Each array element represents a single chunk that will be sent directly to the inference service without further processing. * Fix broken bwc logic in text field mapper introduced by #129126 (#129308) A missing condition in the bwc logic caused a text field to be a stored, while before #129126, this wasn't the case. * [ESQL] Fix SpatialDocValuesExtraction rule replacing TimeSeries agg node (#129273) `TimeSeriesAggregateExec` (TS) node inherits from `AggregateExec` (STATS). The `SpatialDocValuesExtraction` rule was replacing all `AggregateExec` instances with another `AggregateExec`, whether the same class or not. * Make `TransportMoveToStepAction` project-aware (#129252) Future work is necessary to make the YAML tests pass in MP mode. * [DOCS] Adds term vectors API examples (#129328) * [DOCS] Adds term vectors API examples. * Update docs/reference/elasticsearch/rest-apis/term-vectors-examples.md Co-authored-by: Liam Thompson <[email protected]> * [DOCS] Addresses feedback. * [DOCS] Fixes link. --------- Co-authored-by: Liam Thompson <[email protected]> * [ESQL] Fix TopNSetTestCase test and unmute it (#129327) Closes #129257 * ESQL: Change queries ID to be the same as the async (#127472) This PR changes the list and query API for ESQL, such that the ID now follows the same format as async query IDs. This is saved as part of the task status. For async queries, this is easy, but for sync queries, this is slightly more complicated, since when creating them, we don't have access to a node ID. So instead, the status itself is just the doc ID portion of the async execution ID, which is used for salting, since this part needs to be consistent, so that when we list the queries, we can compute the async execution ID correctly. Also, I've removed the individual ID, node, and data node tags, as mentioned in the ticket. In addition, I've changed the accept and content-type to be JSON for lists. Resolves #127187 * Adjust unpromotable shard refresh request validation to allow RefreshResult.NO_REFRESH (#129176) When a primary shard uses the read-only engine, it always returns a RefreshResult.NO_REFRESH for refreshes. Since #93600 we added an extra roundtrip to hook unpromotable shard refresh logic. This hook is always executed, even if there are no unpromotable shards, but the UnpromotableShardRefreshRequest would fail if the primary shard returns a RefreshResult.NO_REFRESH result. Fix to be backported to several versions as it's annoying. Closes #129036 * Add a Multi-Project Search Rest Test (#128657) This commit adds a Rest IT specifically for search in MultiProject. Everything was already working as expected, but we were a bit light on explicit testing for search, which as _the_ core capability of Elasticsearch is worth testing thoroughly and clearly. * Modified LinearRetriever to include minScore * cleaned up * Made the same changes we did in textSimilarity * Fixed a minor error * cleaned up * Minscore is working :) * chore: empty commit to trigger PR update * Update docs/changelog/129359.yaml * Update 10_linear_retriever.yml * [CI] Auto commit changes from spotless --------- Co-authored-by: Tanguy Leroux <[email protected]> Co-authored-by: Martijn van Groningen <[email protected]> Co-authored-by: Rene Groeschke <[email protected]> Co-authored-by: Nik Everett <[email protected]> Co-authored-by: Niels Bauman <[email protected]> Co-authored-by: Benjamin Trent <[email protected]> Co-authored-by: Luigi Dell'Aquila <[email protected]> Co-authored-by: Bogdan Pintea <[email protected]> Co-authored-by: elasticsearchmachine <[email protected]> Co-authored-by: Albert Zaharovits <[email protected]> Co-authored-by: Jim Ferenczi <[email protected]> Co-authored-by: Jan-Kazlouski-elastic <[email protected]> Co-authored-by: Leonardo Hoet <[email protected]> Co-authored-by: lhoet-google <[email protected]> Co-authored-by: Jonathan Buttner <[email protected]> Co-authored-by: Carlos Delgado <[email protected]> Co-authored-by: Jan Kuipers <[email protected]> Co-authored-by: Tim Grein <[email protected]> Co-authored-by: Ievgen Degtiarenko <[email protected]> Co-authored-by: elasticsearchmachine <[email protected]> Co-authored-by: Ignacio Vera <[email protected]> Co-authored-by: Mike Pellegrini <[email protected]> Co-authored-by: Moritz Mack <[email protected]> Co-authored-by: Ying Mao <[email protected]> Co-authored-by: Ioana Tagirta <[email protected]> Co-authored-by: Aurélien FOUCRET <[email protected]> Co-authored-by: John Wagster <[email protected]> Co-authored-by: Larisa Motova <[email protected]> Co-authored-by: Sam Xiao <[email protected]> Co-authored-by: Richard Dennehy <[email protected]> Co-authored-by: Johannes Freden Jansson <[email protected]> Co-authored-by: Alexander Spies <[email protected]> Co-authored-by: István Zoltán Szabó <[email protected]> Co-authored-by: Iván Cea Fontenla <[email protected]> Co-authored-by: Dimitris Rempapis <[email protected]> Co-authored-by: Liam Thompson <[email protected]> Co-authored-by: kruskall <[email protected]> Co-authored-by: Carson Ip <[email protected]> Co-authored-by: Lorenzo Dematté <[email protected]> Co-authored-by: Kathleen DeRusso <[email protected]> Co-authored-by: Samiul Monir <[email protected]> Co-authored-by: Elastic Machine <[email protected]> Co-authored-by: Nick Tindall <[email protected]> Co-authored-by: ywangd <[email protected]> Co-authored-by: rjernst <[email protected]> Co-authored-by: Johannes Fredén <[email protected]> Co-authored-by: Gal Lalouche <[email protected]> Co-authored-by: Tim Vernum <[email protected]>

…129330) * Delegated authorization using Microsoft Graph (SDK) (#128396) * Delegated authorization using Microsoft Graph (SDK) --------- Co-authored-by: elasticsearchmachine <[email protected]> Co-authored-by: Johannes Freden Jansson <[email protected]> Co-authored-by: Johannes Fredén <[email protected]> (cherry picked from commit 63da93d) # Conflicts: # gradle/verification-metadata.xml # libs/native/jna/src/main/java/module-info.java # plugins/microsoft-graph-authz/build.gradle # server/src/main/java/org/elasticsearch/search/aggregations/HasAggregations.java # settings.gradle # test/framework/src/main/java/org/elasticsearch/KnownTransportVersions.java # x-pack/extras/build.gradle # x-pack/extras/plugins/microsoft-graph-authz/src/main/java/org/elasticsearch/xpack/security/authz/microsoft/MicrosoftGraphAuthzPlugin.java # x-pack/extras/plugins/microsoft-graph-authz/src/main/resources/META-INF/services/org.elasticsearch.xpack.core.security.SecurityExtension # x-pack/plugin/security/qa/microsoft-graph-authz-tests/build.gradle # x-pack/plugin/security/qa/microsoft-graph-authz-tests/src/javaRestTest/java/org/elasticsearch/xpack/security/authz/microsoft/MicrosoftGraphAuthzPluginIT.java * fixup! Rebase issue * fixup! Merge issue * fixup! Publish needed test artifact * fixup! Saml artifact and remove multi-project override * fixup! Add missing saml certs --------- Co-authored-by: Richard Dennehy <[email protected]>

richard-dennehy and others added 11 commits May 19, 2025 15:47

basic MS graph authz implementation

a4d85f6

[CI] Auto commit changes from spotless

cfd2906

move client_secret to keystore

9182858

use JSONObjectUtils for parsing json

34b6276

(not working) attempt to use official graph SDK

11cd43a

WIP at least it runs; if only it didn't mysteriously hang

aae596e

WIP unit tests (for debugging the plugin)

5c337b2

WIP at least it works against real azure

108f462

document assorted hacks required to get this thing working

c39c3f6

passing IT tests

5984ada

slight clean up

219aa17

elasticsearchmachine added the v9.1.0 label May 23, 2025

elasticsearchmachine and others added 2 commits May 23, 2025 15:29

[CI] Auto commit changes from spotless

c230ed6

update comment about mystery includes to be more accurate

6d907e5

richard-dennehy commented May 23, 2025

View reviewed changes

...src/main/java/org/elasticsearch/xpack/security/authz/microsoft/MicrosoftGraphAuthzRealm.java Outdated Show resolved Hide resolved

richard-dennehy commented May 23, 2025

View reviewed changes

[CI] Auto commit changes from spotless

0b9a449

jfreden reviewed May 27, 2025

View reviewed changes

richard-dennehy and others added 5 commits May 27, 2025 11:36

clean up some TODOs

451a553

[CI] Auto commit changes from spotless

85d123a

implement unit tests

a0bba42

test retry handling

bec682e

[CI] Auto commit changes from spotless

3798b98

jfreden reviewed May 28, 2025

View reviewed changes

richard-dennehy and others added 6 commits May 28, 2025 16:46

fetch transitive group membership

4552b3e

[CI] Auto commit changes from spotless

d1c22bf

support multiple test users in graph http fixture

ab0c69d

[CI] Auto commit changes from spotless

227e0d9

move manifest patching into Utils.patchJar

668c441

add test for concurrent login

be296be

jfreden and others added 5 commits June 9, 2025 11:46

fixup! Test issue

35e9ce4

Merge branch 'main' into microsoft-graph-implementation-sdk

02feaa0

fixup! Test

4c2e54d

Merge remote-tracking branch 'upstream/main' into microsoft-graph-imp…

bfd2e09

…lementation-sdk

Move plugin to new x-pack home

e5e12ca

jfreden added auto-backport Automatically create backport pull requests when merged v8.19.0 labels Jun 11, 2025

Merge remote-tracking branch 'upstream/main' into microsoft-graph-imp…

d753954

…lementation-sdk

breskeby requested changes Jun 11, 2025

View reviewed changes

fixup! Code review comment

35d2d62

jfreden added the cloud-deploy Publish cloud docker image for Cloud-First-Testing label Jun 11, 2025

jfreden added 4 commits June 11, 2025 13:54

fixup! Add build.gradle

78388f2

fixup! File extension

b134b33

fixup! File extension

b3a9ff9

fixup! File extension

6a41e78

breskeby approved these changes Jun 11, 2025

View reviewed changes

jfreden and others added 4 commits June 11, 2025 15:20

fixup! License headers

bdf0784

Merge remote-tracking branch 'upstream/main' into microsoft-graph-imp…

2a9ba66

…lementation-sdk

fixup! Race condition in test

8a76115

Merge branch 'main' into microsoft-graph-implementation-sdk

6d1821a

jfreden merged commit 63da93d into elastic:main Jun 12, 2025
23 of 24 checks passed

elasticsearchmachine added the backport pending label Jun 12, 2025

This was referenced Jun 12, 2025

Partial Backport of Microsoft Graph Plugin Dependencies #129318

Merged

[8.19] Delegated authorization using Microsoft Graph (SDK) (#128396) #129330

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Delegated authorization using Microsoft Graph (SDK) #128396

Delegated authorization using Microsoft Graph (SDK) #128396

Uh oh!

richard-dennehy commented May 23, 2025 •

edited

Loading

Uh oh!

Uh oh!

richard-dennehy May 23, 2025

Uh oh!

jfreden May 26, 2025 •

edited

Loading

Uh oh!

jfreden May 26, 2025 •

edited

Loading

Uh oh!

jfreden May 27, 2025

Uh oh!

jfreden May 27, 2025 •

edited

Loading

Uh oh!

jfreden May 27, 2025

Uh oh!

jfreden May 28, 2025

Uh oh!

jfreden May 30, 2025

Uh oh!

breskeby Jun 11, 2025

Uh oh!

jfreden Jun 11, 2025

Uh oh!

Uh oh!

elasticsearchmachine commented Jun 12, 2025

Uh oh!

jfreden commented Jun 12, 2025

Uh oh!

Uh oh!

Delegated authorization using Microsoft Graph (SDK) #128396

Delegated authorization using Microsoft Graph (SDK) #128396

Uh oh!

Conversation

richard-dennehy commented May 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

richard-dennehy May 23, 2025

Choose a reason for hiding this comment

Uh oh!

jfreden May 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jfreden May 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jfreden May 27, 2025

Choose a reason for hiding this comment

Uh oh!

jfreden May 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jfreden May 27, 2025

Choose a reason for hiding this comment

Uh oh!

jfreden May 28, 2025

Choose a reason for hiding this comment

Uh oh!

jfreden May 30, 2025

Choose a reason for hiding this comment

Uh oh!

breskeby Jun 11, 2025

Choose a reason for hiding this comment

Uh oh!

jfreden Jun 11, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

elasticsearchmachine commented Jun 12, 2025

💔 Backport failed

Uh oh!

jfreden commented Jun 12, 2025

💚 All backports created successfully

Questions ?

Uh oh!

Uh oh!

richard-dennehy commented May 23, 2025 •

edited

Loading

jfreden May 26, 2025 •

edited

Loading

jfreden May 26, 2025 •

edited

Loading

jfreden May 27, 2025 •

edited

Loading