From 0af647e39bf34943a78fb831f8856d81f4d89a0f Mon Sep 17 00:00:00 2001 From: Michael Peterson Date: Wed, 18 Dec 2024 12:17:12 -0500 Subject: [PATCH 01/10] Clarify the behavior of remote/info and resolve/cluster for connected status of remotes --- docs/reference/cluster/remote-info.asciidoc | 14 ++++++++++++- .../indices/resolve-cluster.asciidoc | 21 +++++++++++++++---- 2 files changed, 30 insertions(+), 5 deletions(-) diff --git a/docs/reference/cluster/remote-info.asciidoc b/docs/reference/cluster/remote-info.asciidoc index 8f2923846df0e..08b48407b830c 100644 --- a/docs/reference/cluster/remote-info.asciidoc +++ b/docs/reference/cluster/remote-info.asciidoc @@ -25,6 +25,15 @@ The cluster remote info API allows you to retrieve all of the configured remote cluster information. It returns connection and endpoint information keyed by the configured remote cluster alias. +TIP: This endpoint returns the information that reflects state on the cluster +you are querying. The `connected` field reports whether the querying cluster is +currently connected to the remote cluster. It does not necessarily reflect whether +the remote cluster is down or unavailable, only whether there is currently an open +connection to it. Elasticsearch does not spontaneously try to reconnect to a +disconnected remote cluster. To trigger a reconnection, attempt a +<>, <>, +or try the <> endpoint. + [[cluster-remote-info-api-response-body]] ==== {api-response-body-title} @@ -34,7 +43,10 @@ by the configured remote cluster alias. `proxy`. `connected`:: - True if there is at least one connection to the remote cluster. + True if there is at least one open connection to the remote cluster. When + false, it means that the cluster no longer has an open connection to the + remote cluster. It does not necessarily mean that the remote cluster is + down or unavailable, just that at some point a connection was lost. `initial_connect_timeout`:: The initial connect timeout for remote cluster connections. diff --git a/docs/reference/indices/resolve-cluster.asciidoc b/docs/reference/indices/resolve-cluster.asciidoc index 48e6bfac4af10..193b10bd3c60d 100644 --- a/docs/reference/indices/resolve-cluster.asciidoc +++ b/docs/reference/indices/resolve-cluster.asciidoc @@ -16,8 +16,9 @@ with this endpoint. For each cluster in the index expression, information is returned about: -1. whether the querying ("local") cluster is currently connected to each remote cluster - in the index expression scope +1. whether the querying ("local") cluster was able to connect to each remote cluster + specified in the index expression. Note that this endpoint actively attempts to + contact the remote clusters, unlike the <> endpoint. 2. whether each remote cluster is configured with `skip_unavailable` as `true` or `false` 3. whether there are any indices, aliases or data streams on that cluster that match the index expression @@ -110,8 +111,21 @@ ignored when frozen. Defaults to `false`. deprecated:[7.16.0] -[discrete] [[usecases-for-resolve-cluster]] + +=== Test availability of remote clusters + +The <> endpoint is commonly used to test whether the "local" +cluster (the cluster being queried) is connected to its remote clusters, but it does not +necessarily reflect whether the remote cluster is available or not. The remote cluster may +be available, while the local cluster is not currently connected to it. + +You can use this API to force a reconnection (for example with `GET _resolve/cluster/*:*`) and +examine the `connected` field in the response for each remote. This will also cause the +<> endpoint to now reflect the fact that a connection has +been (re-)established. + + === Advantages of using this endpoint before a {ccs} You may want to exclude a cluster or index from a search when: @@ -195,7 +209,6 @@ The API returns the following response: <5> The Elasticsearch server version. -[discrete] [[resolve-cluster-api-error-example]] ==== Identifying potential problems with your {ccs} From 4e08de422781a62d95b8b523a40be9a992854716 Mon Sep 17 00:00:00 2001 From: Michael Peterson Date: Thu, 19 Dec 2024 10:31:18 -0500 Subject: [PATCH 02/10] Revised resolve-cluster.asciidoc further --- docs/reference/indices/resolve-cluster.asciidoc | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/docs/reference/indices/resolve-cluster.asciidoc b/docs/reference/indices/resolve-cluster.asciidoc index 193b10bd3c60d..5ad1532f455af 100644 --- a/docs/reference/indices/resolve-cluster.asciidoc +++ b/docs/reference/indices/resolve-cluster.asciidoc @@ -111,8 +111,8 @@ ignored when frozen. Defaults to `false`. deprecated:[7.16.0] +[discrete] [[usecases-for-resolve-cluster]] - === Test availability of remote clusters The <> endpoint is commonly used to test whether the "local" @@ -130,7 +130,7 @@ been (re-)established. You may want to exclude a cluster or index from a search when: -1. A remote cluster is not currently connected and is configured with `skip_unavailable`=`false`. +1. A remote cluster could not be connected to and is configured with `skip_unavailable`=`false`. Executing a {ccs} under those conditions will cause <>. @@ -209,6 +209,7 @@ The API returns the following response: <5> The Elasticsearch server version. +[discrete] [[resolve-cluster-api-error-example]] ==== Identifying potential problems with your {ccs} From fd48388227f383a2ec3dcd3c5df5b7fdd65d332f Mon Sep 17 00:00:00 2001 From: Michael Peterson Date: Thu, 19 Dec 2024 10:39:58 -0500 Subject: [PATCH 03/10] Rewording of docs language before PR review --- docs/reference/cluster/remote-info.asciidoc | 3 +-- docs/reference/indices/resolve-cluster.asciidoc | 6 +++--- 2 files changed, 4 insertions(+), 5 deletions(-) diff --git a/docs/reference/cluster/remote-info.asciidoc b/docs/reference/cluster/remote-info.asciidoc index 08b48407b830c..2c6c0202a7d79 100644 --- a/docs/reference/cluster/remote-info.asciidoc +++ b/docs/reference/cluster/remote-info.asciidoc @@ -26,8 +26,7 @@ remote cluster information. It returns connection and endpoint information keyed by the configured remote cluster alias. TIP: This endpoint returns the information that reflects state on the cluster -you are querying. The `connected` field reports whether the querying cluster is -currently connected to the remote cluster. It does not necessarily reflect whether +you are querying. The `connected` field does not necessarily reflect whether the remote cluster is down or unavailable, only whether there is currently an open connection to it. Elasticsearch does not spontaneously try to reconnect to a disconnected remote cluster. To trigger a reconnection, attempt a diff --git a/docs/reference/indices/resolve-cluster.asciidoc b/docs/reference/indices/resolve-cluster.asciidoc index 5ad1532f455af..91b70e8d46d71 100644 --- a/docs/reference/indices/resolve-cluster.asciidoc +++ b/docs/reference/indices/resolve-cluster.asciidoc @@ -121,9 +121,9 @@ necessarily reflect whether the remote cluster is available or not. The remote c be available, while the local cluster is not currently connected to it. You can use this API to force a reconnection (for example with `GET _resolve/cluster/*:*`) and -examine the `connected` field in the response for each remote. This will also cause the -<> endpoint to now reflect the fact that a connection has -been (re-)established. +the `connected` field in the response will indicate whether it was successful or not. +If a connection was (re-)established, this will also cause the +<> endpoint to now indicate a connected status. === Advantages of using this endpoint before a {ccs} From 277d334cf14d6a893c18c1ed526716607f7570dc Mon Sep 17 00:00:00 2001 From: Michael Peterson Date: Tue, 7 Jan 2025 16:31:48 -0500 Subject: [PATCH 04/10] Adjusted wording around 'force a reconnect' to 'attempt to reconnect' --- docs/reference/indices/resolve-cluster.asciidoc | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/docs/reference/indices/resolve-cluster.asciidoc b/docs/reference/indices/resolve-cluster.asciidoc index 11c18f3acde75..430c377ff4402 100644 --- a/docs/reference/indices/resolve-cluster.asciidoc +++ b/docs/reference/indices/resolve-cluster.asciidoc @@ -126,7 +126,8 @@ cluster (the cluster being queried) is connected to its remote clusters, but it necessarily reflect whether the remote cluster is available or not. The remote cluster may be available, while the local cluster is not currently connected to it. -You can use this API to force a reconnection (for example with `GET _resolve/cluster/*:*`) and +You can use this API to attempt to reconnect to remote clusters +(for example with `GET _resolve/cluster/*:*`) and the `connected` field in the response will indicate whether it was successful or not. If a connection was (re-)established, this will also cause the <> endpoint to now indicate a connected status. From 496586e77111b01ab35f42f8ed8f1ba7e1a29058 Mon Sep 17 00:00:00 2001 From: Michael Peterson Date: Thu, 23 Jan 2025 14:19:03 -0500 Subject: [PATCH 05/10] Minor tweaks --- docs/reference/cluster/remote-info.asciidoc | 4 ++-- docs/reference/indices/resolve-cluster.asciidoc | 2 +- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/reference/cluster/remote-info.asciidoc b/docs/reference/cluster/remote-info.asciidoc index e724ce1d766dc..3f6160d9db669 100644 --- a/docs/reference/cluster/remote-info.asciidoc +++ b/docs/reference/cluster/remote-info.asciidoc @@ -26,8 +26,8 @@ Returns configured remote cluster information. [[cluster-remote-info-api-desc]] ==== {api-description-title} -The cluster remote info API allows you to retrieve all of the configured -remote cluster information. It returns connection and endpoint information keyed +The cluster remote info API allows you to retrieve information about configured +remote clusters. It returns connection and endpoint information keyed by the configured remote cluster alias. TIP: This endpoint returns the information that reflects state on the cluster diff --git a/docs/reference/indices/resolve-cluster.asciidoc b/docs/reference/indices/resolve-cluster.asciidoc index 019cb4c616dd5..84026a6fc4467 100644 --- a/docs/reference/indices/resolve-cluster.asciidoc +++ b/docs/reference/indices/resolve-cluster.asciidoc @@ -44,7 +44,7 @@ Once the proper security permissions are obtained, then you can rely on the `con in the response to determine whether the remote cluster is available and ready for querying. ==== -NOTE: When querying older clusters that do not support the _resolve/cluster endpoint +NOTE: When querying older clusters that do not support the `_resolve/cluster` endpoint without an index expression, the local cluster will send the index expression `dummy*` to those remote clusters, so if an errors occur, you may see a reference to that index expression even though you didn't request it. If it causes a problem, you can instead From d5fd9bbd5bcc0fd6a829bfa6026b2d243067365f Mon Sep 17 00:00:00 2001 From: Michael Peterson Date: Tue, 28 Jan 2025 15:55:06 -0500 Subject: [PATCH 06/10] Added docs on the new timeout param for _resolve/cluster --- .../indices/resolve-cluster.asciidoc | 35 ++++++++++++------- 1 file changed, 23 insertions(+), 12 deletions(-) diff --git a/docs/reference/indices/resolve-cluster.asciidoc b/docs/reference/indices/resolve-cluster.asciidoc index 84026a6fc4467..455398a1041c2 100644 --- a/docs/reference/indices/resolve-cluster.asciidoc +++ b/docs/reference/indices/resolve-cluster.asciidoc @@ -124,6 +124,15 @@ Resources on <> can be specified using the [[resolve-cluster-api-query-params]] ==== {api-query-parms-title} +`timeout`:: +(Optional, TimeValue) Specify a max wait time for remote clusters to respond. +If a remote cluster does not respond within this timeout period, the API response +will show the cluster as not connected and show an error message for that cluster +indicating that it did not respond. The default is unset and the query can take +as long as the networking layer is configured to wait for remote clusters that are +not responding (typically 30 seconds). ++ + include::{es-ref-dir}/rest-api/common-parms.asciidoc[tag=expand-wildcards] + Defaults to `open`. @@ -254,11 +263,12 @@ The API returns the following response: ==== Identifying potential problems with your {ccs} The following request shows several examples of how modifying your query can -prevent search failures. +prevent search failures. Note also that a `timeout` of 5 seconds is sent, which +sets the maximum time the query will wait for remote clusters to respond. [source,console] -------------------------------------------------- -GET /_resolve/cluster/not-present,clust*:my-index*,oldcluster:*?ignore_unavailable=false +GET /_resolve/cluster/not-present,clust*:my-index*,oldcluster:*?ignore_unavailable=false&timeout=5s -------------------------------------------------- // TEST[continued] // TEST[s/,oldcluster:*//] @@ -284,16 +294,14 @@ GET /_resolve/cluster/not-present,clust*:my-index*,oldcluster:*?ignore_unavailab }, "cluster_two": { "connected": false, <3> - "skip_unavailable": false, - "matching_indices": true, - "version": { - "number": "8.13.0", - "build_flavor": "default", - "minimum_wire_compatibility_version": "7.17.0", - "minimum_index_compatibility_version": "7.0.0" - } + "skip_unavailable": false + }, + "cluster_three": { + "connected": false, + "skip_unavailable": true, + "error": "Request timed out before receiving a response from the remote cluster" <4> }, - "oldcluster": { <4> + "oldcluster": { <5> "connected": true, "skip_unavailable": false, "matching_indices": true @@ -315,7 +323,10 @@ could be closed. (You can check this by using the failed). Since this cluster is marked as `skip_unavailable=false`, you should probably exclude this cluster from the search by adding `-cluster_two:*` to the search index expression. -<4> The `oldcluster` remote cluster shows that it has matching indices, but no +<4> For `cluster_three`, the error message indicates that this remote cluster did +not respond within the 5 second timeout window specified, so it is also marked as +not connected. +<5> The `oldcluster` remote cluster shows that it has matching indices, but no version information is included. This indicates that the cluster version predates the introduction of the `_resolve/cluster` API in 8.13.0., so you may want to exclude it from your {ccs}. (Note: the endpoint was able to tell there were From 405aeb8579b005ace6b3f887aa2f87c81d7559c5 Mon Sep 17 00:00:00 2001 From: Michael Peterson Date: Tue, 28 Jan 2025 16:01:31 -0500 Subject: [PATCH 07/10] Revert "Added docs on the new timeout param for _resolve/cluster" This reverts commit d5fd9bbd5bcc0fd6a829bfa6026b2d243067365f. --- .../indices/resolve-cluster.asciidoc | 35 +++++++------------ 1 file changed, 12 insertions(+), 23 deletions(-) diff --git a/docs/reference/indices/resolve-cluster.asciidoc b/docs/reference/indices/resolve-cluster.asciidoc index 455398a1041c2..84026a6fc4467 100644 --- a/docs/reference/indices/resolve-cluster.asciidoc +++ b/docs/reference/indices/resolve-cluster.asciidoc @@ -124,15 +124,6 @@ Resources on <> can be specified using the [[resolve-cluster-api-query-params]] ==== {api-query-parms-title} -`timeout`:: -(Optional, TimeValue) Specify a max wait time for remote clusters to respond. -If a remote cluster does not respond within this timeout period, the API response -will show the cluster as not connected and show an error message for that cluster -indicating that it did not respond. The default is unset and the query can take -as long as the networking layer is configured to wait for remote clusters that are -not responding (typically 30 seconds). -+ - include::{es-ref-dir}/rest-api/common-parms.asciidoc[tag=expand-wildcards] + Defaults to `open`. @@ -263,12 +254,11 @@ The API returns the following response: ==== Identifying potential problems with your {ccs} The following request shows several examples of how modifying your query can -prevent search failures. Note also that a `timeout` of 5 seconds is sent, which -sets the maximum time the query will wait for remote clusters to respond. +prevent search failures. [source,console] -------------------------------------------------- -GET /_resolve/cluster/not-present,clust*:my-index*,oldcluster:*?ignore_unavailable=false&timeout=5s +GET /_resolve/cluster/not-present,clust*:my-index*,oldcluster:*?ignore_unavailable=false -------------------------------------------------- // TEST[continued] // TEST[s/,oldcluster:*//] @@ -294,14 +284,16 @@ GET /_resolve/cluster/not-present,clust*:my-index*,oldcluster:*?ignore_unavailab }, "cluster_two": { "connected": false, <3> - "skip_unavailable": false - }, - "cluster_three": { - "connected": false, - "skip_unavailable": true, - "error": "Request timed out before receiving a response from the remote cluster" <4> + "skip_unavailable": false, + "matching_indices": true, + "version": { + "number": "8.13.0", + "build_flavor": "default", + "minimum_wire_compatibility_version": "7.17.0", + "minimum_index_compatibility_version": "7.0.0" + } }, - "oldcluster": { <5> + "oldcluster": { <4> "connected": true, "skip_unavailable": false, "matching_indices": true @@ -323,10 +315,7 @@ could be closed. (You can check this by using the failed). Since this cluster is marked as `skip_unavailable=false`, you should probably exclude this cluster from the search by adding `-cluster_two:*` to the search index expression. -<4> For `cluster_three`, the error message indicates that this remote cluster did -not respond within the 5 second timeout window specified, so it is also marked as -not connected. -<5> The `oldcluster` remote cluster shows that it has matching indices, but no +<4> The `oldcluster` remote cluster shows that it has matching indices, but no version information is included. This indicates that the cluster version predates the introduction of the `_resolve/cluster` API in 8.13.0., so you may want to exclude it from your {ccs}. (Note: the endpoint was able to tell there were From 45bb773fa19a4e2bfac193d1b7f119c6dca8c775 Mon Sep 17 00:00:00 2001 From: Michael Peterson Date: Tue, 28 Jan 2025 16:03:50 -0500 Subject: [PATCH 08/10] Fixed error in example of disconnected cluster --- docs/reference/indices/resolve-cluster.asciidoc | 9 +-------- 1 file changed, 1 insertion(+), 8 deletions(-) diff --git a/docs/reference/indices/resolve-cluster.asciidoc b/docs/reference/indices/resolve-cluster.asciidoc index 84026a6fc4467..ce6a4f0fbb7b7 100644 --- a/docs/reference/indices/resolve-cluster.asciidoc +++ b/docs/reference/indices/resolve-cluster.asciidoc @@ -284,14 +284,7 @@ GET /_resolve/cluster/not-present,clust*:my-index*,oldcluster:*?ignore_unavailab }, "cluster_two": { "connected": false, <3> - "skip_unavailable": false, - "matching_indices": true, - "version": { - "number": "8.13.0", - "build_flavor": "default", - "minimum_wire_compatibility_version": "7.17.0", - "minimum_index_compatibility_version": "7.0.0" - } + "skip_unavailable": false }, "oldcluster": { <4> "connected": true, From 84ff8e90775a2927fdaed3dc0225ea3715f488c4 Mon Sep 17 00:00:00 2001 From: Michael Peterson Date: Wed, 29 Jan 2025 09:02:41 -0500 Subject: [PATCH 09/10] Removed comments from resolve-cluster docs that don't belong in 8.17.2. Will add those back in follow on PR --- .../indices/resolve-cluster.asciidoc | 29 ++----------------- 1 file changed, 3 insertions(+), 26 deletions(-) diff --git a/docs/reference/indices/resolve-cluster.asciidoc b/docs/reference/indices/resolve-cluster.asciidoc index ce6a4f0fbb7b7..f7d21e8c0b8ea 100644 --- a/docs/reference/indices/resolve-cluster.asciidoc +++ b/docs/reference/indices/resolve-cluster.asciidoc @@ -11,9 +11,7 @@ For the most up-to-date API details, refer to {api-es}/group/endpoint-indices[In -- Resolves the specified index expressions to return information about -each cluster, including the local "querying" cluster, if included. If no index expression -is provided, this endpoint will return information about all the remote -clusters that are configured on the querying cluster. +each cluster, including the local "querying" cluster, if included. This endpoint is useful before doing a <> in order to determine which remote clusters should be included in a search. @@ -29,7 +27,7 @@ For each cluster in scope, information is returned about: contact the remote clusters, unlike the <> endpoint. 2. whether each remote cluster is configured with `skip_unavailable` as `true` or `false` 3. whether there are any indices, aliases or data streams on that cluster that match - the index expression (if one provided) + the index expression 4. whether the search is likely to have errors returned when you do a {ccs} (including any authorization errors if your user does not have permission to query a remote cluster or the indices on that cluster) @@ -44,12 +42,6 @@ Once the proper security permissions are obtained, then you can rely on the `con in the response to determine whether the remote cluster is available and ready for querying. ==== -NOTE: When querying older clusters that do not support the `_resolve/cluster` endpoint -without an index expression, the local cluster will send the index expression `dummy*` -to those remote clusters, so if an errors occur, you may see a reference to that index -expression even though you didn't request it. If it causes a problem, you can instead -include an index expression like `*:*` to this endpoint to bypass the issue. - //// [source,console] -------------------------------- @@ -79,14 +71,6 @@ PUT _cluster/settings // TEST[s/35.238.149.\d+:930\d+/\${transport_host}/] //// -[source,console] ----- -GET /_resolve/cluster ----- -// TEST[continued] - -Returns information about all remote clusters configured on the local cluster. - [source,console] ---- GET /_resolve/cluster/my-index-*,cluster*:my-index-* @@ -142,13 +126,6 @@ ignored when frozen. Defaults to `false`. + deprecated:[7.16.0] -[TIP] -==== -The index options above are only allowed when specifying an index expression. -You will get an error if you specify index options to the _resolve/cluster API -that takes no index expression. -==== - [discrete] [[usecases-for-resolve-cluster]] @@ -159,7 +136,7 @@ cluster (the cluster being queried) is connected to its remote clusters, but it necessarily reflect whether the remote cluster is available or not. The remote cluster may be available, while the local cluster is not currently connected to it. -You can use this API to attempt to reconnect to remote clusters +You can use the resolve-cluster API to attempt to reconnect to remote clusters (for example with `GET _resolve/cluster/*:*`) and the `connected` field in the response will indicate whether it was successful or not. If a connection was (re-)established, this will also cause the From d8dfa4410bbb3f254d72a27eea3e47612872269a Mon Sep 17 00:00:00 2001 From: Michael Peterson Date: Wed, 29 Jan 2025 09:09:08 -0500 Subject: [PATCH 10/10] PR feedback change --- docs/reference/cluster/remote-info.asciidoc | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/docs/reference/cluster/remote-info.asciidoc b/docs/reference/cluster/remote-info.asciidoc index 3f6160d9db669..e91ccc4d8f4a1 100644 --- a/docs/reference/cluster/remote-info.asciidoc +++ b/docs/reference/cluster/remote-info.asciidoc @@ -30,13 +30,13 @@ The cluster remote info API allows you to retrieve information about configured remote clusters. It returns connection and endpoint information keyed by the configured remote cluster alias. -TIP: This endpoint returns the information that reflects state on the cluster -you are querying. The `connected` field does not necessarily reflect whether -the remote cluster is down or unavailable, only whether there is currently an open -connection to it. Elasticsearch does not spontaneously try to reconnect to a -disconnected remote cluster. To trigger a reconnection, attempt a -<>, <>, -or try the <> endpoint. +TIP: This API returns information that reflects current state on the local cluster. +The `connected` field does not necessarily reflect whether a remote cluster is +down or unavailable, only whether there is currently an open connection to it. +Elasticsearch does not spontaneously try to reconnect to a disconnected remote +cluster. To trigger a reconnection, attempt a <>, +<>, or try the +<> endpoint. [[cluster-remote-info-api-response-body]]