Enable Shard-Level Search-load rate metric #128660
Open
+716
−30
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
drempapis
added
Team:Search Foundations
|
Pinging @elastic/es-search-foundations (Team:Search Foundations) |
…apis/elasticsearch into search_load_per_index_multiproject
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This work is part of an approach to monitor search load per index in a multi-project Elasticsearch environment. The goal is to enable dynamic adjustment of replica counts to optimize resource utilization and maintain a balance between insufficient usage and resource saturation
At the core of this solution is the
SearchStatsSettingsclass, which supports half-life calculations withinShardSearchStats::StatsHolderand configures the time interval for scheduled, pull-based collection of search node metrics, particularly relevant in serverless deployments.Search load is tracked on a per-shard basis by computing the Exponentially Weighted Moving Rate (EWMR) in the
ShardSearchStatsclass, using the "took" time from each completed search phase.Additionally, a transport action has been implemented to support the on-demand retrieval of search load metrics directly from search nodes.