Skip to content

Commit 5664599

Browse files
brett0000FFdussault-antoine
brett0000FF
and
dussault-antoine
authored
Adaptive Sampling (#29389)
Co-authored-by: dussault-antoine <[email protected]>
1 parent d0d6ff7 commit 5664599

File tree

9 files changed

+240
-150
lines changed

9 files changed

+240
-150
lines changed

‎config/_default/menus/main.en.yaml

Lines changed: 8 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -3788,21 +3788,26 @@ menu:
37883788
parent: tracing_trace_pipeline
37893789
identifier: tracing_ingestion_controls
37903790
weight: 402
3791+
- name: Adaptive Sampling
3792+
url: tracing/trace_pipeline/adaptive_sampling/
3793+
parent: tracing_trace_pipeline
3794+
identifier: tracing_adaptive_sampling
3795+
weight: 403
37913796
- name: Generate Metrics
37923797
url: tracing/trace_pipeline/generate_metrics/
37933798
parent: tracing_trace_pipeline
37943799
identifier: tracing_pipeline_generate_metrics
3795-
weight: 403
3800+
weight: 404
37963801
- name: Trace Retention
37973802
url: tracing/trace_pipeline/trace_retention/
37983803
parent: tracing_trace_pipeline
37993804
identifier: tracing_retention
3800-
weight: 404
3805+
weight: 405
38013806
- name: Usage Metrics
38023807
url: tracing/trace_pipeline/metrics/
38033808
parent: tracing_trace_pipeline
38043809
identifier: tracing_pipeline_metrics
3805-
weight: 405
3810+
weight: 406
38063811
- name: Correlate Traces with Other Telemetry
38073812
url: tracing/other_telemetry/
38083813
parent: tracing

‎content/en/tracing/guide/adaptive_sampling.md

Lines changed: 0 additions & 112 deletions
This file was deleted.

‎content/en/tracing/trace_pipeline/adaptive_sampling.md

Lines changed: 157 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,157 @@
1+
---
2+
title: Adaptive Sampling
3+
disable_toc: false
4+
aliases:
5+
- /tracing/guide/adaptive_sampling
6+
further_reading:
7+
- link: "/tracing/trace_pipeline/ingestion_mechanisms"
8+
tag: "Documentation"
9+
text: "Ingestion Mechanisms"
10+
- link: "/tracing/trace_pipeline/ingestion_controls"
11+
tag: "Documentation"
12+
text: "Ingestion Controls"
13+
---
14+
15+
{{< callout url="https://www.datadoghq.com/private-beta/resource-based-sampling-adaptive-sampling/" header="Request access to the Preview!" >}}
16+
Adaptive sampling is in Preview. To request access, complete the form.
17+
{{< /callout >}}
18+
19+
## Overview
20+
21+
Datadog **adaptive sampling** helps you capture more relevant traces while remaining close to a specific budget (ingested gigabytes).
22+
23+
When you choose adaptive sampling as your sampling strategy, you select a target monthly volume for trace ingestion for one or more services. This ensures that the consumption of these services matches the target volume at the end of the month, while keeping visibility over their endpoints.
24+
25+
Adaptive sampling uses [remote configuration][3] plus the existing [sampling rules][7] mechanisms to dynamically adjust sampling rates for each environment, service, and resource combination. This helps you to:
26+
- Match your specified monthly budget.
27+
- Ensure visibility for low-traffic services and endpoints by capturing at least one trace for each combination of service, resource, and environment every 5 minutes.
28+
29+
To configure services to use adaptive sampling, follow the instructions listed below.
30+
31+
## Requirements
32+
33+
- Datadog Agent [7.53.0][2] or higher.
34+
- [Remote Configuration][3] enabled for your Agent.
35+
- `APM Remote Configuration Write` [permission][4].
36+
**Note**: If you don't have this permission, ask your Datadog admin to update your permissions from your organization settings.
37+
38+
### Tracing library versions
39+
40+
The following table lists minimum tracing library versions required for adaptive sampling:
41+
42+
| Language | Minimum version required |
43+
|-------------|--------------------------|
44+
| Java | [v1.34.0][5] |
45+
| Go | [v1.68.0][6] |
46+
| Python | [v2.9.6][10] |
47+
| Ruby | [v2.0.0][11] |
48+
| Node.js | [v5.16.0][12] |
49+
| .NET | [v2.54.0][13] |
50+
| C++/Proxies | [v0.2.2][14] |
51+
| PHP | [v1.4.0][17] |
52+
53+
## Configure the adaptive sampling target
54+
55+
To get started with adaptive sampling, you first need to pick a target strategy setting:
56+
57+
- **Set Budget by Number of APM Hosts**: Configure a budget that is proportional to your allotment and the number of services onboarded (for example, based on the number of APM hosts)
58+
- **Set Budget by Data Volume**: Configure a fixed target in gigabytes per month
59+
60+
61+
| | Budget by Number of APM Hosts | Budget by Data Volume |
62+
|----------|--------------------------------------------------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------|
63+
| **Pros** | Scales with the number of APM hosts and the number of services onboarded; you only have to set it once | Ensures you never go over budget |
64+
| **Cons** | Not a good fit if you want to stay below a specific volume, as it may vary depending on the number of hosts reporting APM data to Datadog | You have to edit the budget every time you onboard a new service to adaptive sampling |
65+
66+
To set the adaptive sampling monthly target:
67+
1. Navigate to the [Ingestion Control][18] page.
68+
2. Click **Manage Adaptive Sampling Target**.
69+
{{< img src="/tracing/guide/adaptive_sampling/adaptive_sampling_target_cta.png" alt="Call to action to set adaptive sampling target" style="width:100%;">}}
70+
3. Choose a target strategy for sampling:
71+
- [Set budget by number of APM hosts](#set-budget-by-number-of-apm-hosts-recommended)
72+
- [Set budget by data volume](#set-budget-by-data-volume)
73+
4. Click **Apply**.
74+
75+
### Set budget by number of APM hosts (Recommended)
76+
77+
{{< img src="/tracing/guide/adaptive_sampling/percentage_based_target_setting.png" alt="Percentage based target setting" style="width:100%;">}}
78+
79+
Set your monthly target to a percentage of your allotment. At the bottom of the page, you are provided with a more complete explanation of how that percentage is converted in a monthly target volume. It is the product of:
80+
81+
- The **global allotment**: `150GB * number_of_APM_hosts + 50GB * number_of_traced_serverless_invocations (if applicable) + 10GB * number_of_fargate_tasks (if applicable)`
82+
- The **percentage of allotment** configured above
83+
- The **contribution of onboarded services** to the allotment. For example, if the services onboarded to adaptive sampling contribute to 10% of the total ingested volume, Datadog targets 10% of the global allotment. This number increases with the number of services onboarded.
84+
85+
{{< img src="/tracing/guide/adaptive_sampling/percentage_based_target_computation.png" alt="Percentage based target computation" style="width:100%;">}}
86+
87+
That monthly target volume is recomputed every 30 minutes.
88+
89+
### Set budget by data volume
90+
91+
{{< img src="/tracing/guide/adaptive_sampling/volume_based_target_setting.png" alt="Volume based target setting" style="width:100%;">}}
92+
93+
If you are configuring the first service to adaptive sampling, ensure that the ingestion volume target is `>0`. For subsequent services, you should increase the allocated budget after the new service is onboarded to account for the new volume.
94+
<div class="alert alert-info">The configured budget is only allocated to services enrolled in adaptive sampling. It does not include ingested volume from services not enrolled in adaptive sampling, local sampling rules, or other <a href="/tracing/trace_pipeline/ingestion_mechanisms#in-the-agent">sampling mechanisms</a> configured locally in the Agent or tracing libraries.</div>
95+
96+
## Configure adaptive sampling for a service
97+
98+
### View sampling rates by resource for a service
99+
100+
Before you configure adaptive sampling for a service, you can view the current ingestion configuration for the service.
101+
102+
To see configured sampling rates:
103+
104+
1. Navigate to the [Ingestion Control][18] page.
105+
2. Click a service to view the **Service Ingestion Summary**.
106+
3. View the table listing the applied sampling rates by resource of the service.
107+
108+
{{< img src="/tracing/trace_indexing_and_ingestion/resource_sampling_rates.png" alt="Sampling rates table by resource" style="width:100%;">}}
109+
110+
The table includes:
111+
- **Ingested bytes**: Ingested bytes from spans of the service and resource.
112+
- **Downstream bytes**: Ingested bytes from spans where the sampling decision starts from that service and resource, including downstream services.
113+
- **Configuration**: Source of the resource sampling rate:
114+
- `AUTOMATIC`: [Default head-based sampling mechanism][8] from the Agent.
115+
- `CONFIGURED LOCAL`: [Sampling rule][7] set locally in the tracing library.
116+
- `CONFIGURED REMOTE`: Remote sampling rule set from the Datadog UI.
117+
- `ADAPTIVE REMOTE`: Adaptive sampling rules set by Datadog.
118+
119+
### Onboard a service to adaptive sampling
120+
121+
To onboard a service to adaptive sampling:
122+
123+
1. Navigate to the [Ingestion Control][18] page.
124+
2. Click a service to view the **Service Ingestion Summary**.
125+
3. Click **Manage Ingestion Rate**.
126+
4. Choose **Datadog adaptive sampling rates** as your service's sampling strategy.
127+
5. (Optional) Configure explicit [sampling rates][15] for specific resources, for which you would like to capture more (for example, 100% of `GET /checkout` endpoints) or less (for example, 0.1% of `/health` requests) data.
128+
6. Click **Apply**.
129+
130+
<div class="alert alert-info">If applying this configuration <strong>Remotely</strong> is disabled, ensure the <a href="#requirements">Remote Configuration requirements</a> are met.</div>
131+
132+
{{< img src="/tracing/guide/adaptive_sampling/adaptive_sampling_setting_modal.png" alt="Adaptive sampling setting modal" style="width:70%;">}}
133+
134+
The configuration should take effect in 5-6 minutes, the time it takes for Datadog to observe the service's traffic pattern, compute, then apply the sampling rates. Resources that have been configured remotely display as `Configured Remote` in the **Configuration** column.
135+
136+
## Further reading
137+
138+
{{< partial name="whats-next/whats-next.html" >}}
139+
140+
[1]: /tracing/trace_pipeline/ingestion_controls#service-ingestion-summary
141+
[2]: https://github.com/DataDog/datadog-agent/releases/tag/7.53.0
142+
[3]: /agent/remote_config
143+
[4]: /account_management/rbac/permissions/
144+
[5]: https://github.com/DataDog/dd-trace-java/releases/tag/v1.34.0
145+
[6]: https://github.com/DataDog/dd-trace-go/releases/tag/v1.68.0
146+
[7]: /tracing/trace_pipeline/ingestion_mechanisms#in-tracing-libraries-user-defined-rules
147+
[8]: /tracing/trace_pipeline/ingestion_mechanisms#in-the-agent
148+
[9]: /tracing/trace_explorer/#live-search-for-15-minutes
149+
[10]: https://github.com/DataDog/dd-trace-py/releases/tag/v2.9.6
150+
[11]: https://github.com/DataDog/dd-trace-rb/releases/tag/v2.0.0
151+
[12]: https://github.com/DataDog/dd-trace-js/releases/tag/v5.16.0
152+
[13]: https://github.com/DataDog/dd-trace-dotnet/releases/tag/v2.54.0
153+
[14]: https://github.com/DataDog/dd-trace-cpp/releases/tag/v0.2.2
154+
[15]: /tracing/trace_pipeline/ingestion_controls#configure-the-service-ingestion-rates-by-resource
155+
[16]: /tracing/trace_pipeline/ingestion_controls
156+
[17]: https://github.com/DataDog/dd-trace-php/releases/tag/1.4.0
157+
[18]: https://app.datadoghq.com/apm/traces/ingestion-control

0 commit comments

Comments
 (0)