Extract trace context from embedded SQS message attribute '_datadog' by mcculls · Pull Request #5920 · DataDog/dd-trace-java

mcculls · 2023-09-20T11:38:02Z

Motivation

The _datadog message attribute is used by DSM and certain language tracers to propagate context instead of AWS' AWSTraceHeader. The attribute contains trace context as a JSON string, or encoded with Base64 as a binary value.

pr-commenter · 2023-09-20T12:18:51Z

Benchmarks

Startup

Parameters

	Baseline	Candidate
commit	1.22.0-SNAPSHOT~ecccc528c0	1.22.0-SNAPSHOT~d45b54c0b6
config	baseline	candidate

See matching parameters

	Baseline	Candidate
application	insecure-bank	insecure-bank
module	Agent	Agent
parent	None	None
variant	appsec	appsec

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 62 cases.

Startup time reports for insecure-bank

gantt
    title insecure-bank - global startup overhead: candidate=1.22.0-SNAPSHOT~d45b54c0b6, baseline=1.22.0-SNAPSHOT~ecccc528c0

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.003 s) : 0, 1003079
Total [baseline] (8.676 s) : 0, 8676027
Agent [candidate] (1.013 s) : 0, 1013379
Total [candidate] (8.658 s) : 0, 8658185
section appsec
Agent [baseline] (1.09 s) : 0, 1090474
Total [baseline] (8.727 s) : 0, 8727329
Agent [candidate] (1.093 s) : 0, 1092643
Total [candidate] (8.718 s) : 0, 8717974
section iast
Agent [baseline] (1.113 s) : 0, 1112927
Total [baseline] (9.24 s) : 0, 9239696
Agent [candidate] (1.115 s) : 0, 1115401
Total [candidate] (9.174 s) : 0, 9173999
section profiling
Agent [baseline] (1.174 s) : 0, 1173686
Total [baseline] (8.887 s) : 0, 8886912
Agent [candidate] (1.179 s) : 0, 1178713
Total [candidate] (8.867 s) : 0, 8866914

baseline results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.003 s	-
Agent	appsec	1.09 s	87.395 ms (8.7%)
Agent	iast	1.113 s	109.848 ms (11.0%)
Agent	profiling	1.174 s	170.607 ms (17.0%)
Total	tracing	8.676 s	-
Total	appsec	8.727 s	51.302 ms (0.6%)
Total	iast	9.24 s	563.669 ms (6.5%)
Total	profiling	8.887 s	210.885 ms (2.4%)

candidate results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.013 s	-
Agent	appsec	1.093 s	79.264 ms (7.8%)
Agent	iast	1.115 s	102.022 ms (10.1%)
Agent	profiling	1.179 s	165.334 ms (16.3%)
Total	tracing	8.658 s	-
Total	appsec	8.718 s	59.789 ms (0.7%)
Total	iast	9.174 s	515.814 ms (6.0%)
Total	profiling	8.867 s	208.729 ms (2.4%)

gantt
    title insecure-bank - break down per module: candidate=1.22.0-SNAPSHOT~d45b54c0b6, baseline=1.22.0-SNAPSHOT~ecccc528c0

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (621.109 ms) : 0, 621109
BytebuddyAgent [candidate] (627.38 ms) : 0, 627380
GlobalTracer [baseline] (292.303 ms) : 0, 292303
GlobalTracer [candidate] (295.883 ms) : 0, 295883
AppSec [baseline] (48.888 ms) : 0, 48888
AppSec [candidate] (48.984 ms) : 0, 48984
Remote Config [baseline] (671.864 µs) : 0, 672
Remote Config [candidate] (696.159 µs) : 0, 696
Telemetry [baseline] (5.972 ms) : 0, 5972
Telemetry [candidate] (6.013 ms) : 0, 6013
section appsec
BytebuddyAgent [baseline] (622.185 ms) : 0, 622185
BytebuddyAgent [candidate] (623.473 ms) : 0, 623473
GlobalTracer [baseline] (292.326 ms) : 0, 292326
GlobalTracer [candidate] (293.035 ms) : 0, 293035
AppSec [baseline] (135.167 ms) : 0, 135167
AppSec [candidate] (135.292 ms) : 0, 135292
Remote Config [baseline] (639.882 µs) : 0, 640
Remote Config [candidate] (638.901 µs) : 0, 639
Telemetry [baseline] (5.884 ms) : 0, 5884
Telemetry [candidate] (5.855 ms) : 0, 5855
section iast
BytebuddyAgent [baseline] (735.266 ms) : 0, 735266
BytebuddyAgent [candidate] (736.493 ms) : 0, 736493
GlobalTracer [baseline] (276.954 ms) : 0, 276954
GlobalTracer [candidate] (278.072 ms) : 0, 278072
AppSec [baseline] (45.715 ms) : 0, 45715
AppSec [candidate] (45.897 ms) : 0, 45897
IAST [baseline] (14.455 ms) : 0, 14455
IAST [candidate] (14.359 ms) : 0, 14359
Remote Config [baseline] (563.164 µs) : 0, 563
Remote Config [candidate] (559.267 µs) : 0, 559
Telemetry [baseline] (5.757 ms) : 0, 5757
Telemetry [candidate] (5.785 ms) : 0, 5785
section profiling
BytebuddyAgent [baseline] (632.378 ms) : 0, 632378
BytebuddyAgent [candidate] (635.028 ms) : 0, 635028
GlobalTracer [baseline] (352.828 ms) : 0, 352828
GlobalTracer [candidate] (354.641 ms) : 0, 354641
AppSec [baseline] (48.807 ms) : 0, 48807
AppSec [candidate] (48.682 ms) : 0, 48682
Remote Config [baseline] (638.815 µs) : 0, 639
Remote Config [candidate] (665.188 µs) : 0, 665
Telemetry [baseline] (5.991 ms) : 0, 5991
Telemetry [candidate] (5.984 ms) : 0, 5984
ProfilingAgent [baseline] (80.141 ms) : 0, 80141
ProfilingAgent [candidate] (80.663 ms) : 0, 80663
Profiling [baseline] (80.166 ms) : 0, 80166
Profiling [candidate] (80.688 ms) : 0, 80688

Startup time reports for petclinic

gantt
    title petclinic - global startup overhead: candidate=1.22.0-SNAPSHOT~d45b54c0b6, baseline=1.22.0-SNAPSHOT~ecccc528c0

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.019 s) : 0, 1018753
Total [baseline] (9.292 s) : 0, 9291655
Agent [candidate] (1.006 s) : 0, 1005710
Total [candidate] (9.215 s) : 0, 9214677
section appsec
Agent [baseline] (1.091 s) : 0, 1090549
Total [baseline] (9.272 s) : 0, 9272336
Agent [candidate] (1.109 s) : 0, 1109111
Total [candidate] (9.297 s) : 0, 9297476
section iast
Agent [baseline] (1.117 s) : 0, 1116558
Total [baseline] (9.468 s) : 0, 9467659
Agent [candidate] (1.12 s) : 0, 1119530
Total [candidate] (9.458 s) : 0, 9457699
section profiling
Agent [baseline] (1.175 s) : 0, 1174545
Total [baseline] (9.403 s) : 0, 9403489
Agent [candidate] (1.191 s) : 0, 1191430
Total [candidate] (9.452 s) : 0, 9451943

baseline results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.019 s	-
Agent	appsec	1.091 s	71.796 ms (7.0%)
Agent	iast	1.117 s	97.805 ms (9.6%)
Agent	profiling	1.175 s	155.792 ms (15.3%)
Total	tracing	9.292 s	-
Total	appsec	9.272 s	-19.319 ms (-0.2%)
Total	iast	9.468 s	176.004 ms (1.9%)
Total	profiling	9.403 s	111.834 ms (1.2%)

candidate results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.006 s	-
Agent	appsec	1.109 s	103.402 ms (10.3%)
Agent	iast	1.12 s	113.82 ms (11.3%)
Agent	profiling	1.191 s	185.721 ms (18.5%)
Total	tracing	9.215 s	-
Total	appsec	9.297 s	82.8 ms (0.9%)
Total	iast	9.458 s	243.022 ms (2.6%)
Total	profiling	9.452 s	237.267 ms (2.6%)

gantt
    title petclinic - break down per module: candidate=1.22.0-SNAPSHOT~d45b54c0b6, baseline=1.22.0-SNAPSHOT~ecccc528c0

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (631.09 ms) : 0, 631090
BytebuddyAgent [candidate] (622.163 ms) : 0, 622163
GlobalTracer [baseline] (297.038 ms) : 0, 297038
GlobalTracer [candidate] (294.14 ms) : 0, 294140
AppSec [baseline] (49.074 ms) : 0, 49074
AppSec [candidate] (48.569 ms) : 0, 48569
Remote Config [baseline] (683.209 µs) : 0, 683
Remote Config [candidate] (667.59 µs) : 0, 668
Telemetry [baseline] (6.055 ms) : 0, 6055
Telemetry [candidate] (5.989 ms) : 0, 5989
section appsec
BytebuddyAgent [baseline] (622.507 ms) : 0, 622507
BytebuddyAgent [candidate] (632.879 ms) : 0, 632879
GlobalTracer [baseline] (292.232 ms) : 0, 292232
GlobalTracer [candidate] (298.219 ms) : 0, 298219
AppSec [baseline] (134.942 ms) : 0, 134942
AppSec [candidate] (136.63 ms) : 0, 136630
Remote Config [baseline] (636.187 µs) : 0, 636
Remote Config [candidate] (656.535 µs) : 0, 657
Telemetry [baseline] (5.862 ms) : 0, 5862
Telemetry [candidate] (5.953 ms) : 0, 5953
section iast
BytebuddyAgent [baseline] (737.438 ms) : 0, 737438
BytebuddyAgent [candidate] (739.252 ms) : 0, 739252
GlobalTracer [baseline] (278.067 ms) : 0, 278067
GlobalTracer [candidate] (278.926 ms) : 0, 278926
AppSec [baseline] (45.864 ms) : 0, 45864
AppSec [candidate] (46.044 ms) : 0, 46044
IAST [baseline] (14.469 ms) : 0, 14469
IAST [candidate] (14.583 ms) : 0, 14583
Remote Config [baseline] (558.791 µs) : 0, 559
Remote Config [candidate] (563.575 µs) : 0, 564
Telemetry [baseline] (5.743 ms) : 0, 5743
Telemetry [candidate] (5.82 ms) : 0, 5820
section profiling
BytebuddyAgent [baseline] (632.522 ms) : 0, 632522
BytebuddyAgent [candidate] (643.672 ms) : 0, 643672
GlobalTracer [baseline] (352.814 ms) : 0, 352814
GlobalTracer [candidate] (357.155 ms) : 0, 357155
AppSec [baseline] (48.752 ms) : 0, 48752
AppSec [candidate] (49.276 ms) : 0, 49276
Remote Config [baseline] (645.205 µs) : 0, 645
Remote Config [candidate] (656.022 µs) : 0, 656
Telemetry [baseline] (6.039 ms) : 0, 6039
Telemetry [candidate] (6.08 ms) : 0, 6080
ProfilingAgent [baseline] (80.885 ms) : 0, 80885
ProfilingAgent [candidate] (80.786 ms) : 0, 80786
Profiling [baseline] (80.91 ms) : 0, 80910
Profiling [candidate] (80.811 ms) : 0, 80811

Load

Parameters

	Baseline	Candidate
commit	1.22.0-SNAPSHOT~ecccc528c0	1.22.0-SNAPSHOT~d45b54c0b6
config	baseline	candidate
end_time	2023-09-22T00:27:25	2023-09-22T00:45:31
start_time	2023-09-22T00:27:08	2023-09-22T00:45:14

See matching parameters

	Baseline	Candidate
application	insecure-bank	insecure-bank
variant	appsec	appsec

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 24 cases.

Request duration reports for insecure-bank

gantt
    title insecure-bank - request duration [CI 0.99] : candidate=1.22.0-SNAPSHOT~d45b54c0b6, baseline=1.22.0-SNAPSHOT~ecccc528c0
    dateFormat X
    axisFormat %s
section baseline
no_agent (367.775 µs) : 348, 387
.   : milestone, 368,
appsec (672.692 µs) : 652, 693
.   : milestone, 673,
iast (467.795 µs) : 447, 489
.   : milestone, 468,
iast_FULL (514.86 µs) : 494, 535
.   : milestone, 515,
iast_INACTIVE (428.489 µs) : 408, 449
.   : milestone, 428,
profiling (444.539 µs) : 423, 467
.   : milestone, 445,
tracing (449.013 µs) : 427, 471
.   : milestone, 449,
section candidate
no_agent (363.522 µs) : 344, 383
.   : milestone, 364,
appsec (672.262 µs) : 652, 693
.   : milestone, 672,
iast (460.001 µs) : 439, 481
.   : milestone, 460,
iast_FULL (528.415 µs) : 508, 549
.   : milestone, 528,
iast_INACTIVE (438.768 µs) : 417, 461
.   : milestone, 439,
profiling (446.781 µs) : 426, 467
.   : milestone, 447,
tracing (438.474 µs) : 417, 460
.   : milestone, 438,

baseline results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	367.775 µs [348.051 µs, 387.499 µs]	-
appsec	672.692 µs [652.145 µs, 693.239 µs]	304.917 µs (82.9%)
iast	467.795 µs [446.971 µs, 488.62 µs]	100.02 µs (27.2%)
iast_FULL	514.86 µs [494.347 µs, 535.373 µs]	147.085 µs (40.0%)
iast_INACTIVE	428.489 µs [407.749 µs, 449.23 µs]	60.715 µs (16.5%)
profiling	444.539 µs [422.574 µs, 466.504 µs]	76.764 µs (20.9%)
tracing	449.013 µs [427.466 µs, 470.56 µs]	81.238 µs (22.1%)

candidate results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	363.522 µs [343.7 µs, 383.344 µs]	-
appsec	672.262 µs [651.586 µs, 692.938 µs]	308.74 µs (84.9%)
iast	460.001 µs [438.955 µs, 481.047 µs]	96.479 µs (26.5%)
iast_FULL	528.415 µs [507.524 µs, 549.305 µs]	164.893 µs (45.4%)
iast_INACTIVE	438.768 µs [416.975 µs, 460.561 µs]	75.246 µs (20.7%)
profiling	446.781 µs [426.302 µs, 467.26 µs]	83.259 µs (22.9%)
tracing	438.474 µs [417.389 µs, 459.558 µs]	74.952 µs (20.6%)

Request duration reports for petclinic

gantt
    title petclinic - request duration [CI 0.99] : candidate=1.22.0-SNAPSHOT~d45b54c0b6, baseline=1.22.0-SNAPSHOT~ecccc528c0
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.337 ms) : 1318, 1356
.   : milestone, 1337,
appsec (1.694 ms) : 1670, 1719
.   : milestone, 1694,
iast (1.462 ms) : 1437, 1486
.   : milestone, 1462,
profiling (1.47 ms) : 1443, 1496
.   : milestone, 1470,
tracing (1.45 ms) : 1425, 1475
.   : milestone, 1450,
section candidate
no_agent (1.343 ms) : 1324, 1362
.   : milestone, 1343,
appsec (1.693 ms) : 1668, 1718
.   : milestone, 1693,
iast (1.462 ms) : 1437, 1486
.   : milestone, 1462,
profiling (1.462 ms) : 1437, 1487
.   : milestone, 1462,
tracing (1.455 ms) : 1430, 1479
.   : milestone, 1455,

baseline results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	1.337 ms [1.318 ms, 1.356 ms]	-
appsec	1.694 ms [1.67 ms, 1.719 ms]	357.023 µs (26.7%)
iast	1.462 ms [1.437 ms, 1.486 ms]	124.413 µs (9.3%)
profiling	1.47 ms [1.443 ms, 1.496 ms]	132.687 µs (9.9%)
tracing	1.45 ms [1.425 ms, 1.475 ms]	112.519 µs (8.4%)

candidate results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	1.343 ms [1.324 ms, 1.362 ms]	-
appsec	1.693 ms [1.668 ms, 1.718 ms]	349.339 µs (26.0%)
iast	1.462 ms [1.437 ms, 1.486 ms]	118.227 µs (8.8%)
profiling	1.462 ms [1.437 ms, 1.487 ms]	118.73 µs (8.8%)
tracing	1.455 ms [1.43 ms, 1.479 ms]	111.182 µs (8.3%)

...va-sqs-2.0/src/main/java/datadog/trace/instrumentation/aws/v2/sqs/MessageExtractAdapter.java

The '_datadog' message attribute is used by DSM and certain language tracers to propagate context instead of AWS' 'AWSTraceHeader'. The attribute contains trace context as a JSON string, or encoded with Base64 as a binary value.

mcculls added the inst: aws sdk AWS SDK instrumentation label Sep 20, 2023

mcculls requested a review from a team as a code owner September 20, 2023 11:38

mcculls requested review from PerfectSlayer and am312 September 20, 2023 11:38

mcculls added the type: enhancement Enhancements and improvements label Sep 20, 2023

mcculls marked this pull request as draft September 20, 2023 11:54

mcculls changed the title ~~Extract trace context from embedded SQS message attribute '_datadog'~~ [WIPExtract trace context from embedded SQS message attribute '_datadog' Sep 20, 2023

mcculls changed the title ~~[WIPExtract trace context from embedded SQS message attribute '_datadog'~~ [WIP] Extract trace context from embedded SQS message attribute '_datadog' Sep 20, 2023

mcculls force-pushed the mcculls/handle-sqs-datadog-attributes branch from 08474ef to a8a986a Compare September 20, 2023 13:55

mcculls changed the title ~~[WIP] Extract trace context from embedded SQS message attribute '_datadog'~~ Extract trace context from embedded SQS message attribute '_datadog' Sep 20, 2023

mcculls force-pushed the mcculls/handle-sqs-datadog-attributes branch 2 times, most recently from 6acdb58 to 32350ac Compare September 20, 2023 14:27

mcculls marked this pull request as ready for review September 20, 2023 14:27

mcculls force-pushed the mcculls/handle-sqs-datadog-attributes branch 3 times, most recently from dd68cd9 to 9c2ff00 Compare September 20, 2023 18:35

PerfectSlayer reviewed Sep 21, 2023

View reviewed changes

...va-sqs-2.0/src/main/java/datadog/trace/instrumentation/aws/v2/sqs/MessageExtractAdapter.java Outdated Show resolved Hide resolved

mcculls force-pushed the mcculls/handle-sqs-datadog-attributes branch 5 times, most recently from 5f49d35 to 8123129 Compare September 21, 2023 23:59

mcculls force-pushed the mcculls/handle-sqs-datadog-attributes branch from 8123129 to d45b54c Compare September 22, 2023 00:04

PerfectSlayer approved these changes Sep 22, 2023

View reviewed changes

mcculls merged commit 87c51d2 into master Sep 22, 2023

mcculls deleted the mcculls/handle-sqs-datadog-attributes branch September 22, 2023 17:09

github-actions bot added this to the 1.22.0 milestone Sep 22, 2023

vandonr mentioned this pull request Jul 12, 2024

Fix parsing of binary datadog headers in SQS #7324

Merged

ygree mentioned this pull request Mar 13, 2026

Inject APM trace headers into SQS _datadog message attribute on send #10843

Draft

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Extract trace context from embedded SQS message attribute '_datadog'#5920

Extract trace context from embedded SQS message attribute '_datadog'#5920
mcculls merged 1 commit intomasterfrom
mcculls/handle-sqs-datadog-attributes

mcculls commented Sep 20, 2023

Uh oh!

pr-commenter bot commented Sep 20, 2023 •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

mcculls commented Sep 20, 2023

Motivation

Uh oh!

pr-commenter bot commented Sep 20, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Benchmarks

Startup

Parameters

Summary

Load

Parameters

Summary

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

pr-commenter bot commented Sep 20, 2023 •

edited

Loading