Add Sofa RPC instrumentation (Bolt, H2C, REST, Triple)#11135
Add Sofa RPC instrumentation (Bolt, H2C, REST, Triple)#11135ValentinZakharov wants to merge 10 commits intomasterfrom
Conversation
BenchmarksStartupParameters
See matching parameters
SummaryFound 0 performance improvements and 0 performance regressions! Performance is the same for 62 metrics, 9 unstable metrics. Startup time reports for insecure-bankgantt
title insecure-bank - global startup overhead: candidate=1.62.0-SNAPSHOT~614e75b995, baseline=1.62.0-SNAPSHOT~6a692d2803
dateFormat X
axisFormat %s
section tracing
Agent [baseline] (1.063 s) : 0, 1062882
Total [baseline] (8.826 s) : 0, 8825994
Agent [candidate] (1.054 s) : 0, 1054354
Total [candidate] (8.823 s) : 0, 8823104
section iast
Agent [baseline] (1.231 s) : 0, 1231159
Total [baseline] (9.486 s) : 0, 9486350
Agent [candidate] (1.238 s) : 0, 1237651
Total [candidate] (9.496 s) : 0, 9496307
gantt
title insecure-bank - break down per module: candidate=1.62.0-SNAPSHOT~614e75b995, baseline=1.62.0-SNAPSHOT~6a692d2803
dateFormat X
axisFormat %s
section tracing
crashtracking [baseline] (1.241 ms) : 0, 1241
crashtracking [candidate] (1.214 ms) : 0, 1214
BytebuddyAgent [baseline] (637.203 ms) : 0, 637203
BytebuddyAgent [candidate] (634.122 ms) : 0, 634122
AgentMeter [baseline] (29.827 ms) : 0, 29827
AgentMeter [candidate] (29.624 ms) : 0, 29624
GlobalTracer [baseline] (249.897 ms) : 0, 249897
GlobalTracer [candidate] (248.502 ms) : 0, 248502
AppSec [baseline] (32.738 ms) : 0, 32738
AppSec [candidate] (32.341 ms) : 0, 32341
Debugger [baseline] (59.49 ms) : 0, 59490
Debugger [candidate] (58.915 ms) : 0, 58915
Remote Config [baseline] (601.405 µs) : 0, 601
Remote Config [candidate] (597.163 µs) : 0, 597
Telemetry [baseline] (8.069 ms) : 0, 8069
Telemetry [candidate] (8.0 ms) : 0, 8000
Flare Poller [baseline] (7.524 ms) : 0, 7524
Flare Poller [candidate] (5.014 ms) : 0, 5014
section iast
crashtracking [baseline] (1.237 ms) : 0, 1237
crashtracking [candidate] (1.23 ms) : 0, 1230
BytebuddyAgent [baseline] (809.708 ms) : 0, 809708
BytebuddyAgent [candidate] (816.864 ms) : 0, 816864
AgentMeter [baseline] (11.419 ms) : 0, 11419
AgentMeter [candidate] (11.544 ms) : 0, 11544
GlobalTracer [baseline] (238.933 ms) : 0, 238933
GlobalTracer [candidate] (239.234 ms) : 0, 239234
AppSec [baseline] (25.851 ms) : 0, 25851
AppSec [candidate] (26.352 ms) : 0, 26352
Debugger [baseline] (66.317 ms) : 0, 66317
Debugger [candidate] (63.276 ms) : 0, 63276
Remote Config [baseline] (526.023 µs) : 0, 526
Remote Config [candidate] (523.784 µs) : 0, 524
Telemetry [baseline] (7.708 ms) : 0, 7708
Telemetry [candidate] (7.666 ms) : 0, 7666
Flare Poller [baseline] (3.346 ms) : 0, 3346
Flare Poller [candidate] (3.39 ms) : 0, 3390
IAST [baseline] (29.266 ms) : 0, 29266
IAST [candidate] (31.531 ms) : 0, 31531
Startup time reports for petclinicgantt
title petclinic - global startup overhead: candidate=1.62.0-SNAPSHOT~614e75b995, baseline=1.62.0-SNAPSHOT~6a692d2803
dateFormat X
axisFormat %s
section tracing
Agent [baseline] (1.066 s) : 0, 1065871
Total [baseline] (11.069 s) : 0, 11069438
Agent [candidate] (1.052 s) : 0, 1051917
Total [candidate] (11.044 s) : 0, 11043610
section appsec
Agent [baseline] (1.258 s) : 0, 1258334
Total [baseline] (10.951 s) : 0, 10950677
Agent [candidate] (1.258 s) : 0, 1258009
Total [candidate] (11.071 s) : 0, 11071140
section iast
Agent [baseline] (1.242 s) : 0, 1242488
Total [baseline] (11.223 s) : 0, 11222609
Agent [candidate] (1.235 s) : 0, 1234569
Total [candidate] (11.315 s) : 0, 11314688
section profiling
Agent [baseline] (1.181 s) : 0, 1180973
Total [baseline] (11.007 s) : 0, 11007167
Agent [candidate] (1.185 s) : 0, 1184660
Total [candidate] (10.972 s) : 0, 10971630
gantt
title petclinic - break down per module: candidate=1.62.0-SNAPSHOT~614e75b995, baseline=1.62.0-SNAPSHOT~6a692d2803
dateFormat X
axisFormat %s
section tracing
crashtracking [baseline] (1.229 ms) : 0, 1229
crashtracking [candidate] (1.207 ms) : 0, 1207
BytebuddyAgent [baseline] (637.356 ms) : 0, 637356
BytebuddyAgent [candidate] (632.354 ms) : 0, 632354
AgentMeter [baseline] (29.727 ms) : 0, 29727
AgentMeter [candidate] (29.406 ms) : 0, 29406
GlobalTracer [baseline] (250.867 ms) : 0, 250867
GlobalTracer [candidate] (247.883 ms) : 0, 247883
AppSec [baseline] (32.597 ms) : 0, 32597
AppSec [candidate] (32.336 ms) : 0, 32336
Debugger [baseline] (60.189 ms) : 0, 60189
Debugger [candidate] (59.66 ms) : 0, 59660
Remote Config [baseline] (605.461 µs) : 0, 605
Remote Config [candidate] (591.384 µs) : 0, 591
Telemetry [baseline] (8.139 ms) : 0, 8139
Telemetry [candidate] (8.831 ms) : 0, 8831
Flare Poller [baseline] (8.917 ms) : 0, 8917
Flare Poller [candidate] (3.607 ms) : 0, 3607
section appsec
crashtracking [baseline] (1.221 ms) : 0, 1221
crashtracking [candidate] (1.219 ms) : 0, 1219
BytebuddyAgent [baseline] (672.992 ms) : 0, 672992
BytebuddyAgent [candidate] (673.528 ms) : 0, 673528
AgentMeter [baseline] (12.099 ms) : 0, 12099
AgentMeter [candidate] (12.199 ms) : 0, 12199
GlobalTracer [baseline] (248.209 ms) : 0, 248209
GlobalTracer [candidate] (247.809 ms) : 0, 247809
AppSec [baseline] (185.642 ms) : 0, 185642
AppSec [candidate] (184.618 ms) : 0, 184618
Debugger [baseline] (65.692 ms) : 0, 65692
Debugger [candidate] (66.379 ms) : 0, 66379
Remote Config [baseline] (581.246 µs) : 0, 581
Remote Config [candidate] (570.565 µs) : 0, 571
Telemetry [baseline] (7.884 ms) : 0, 7884
Telemetry [candidate] (7.759 ms) : 0, 7759
Flare Poller [baseline] (3.451 ms) : 0, 3451
Flare Poller [candidate] (3.398 ms) : 0, 3398
IAST [baseline] (24.289 ms) : 0, 24289
IAST [candidate] (24.277 ms) : 0, 24277
section iast
crashtracking [baseline] (1.23 ms) : 0, 1230
crashtracking [candidate] (1.224 ms) : 0, 1224
BytebuddyAgent [baseline] (816.935 ms) : 0, 816935
BytebuddyAgent [candidate] (810.737 ms) : 0, 810737
AgentMeter [baseline] (11.641 ms) : 0, 11641
AgentMeter [candidate] (11.486 ms) : 0, 11486
GlobalTracer [baseline] (241.787 ms) : 0, 241787
GlobalTracer [candidate] (240.248 ms) : 0, 240248
AppSec [baseline] (28.505 ms) : 0, 28505
AppSec [candidate] (27.511 ms) : 0, 27511
Debugger [baseline] (63.553 ms) : 0, 63553
Debugger [candidate] (64.754 ms) : 0, 64754
Remote Config [baseline] (529.953 µs) : 0, 530
Remote Config [candidate] (530.936 µs) : 0, 531
Telemetry [baseline] (7.764 ms) : 0, 7764
Telemetry [candidate] (7.78 ms) : 0, 7780
Flare Poller [baseline] (3.395 ms) : 0, 3395
Flare Poller [candidate] (3.493 ms) : 0, 3493
IAST [baseline] (30.973 ms) : 0, 30973
IAST [candidate] (30.777 ms) : 0, 30777
section profiling
crashtracking [baseline] (1.187 ms) : 0, 1187
crashtracking [candidate] (1.172 ms) : 0, 1172
BytebuddyAgent [baseline] (689.073 ms) : 0, 689073
BytebuddyAgent [candidate] (691.46 ms) : 0, 691460
AgentMeter [baseline] (8.899 ms) : 0, 8899
AgentMeter [candidate] (9.027 ms) : 0, 9027
GlobalTracer [baseline] (207.03 ms) : 0, 207030
GlobalTracer [candidate] (207.436 ms) : 0, 207436
AppSec [baseline] (32.523 ms) : 0, 32523
AppSec [candidate] (32.559 ms) : 0, 32559
Debugger [baseline] (65.517 ms) : 0, 65517
Debugger [candidate] (65.846 ms) : 0, 65846
Remote Config [baseline] (579.107 µs) : 0, 579
Remote Config [candidate] (569.965 µs) : 0, 570
Telemetry [baseline] (7.854 ms) : 0, 7854
Telemetry [candidate] (7.783 ms) : 0, 7783
Flare Poller [baseline] (3.511 ms) : 0, 3511
Flare Poller [candidate] (3.508 ms) : 0, 3508
ProfilingAgent [baseline] (93.592 ms) : 0, 93592
ProfilingAgent [candidate] (94.044 ms) : 0, 94044
Profiling [baseline] (94.153 ms) : 0, 94153
Profiling [candidate] (94.608 ms) : 0, 94608
LoadParameters
See matching parameters
SummaryFound 0 performance improvements and 3 performance regressions! Performance is the same for 17 metrics, 16 unstable metrics.
Request duration reports for petclinicgantt
title petclinic - request duration [CI 0.99] : candidate=1.62.0-SNAPSHOT~614e75b995, baseline=1.62.0-SNAPSHOT~6a692d2803
dateFormat X
axisFormat %s
section baseline
no_agent (17.254 ms) : 17083, 17425
. : milestone, 17254,
appsec (18.382 ms) : 18197, 18568
. : milestone, 18382,
code_origins (17.857 ms) : 17680, 18035
. : milestone, 17857,
iast (18.055 ms) : 17876, 18233
. : milestone, 18055,
profiling (18.368 ms) : 18185, 18550
. : milestone, 18368,
tracing (17.945 ms) : 17766, 18124
. : milestone, 17945,
section candidate
no_agent (19.316 ms) : 19119, 19513
. : milestone, 19316,
appsec (19.155 ms) : 18961, 19349
. : milestone, 19155,
code_origins (17.853 ms) : 17677, 18029
. : milestone, 17853,
iast (18.08 ms) : 17900, 18260
. : milestone, 18080,
profiling (18.136 ms) : 17955, 18316
. : milestone, 18136,
tracing (17.863 ms) : 17688, 18038
. : milestone, 17863,
Request duration reports for insecure-bankgantt
title insecure-bank - request duration [CI 0.99] : candidate=1.62.0-SNAPSHOT~614e75b995, baseline=1.62.0-SNAPSHOT~6a692d2803
dateFormat X
axisFormat %s
section baseline
no_agent (1.238 ms) : 1226, 1250
. : milestone, 1238,
iast (3.359 ms) : 3309, 3408
. : milestone, 3359,
iast_FULL (6.061 ms) : 5999, 6124
. : milestone, 6061,
iast_GLOBAL (3.611 ms) : 3549, 3673
. : milestone, 3611,
profiling (2.089 ms) : 2068, 2109
. : milestone, 2089,
tracing (1.976 ms) : 1959, 1993
. : milestone, 1976,
section candidate
no_agent (1.279 ms) : 1266, 1292
. : milestone, 1279,
iast (3.312 ms) : 3267, 3358
. : milestone, 3312,
iast_FULL (6.134 ms) : 6071, 6196
. : milestone, 6134,
iast_GLOBAL (3.559 ms) : 3501, 3617
. : milestone, 3559,
profiling (2.277 ms) : 2254, 2299
. : milestone, 2277,
tracing (1.912 ms) : 1894, 1931
. : milestone, 1912,
DacapoParameters
See matching parameters
SummaryFound 1 performance improvements and 0 performance regressions! Performance is the same for 10 metrics, 1 unstable metrics.
Execution time for biojavagantt
title biojava - execution time [CI 0.99] : candidate=1.62.0-SNAPSHOT~614e75b995, baseline=1.62.0-SNAPSHOT~6a692d2803
dateFormat X
axisFormat %s
section baseline
no_agent (15.423 s) : 15423000, 15423000
. : milestone, 15423000,
appsec (14.597 s) : 14597000, 14597000
. : milestone, 14597000,
iast (18.317 s) : 18317000, 18317000
. : milestone, 18317000,
iast_GLOBAL (18.066 s) : 18066000, 18066000
. : milestone, 18066000,
profiling (14.906 s) : 14906000, 14906000
. : milestone, 14906000,
tracing (15.184 s) : 15184000, 15184000
. : milestone, 15184000,
section candidate
no_agent (15.034 s) : 15034000, 15034000
. : milestone, 15034000,
appsec (14.779 s) : 14779000, 14779000
. : milestone, 14779000,
iast (18.293 s) : 18293000, 18293000
. : milestone, 18293000,
iast_GLOBAL (17.712 s) : 17712000, 17712000
. : milestone, 17712000,
profiling (14.73 s) : 14730000, 14730000
. : milestone, 14730000,
tracing (14.742 s) : 14742000, 14742000
. : milestone, 14742000,
Execution time for tomcatgantt
title tomcat - execution time [CI 0.99] : candidate=1.62.0-SNAPSHOT~614e75b995, baseline=1.62.0-SNAPSHOT~6a692d2803
dateFormat X
axisFormat %s
section baseline
no_agent (1.485 ms) : 1474, 1497
. : milestone, 1485,
appsec (3.83 ms) : 3607, 4053
. : milestone, 3830,
iast (2.276 ms) : 2207, 2346
. : milestone, 2276,
iast_GLOBAL (2.321 ms) : 2251, 2392
. : milestone, 2321,
profiling (2.089 ms) : 2035, 2144
. : milestone, 2089,
tracing (2.098 ms) : 2044, 2153
. : milestone, 2098,
section candidate
no_agent (1.493 ms) : 1481, 1504
. : milestone, 1493,
appsec (2.548 ms) : 2493, 2603
. : milestone, 2548,
iast (2.276 ms) : 2206, 2345
. : milestone, 2276,
iast_GLOBAL (2.32 ms) : 2249, 2390
. : milestone, 2320,
profiling (2.523 ms) : 2360, 2687
. : milestone, 2523,
tracing (2.09 ms) : 2036, 2143
. : milestone, 2090,
|
96c7d44 to
ea98882
Compare
|
For info, I clean up the duplicate "inst:sofa rpc" (without the space after |
b3f3860 to
0555301
Compare
| @@ -0,0 +1 @@ | |||
| apply from: "$rootDir/gradle/java.gradle" | |||
There was a problem hiding this comment.
Note you can delete dd-java-agent/instrumentation/sofarpc/build.gradle and omit the ":dd-java-agent:instrumentation:sofarpc", entry from settings.gradle if this parent directory doesn't contain any source / resources.
mcculls
left a comment
There was a problem hiding this comment.
LGTM, thanks for the comprehensive tests
| class SofaRpcRestTest extends InstrumentationSpecification { | ||
|
|
||
| @Shared | ||
| int port = 12205 | ||
|
|
||
| @Shared | ||
| ProviderBootstrap restProviderBootstrap | ||
|
|
There was a problem hiding this comment.
Just curious if tests can be implemented with JUnit?
Recently we added a lot of functionality and base classes to simplify conversion from Groovy to JUnit.
Assuming this is new code it would benefit to have JUnit tests if possible
…n, scope/span ordering
…span hierarchy, add REST and Triple tests
…er spans, rpc.method tag
614e75b to
17ebcd0
Compare
What Does This Do
Adds auto-instrumentation for Sofa RPC, a high-perfomance RPC framework. Produces
sofarpc.requestclient and server spans with distributed trace propagation for all four supported transports: Bolt, H2C, REST and Triple(gRPC)Motivation
Sofa RPC is widely used in Ant Financial's ecosystem and it's not covered by any existing dd-trace-java integration. WIthout this instrumentaiton, traces break at the service boundary - downstream PRC calls are unlinked from traces
Additional Notes
Module structure
A single module
sofarpc-5.0covers all protocols and all public version of sofa-rpc. The framework was open-sources at 5.0.0; no earlier versions exists in public repositoriesHow server spans are coordinated
ProviderProxyInvoker.invoke()is a single entry point for all protocols on the server side - but it has noo info about the transport delivered the request. Transport specific instrumentations write the protocol name into ThreadLocal (SofaRpcProtocolContext) beforeinvoke()is called.ProviderProxuInvokerInstrumentaitonreads it to created the span and tag with the protocol. If the ThreadLocal is empty (uninstrumented transport), no span is created.Trace propagation by protocol
Bolt and H2C - context travels inside SofaRequest.requestProps (the framework's own key-value bag)

REST -

SofaRequest.requestPropsis not serialized as HTTP headers by the REST transport. Context instead travels as standard HTTP headers, injected and extracted by HTTP layer (Apache HttpClient on the client side, Netty on the server side). TheRestServerHandlerInstumentationsets the protocol context so thatProviderProxyInvoketInstrumentationactivated and creates asofarpc.request[server] span that naturally attaches to the activenetty.request` spanTriple - uses gRPC unses the hood. The grpc-1.5 instrumentation handles trace propagation via gRPC Metadata. The

TripleServerInstrumentationsetsprotocol="tri"so thatProviderProxyInvokerInstrumentaitoncreates asofarpc.request[server]span with child of the activegrpc.serverspanSpan attributes
Resource name format:
{rpc.service}/{rpc.method}(e.g.com.example.HelloService:1.0/sayHello)Known limitations
SofaRequest.getTaargetServiceUniqueName()returns null on the server side - the service name is not propagated thought the JAX-RS layer.Contributor Checklist
type:and (comp:orinst:) labels in addition to any other useful labelsclose,fix, or any linking keywords when referencing an issueUse
solvesinstead, and assign the PR milestone to the issueJira ticket: [PROJ-IDENT]
Note: Once your PR is ready to merge, add it to the merge queue by commenting
/merge./merge -ccancels the queue request./merge -f --reason "reason"skips all merge queue checks; please use this judiciously, as some checks do not run at the PR-level. For more information, see this doc.