agent side percentage sampling config option

[Frefreak]

Please answer these questions before submitting your issue.

• Why do you submit this issue?
• Question or discussion
• Bug
• Requirement
• Feature or performance improvement

---

Question

• What do you want to know?
  For sampling, it seems currently (v8.2.0/v8.3.0) there are primarily 2 ways to control trace sampling. One is in server side using sampleRate but the bandwidth is still wasted and may not be very flexible if has many multiple sources of agent. The other is via agent side agent.sample_n_per_3_secs option but the use case is kind of counter-intuitive. In many scenarios one may want a percentage sampling rate option in agent side. What's the reason of not providing that but using a "traces in 3 seconds" mechanism instead? This looks a bit odd and from looking at the SamplingService code a percentage way would not be difficult to implement.

---

Bug

• Which version of SkyWalking, OS and JRE?
  v8.2.0
  openjdk version "1.8.0_265"

---

Requirement or improvement

• provide an agent config similar to agent.sample_n_per_3_secs but in percentage (like 10 in every 10000). We can try to implement this and open a PR if that's OK.

[wu-sheng]

Hi, thanks for asking. The key is why do you need client-side sampleRate. I need to correct you

but the bandwidth is still wasted

This conclusion is not correct. The metrics of services/instance/etc. are depending on it. The sampling is just reducing the storage load, but the metrics are still accurate.

may not be very flexible if has many multiple sources of agent

I am not sure what do you mean about this. What kind of flexibility?

What's the reason of not providing that but using a "traces in 3 seconds" mechanism instead

I could ask the same question revertly, what is the reason for requiring rate sampling?

---

The key of the new feature is the scenario, what it could bring more.

[Frefreak]

Hi, thanks for the timely response. One of our use case is with MQ like Kafka.
My previous understanding is that if I only set sampleRate for server side, the headers will still be added for every message and stored in Kafka brokers (kafka storage and the traffic for sending it to brokers are not affected by this option, so the overhead may be un-negligible under a high amount of throughput). That's why I want a "client side" sampling. I'm not very sure of this behavior so please correct me if this is not the fact.

The sample n in 3 seconds client config works for us, but a percentage one would be easier to reason about when specifying this option since different topics may have different messages per seconds, in which case a percentage sampling rate seems more suitable.

[wu-sheng]

My previous understanding is that if I only set sampleRate for server side, the headers will still be added for every message and stored in Kafka brokers (kafka storage and the traffic for sending it to brokers are not affected by this option, so the overhead may be un-negligible under a high amount of throughput). That's why I want a "client side" sampling. I'm not very sure of this behavior so please correct me if this is not the fact.

First of all, the so-called service-side samplingRate has nothing about transportation, no matter for gRPC or Kafka. This is not related.

The sample n in 3 seconds client config works for us, but a percentage one would be easier to reason about when specifying this option since different topics may have different messages per seconds, in which case a percentage sampling rate seems more suitable

I think you have a misunderstanding about the sampling. It is not just for the network, sampling at the client side is for reducing the whole agent load, including tracing context, header injection/extraction, span creation/operations, etc. are all not working if not sampling.
Kafka reporter is just a pluggable and optional thing. No core-level mechanism related to it.

---

The key of all these things, SkyWalking is targeting the APM, rather than just tracing. So, metrics and topology really matter. Any sampling mechanism at the client side would have a side-effect on this. We can't have a clear scenario about sampling rate(client-side) is better than the current one.

Our final agenda is using on building SkyWalking Satellite, which could be deployed as a sidecar, so no network cost. Then it will analyze the traces to get metrics(we call those sources) and forward to the OAP.

[Frefreak]

First of all, the so-called service-side samplingRate has nothing about transportation, no matter for gRPC or Kafka. This is not related.
Thanks, this is clear now. We have understood that server side samplingRate is not what we want.

Let's focus on the client side. The already existing option agent.sample_n_per_3_secs is exactly what we want for sampling, but we feels like a "percentage sampling" similar to that would also be nice to have. At least it would be useful in a multi-service scenario.

[pierre94]

I also have a question about this configuration agent.sample_n_per_3_secs: why sample per 3 secs，not 1 or 4? Is there any special significance？

[wu-sheng]

but we feels like a "percentage sampling" similar to that would also be nice to have.

Why it is nice?

At least it would be useful in a multi-service scenario.

How useful? Do you know sampling rate is never accurate due to make sure the trace needs to continue(have a context in the header) despite not sampling = true

[wu-sheng]

I also have a question about this configuration agent.sample_n_per_3_secs: why sample per 3 secs，not 1 or 4? Is there any special significance？

2 directions,

1. If the reset period is too short, it always samples all requests, then there is no point.
2. If it is too large, then only the first several seconds of N have sampled values.