Add observability stack with jaeger and opentelemetry-collector

[NickLarsenNZ]

Make a Stack to aid in development and validation of webhook telemetry (and beyond). This will evolve into a full demo when operators/webhooks and/or products have been instrumented.

Stack

• Jaeger Deployment
  • Actually also added Loki (logs) and Tempo (a second trace sink to show that multiple can be used), and the Grafana dashboard app.
• OpenTelemetry Operator Deployment
  • OpenTelemetry Collector to route logs and traces
• Deploy an instrumented operator with a webhook
  • The commons operator is deployed by this stack, but is not yet instrumented.

Demo

Important

A demo is not included for now, because there is no operator/webhook that has been instrumented yet. This can be done in a future PR.

[soenkeliebau]

Just on very high level comment, because I was working on integrating openobserve into the demos somewhere else, which would in theory also support OpenTelemetry I believe.

Is it maybe worth taking a brief look at what Jaeger gives us more than OpenObserve? If OpenObserve suffices, we'd have one tool to take care of logs, metrics and traces all in one fell swoop for the demos ..

[NickLarsenNZ]

Just on very high level comment, because I was working on integrating openobserve into the demos somewhere else, which would in theory also support OpenTelemetry I believe.

Is it maybe worth taking a brief look at what Jaeger gives us more than OpenObserve? If OpenObserve suffices, we'd have one tool to take care of logs, metrics and traces all in one fell swoop for the demos ..

I've gone with the Grafana stack as it has been IMO the best all-round observability stack.
Jaeger was added as an additional tool to show how the opentelemetry-collector can take one stream of logs/metrics/traces and send them to multiple things (out of scope, but the benefit is allowing end users to easily migrate to other tools without a hard cut which is often required when observability SaaS vendors' SDKs are used in code. In Rust, the tracing crate helps there too).

At this point in time, only the stack exists, so it wasn't described, but my intention is that a demo would explain all of this.

[NickLarsenNZ]

This is basically done, but I was waiting on a couple of changes to go into stackable-webhooks/stackable-telemetry (see task list in stackabletech/issues#531) to ensure a problem wasn't being created by the opentelemetry-collector.

I have some local changes/clean up to finalize this (I aim to do that after my holiday).

Note

It is only a stack at this point, and not an entire demo (that is best left to when we have instrumented real webhooks/operators and/or enabled instrumentation on products).

[sbernauer]

• Please expose using NodePorts
• Deploy collector Pod (and not only sidecars)
• URLs in stacks.yaml need updating (just before merging)

[NickLarsenNZ]

• Please expose using NodePorts

I have done so for Grafana (b690482), but am unable to do the same for the OpenTelemetry Collector (see open-telemetry/opentelemetry-operator#902).

• Deploy collector Pod (and not only sidecars)

Done in d9336f2

• URLs in stacks.yaml need updating (just before merging)

Done in 8bea6ae

[sbernauer]

LGTM. Can you please add a comment everywhere you were not able to use a NodePort including the link to the issue if possible?

[sbernauer]

Thanks!

[NickLarsenNZ]

LGTM. Can you please add a comment everywhere you were not able to use a NodePort including the link to the issue if possible?

Done in d8c58d0