Skip to content

Commit

Permalink
Removed observability code.
Browse files Browse the repository at this point in the history 
A Service Weaver application has logs, traces, and metrics. Before this
PR, `weaver kube deploy` generated Kubernetes YAML to set up Prometheus,
Jaegar, Grafana, and Loki. The `weaver kube` babysitter exported logs,
traces, and metrics to these systems. However, there was no way for a
user to customize how telemetry was handled. If you wanted to export
traces to Datadog, for example, well, you couldn't.

To address this problem in the long term, we're considering introducing
a general purpose mechanism that would allow developers to customize
deployers using a set of components. A metric exporting component, for
example, would customize how metrics are exported.

In the short term, we're going to introduce a much simpler and less
general approach where a developer can plug in custom ways to manage
telemetry. This also means that `weaver kube` no longer has to generate
YAML to run Prometheus, Jaegar, and whatnot. Instead, they will be
plugins.
  • Loading branch information
@mwhittaker
mwhittaker committed Oct 25, 2023
1 parent 2b300ff commit 207ef1d
Show file tree
Hide file tree
Showing 10 changed files with 91 additions and 1,850 deletions.
25 changes: 1 addition & 24 deletions cmd/weaver-kube/deploy.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -83,30 +83,7 @@ Container Image Names:
You can specify any combination of the various options or none. E.g.,
listeners.foo = {public = true, serice_name = "unique_name"}
c) Observability - if nothing is specified, the kube deployer will
automatically launch Prometheus, Jaeger, Loki and Grafana to retrieve your
application's metrics, traces, logs, and to provide custom dashboards.
If you don't want one or more of these services to run, you can simply
disable them. E.g., :
[kube.observability]
prometheus_service = "none"
jaeger_service = "none"
loki_service = "none"
grafana_service = "none"
If you want to plugin one or more of your existing Prometheus, Jaeger,
Loki, Grafana, you can specify their service name:
[kube.observability]
prometheus_service = "your_prometheus_service_name"
jaeger_service = "your_jaeger_service_name"
loki_service = "your_loki_service_name"
grafana_service = "your_granfa_service_name"
Note that we support only the Prometheus, Jaeger, Loki, Grafana stack for
observability right now.
d) Configure resource requirements for the pods [1]. E.g.,
c) Configure resource requirements for the pods [1]. E.g.,
[kube.resources]
requests_cpu = "200m"
requests_mem = "256Mi"
Expand Down
3 changes: 1 addition & 2 deletions go.mod
View file
Original file line number Diff line number Diff line change
Expand Up @@ -6,8 +6,6 @@ require (
github.com/ServiceWeaver/weaver v0.22.0
github.com/google/uuid v1.3.1
go.opentelemetry.io/otel v1.19.0
go.opentelemetry.io/otel/exporters/jaeger v1.16.0
go.opentelemetry.io/otel/sdk v1.16.0
go.opentelemetry.io/otel/trace v1.19.0
golang.org/x/exp v0.0.0-20230713183714-613f0c0eb8a1
golang.org/x/sync v0.3.0
Expand Down Expand Up @@ -54,6 +52,7 @@ require (
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.45.0 // indirect
go.opentelemetry.io/otel/exporters/stdout/stdouttrace v1.16.0 // indirect
go.opentelemetry.io/otel/metric v1.19.0 // indirect
go.opentelemetry.io/otel/sdk v1.16.0 // indirect
golang.org/x/mod v0.12.0 // indirect
golang.org/x/net v0.17.0 // indirect
golang.org/x/oauth2 v0.0.0-20220223155221-ee480838109b // indirect
Expand Down
3 changes: 0 additions & 3 deletions go.sum
View file
Original file line number Diff line number Diff line change
Expand Up @@ -208,7 +208,6 @@ github.com/stoewer/go-strcase v1.2.0 h1:Z2iHWqGXH00XYgqDmNgQbIBxf3wrNq0F3feEy0ai
github.com/stoewer/go-strcase v1.2.0/go.mod h1:IBiWB2sKIp3wVVQ3Y035++gc+knqhUQag1KpM8ahLw8=
github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
github.com/stretchr/objx v0.4.0/go.mod h1:YvHI0jy2hoMjB+UWwv71VJQ9isScKT/TqJzVSSt89Yw=
github.com/stretchr/objx v0.5.0 h1:1zr/of2m5FGMsad5YfcqgdqdWrIhu+EBEJRhR1U7z/c=
github.com/stretchr/objx v0.5.0/go.mod h1:Yh+to48EsGEfYuaHDzXPcE3xhTkx73EhmCGUpEOglKo=
github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI=
github.com/stretchr/testify v1.4.0/go.mod h1:j7eGeouHqKxXV5pUuKE4zz7dFj8WfuZ+81PSLYec5m4=
Expand All @@ -231,8 +230,6 @@ go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.45.0 h1:x8Z78aZ
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.45.0/go.mod h1:62CPTSry9QZtOaSsE3tOzhx6LzDhHnXJ6xHeMNNiM6Q=
go.opentelemetry.io/otel v1.19.0 h1:MuS/TNf4/j4IXsZuJegVzI1cwut7Qc00344rgH7p8bs=
go.opentelemetry.io/otel v1.19.0/go.mod h1:i0QyjOq3UPoTzff0PJB2N66fb4S0+rSbSB15/oyH9fY=
go.opentelemetry.io/otel/exporters/jaeger v1.16.0 h1:YhxxmXZ011C0aDZKoNw+juVWAmEfv/0W2XBOv9aHTaA=
go.opentelemetry.io/otel/exporters/jaeger v1.16.0/go.mod h1:grYbBo/5afWlPpdPZYhyn78Bk04hnvxn2+hvxQhKIQM=
go.opentelemetry.io/otel/exporters/stdout/stdouttrace v1.16.0 h1:+XWJd3jf75RXJq29mxbuXhCXFDG3S3R4vBUeSI2P7tE=
go.opentelemetry.io/otel/exporters/stdout/stdouttrace v1.16.0/go.mod h1:hqgzBPTf4yONMFgdZvL/bK42R/iinTyVQtiWihs3SZc=
go.opentelemetry.io/otel/metric v1.19.0 h1:aTzpGtV0ar9wlV4Sna9sdJyII5jTVJEvKETPiOKwvpE=
Expand Down
65 changes: 20 additions & 45 deletions internal/impl/babysitter.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -32,10 +32,7 @@ import (
"github.com/ServiceWeaver/weaver/runtime/metrics"
"github.com/ServiceWeaver/weaver/runtime/prometheus"
"github.com/ServiceWeaver/weaver/runtime/protos"
"github.com/ServiceWeaver/weaver/runtime/traces"
"github.com/google/uuid"
"go.opentelemetry.io/otel/exporters/jaeger"
"go.opentelemetry.io/otel/sdk/trace"
"golang.org/x/sync/errgroup"
v1 "k8s.io/api/core/v1"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
Expand All @@ -54,14 +51,13 @@ var logDir = filepath.Join(runtime.LogsDir(), "kube")

// babysitter starts and manages a weavelet inside the Pod.
type babysitter struct {
ctx context.Context
cfg *BabysitterConfig
app *protos.AppConfig
envelope *envelope.Envelope
logger *slog.Logger
traceExporter *jaeger.Exporter
clientset *kubernetes.Clientset
printer *logging.PrettyPrinter
ctx context.Context
cfg *BabysitterConfig
app *protos.AppConfig
envelope *envelope.Envelope
logger *slog.Logger
clientset *kubernetes.Clientset
printer *logging.PrettyPrinter

mu sync.Mutex
watching map[string]struct{} // components being watched
Expand Down Expand Up @@ -98,18 +94,6 @@ func NewBabysitter(ctx context.Context, app *protos.AppConfig, config *Babysitte
Write: logSaver,
})

// Create the trace exporter.
var traceExporter *jaeger.Exporter
if config.TraceServiceUrl != "" {
// Export traces if there is a tracing service running that is able to
// receive these traces.
endpoint := jaeger.WithCollectorEndpoint(jaeger.WithEndpoint(config.TraceServiceUrl))
traceExporter, err = jaeger.New(endpoint)
if err != nil {
return nil, fmt.Errorf("NewBabysitter: create trace exporter: %w", err)
}
}

// Create a Kubernetes config.
kubeConfig, err := rest.InClusterConfig()
if err != nil {
Expand All @@ -122,15 +106,14 @@ func NewBabysitter(ctx context.Context, app *protos.AppConfig, config *Babysitte

// Create the babysitter.
b := &babysitter{
ctx: ctx,
cfg: config,
app: app,
envelope: e,
logger: logger,
traceExporter: traceExporter,
clientset: clientset,
printer: logging.NewPrettyPrinter(false /*colors disabled*/),
watching: map[string]struct{}{},
ctx: ctx,
cfg: config,
app: app,
envelope: e,
logger: logger,
clientset: clientset,
printer: logging.NewPrettyPrinter(false /*colors disabled*/),
watching: map[string]struct{}{},
}

// Inform the weavelet of the components it should host.
Expand All @@ -143,9 +126,9 @@ func NewBabysitter(ctx context.Context, app *protos.AppConfig, config *Babysitte

func (b *babysitter) Serve() error {
// Run an HTTP server that exports metrics.
lis, err := net.Listen("tcp", fmt.Sprintf(":%d", defaultMetricsPort))
lis, err := net.Listen("tcp", fmt.Sprintf(":%d", prometheusPort))
if err != nil {
return fmt.Errorf("Babysitter.Serve: listen on port %d: %w", defaultMetricsPort, err)
return fmt.Errorf("Babysitter.Serve: listen on port %d: %w", prometheusPort, err)
}
mux := http.NewServeMux()
mux.HandleFunc(prometheusEndpoint, func(w http.ResponseWriter, r *http.Request) {
Expand All @@ -165,9 +148,7 @@ func (b *babysitter) Serve() error {
return b.envelope.Serve(b)
})

err = group.Wait()
b.traceExporter.Shutdown(b.ctx) //nolint:errcheck // response write error
return err
return group.Wait()
}

// ActivateComponent implements the envelope.EnvelopeHandler interface.
Expand Down Expand Up @@ -274,14 +255,8 @@ func (b *babysitter) HandleLogEntry(_ context.Context, entry *protos.LogEntry) e

// HandleTraceSpans implements the envelope.EnvelopeHandler interface.
func (b *babysitter) HandleTraceSpans(ctx context.Context, spans *protos.TraceSpans) error {
if b.traceExporter == nil {
return nil
}
var spansToExport []trace.ReadOnlySpan
for _, span := range spans.Span {
spansToExport = append(spansToExport, &traces.ReadSpan{Span: span})
}
return b.traceExporter.ExportSpans(ctx, spansToExport)
// TODO(mwhittaker): Implement with plugins.
return nil
}

// GetSelfCertificate implements the envelope.EnvelopeHandler interface.
Expand Down
35 changes: 0 additions & 35 deletions internal/impl/config.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -64,41 +64,6 @@ type kubeConfig struct {
// If a listener isn't specified in the map, default options will be used.
Listeners map[string]*listenerConfig

// Observability controls how the deployer will export observability information
// such as logs, metrics and traces, keyed by service. If no options are
// specified, the deployer will launch corresponding services for exporting logs,
// metrics and traces automatically.
//
// The key must be one of the following strings:
// "prometheus_service" - to export metrics to Prometheus [1]
// "jaeger_service" - to export traces to Jaeger [2]
// "loki_service" - to export logs to Grafana Loki [3]
// "grafana_service" - to visualize/manipulate observability information [4]
//
// Possible values for each service:
// 1) do not specify a value at all; leave it empty
// this is the default behavior; kube deployer will automatically create the
// observability service for you.
//
// 2) "none"
// kube deployer will not export the corresponding observability information to
// any service. E.g., prometheus_service = "none" means that the user will not
// be able to see any metrics at all. This can be useful for testing or
// benchmarking the performance of your application.
//
// 3) "your_observability_service_name"
// if you already have a running service to collect metrics, traces or logs,
// then you can simply specify the service name, and your application will
// automatically export the corresponding information to your service. E.g.,
// jaeger_service = "jaeger-all-in-one" will enable your running Jaeger
// "service/jaeger-all-in-one" to capture all the app traces.
//
// [1] - https://prometheus.io/
// [2] - https://www.jaegertracing.io/
// [3] - https://grafana.com/oss/loki/
// [4] - https://grafana.com/
Observability map[string]string

// Resources needed to run the pods. Note that the resources should satisfy
// the format specified in [1].
//
Expand Down
Loading

0 comments on commit 207ef1d

Please sign in to comment.