Distributed Tracing Collaboration between FSM and OpenTelemetry

Show the collaboration between FSM and Otel for distributed tracing.

This doc shows you the example bring your own (BYO) tracing colloctor for distributed tracing. The OpenTelemetry Collector works as the tracing collector to aggregate spans and sinks to Jaeger (in this example) or other system.

Prerequisites

  • Kubernetes cluster running Kubernetes v1.19.0 or greater.
  • FSM installed on the Kubernetes cluster.
  • kubectl installed and access to the cluster’s API server.
  • fsm CLI installed.

Jaeger

For the sake of demonstration, we use the jaegertracing/all-in-one image to deploy Jaeger here. This image includes components such as the Jaeger collector, memory storage, query service, and UI, making it very suitable for development and testing.

Enable support for OTLP (OpenTelemetry Protocol) through the environment variable COLLECTOR_OTLP_ENABLED.

kubectl apply -f - <<EOF
apiVersion: apps/v1
kind: Deployment
metadata:
  name: jaeger
spec:
  replicas: 1
  selector:
    matchLabels:
      app: jaeger
  template:
    metadata:
      labels:
        app: jaeger
    spec:
      containers:
      - name: jaeger
        image: jaegertracing/all-in-one:latest
        env:
        - name: COLLECTOR_OTLP_ENABLED
          value: "true"
        ports:
        - containerPort: 16686
        - containerPort: 14268
---
apiVersion: v1
kind: Service
metadata:
  name: jaeger
spec:
  selector:
    app: jaeger
  type: ClusterIP
  ports:
    - name: ui
      port: 16686
      targetPort: 16686
    - name: collector
      port: 14268
      targetPort: 14268
    - name: http
      protocol: TCP
      port: 4318
      targetPort: 4318
    - name: grpc
      protocol: TCP
      port: 4317
      targetPort: 4317      
EOF

Install cert-manager

The Otel Operator relies on cert-manager for certificate management, and cert-manager needs to be installed before installing the operator.

kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.13.2/cert-manager.yaml

Install OpenTelemetry Operator

Execute the following command to install Otel Operator.

kubectl apply -f https://github.com/open-telemetry/opentelemetry-operator/releases/latest/download/opentelemetry-operator.yaml

Configuring OpenTelemetry Collector

For detailed configuration of the Otel collector, refer to the official documentation.

  • Receivers: configure otlp to receive tracing information from applications, and zipkin to receive reports from sidecar, using endpoint 0.0.0.0:9411.
  • Exporters: configure Jager’s otlp endpoint jaeger.default:4317.
  • Pipeline services: use otlp and zipkin as input sources, directing output to jaeger.
kubectl apply -f - <<EOF
apiVersion: opentelemetry.io/v1alpha1
kind: OpenTelemetryCollector
metadata:
  name: otel
spec:
  config: |
    receivers:
      zipkin:
        endpoint: "0.0.0.0:9411"

    exporters:
      otlp/jaeger:
        endpoint: "jaeger.default:4317"
        tls:
          insecure: true

    service:
      pipelines:
        traces:
          receivers: [zipkin]
          exporters: [otlp/jaeger]
EOF

Update Mesh Configuration

In order to aggregate spans to OpenTelemetry Collector, we need to let mesh know the address of aggregator.

Following the FSM Tracing Doc, we can enable it during installation or update the configuration after installed.

kubectl patch meshconfig fsm-mesh-config -n fsm-system -p '{"spec":{"observability":{"tracing":{"enable":true,"address": "otel-collector.default","port":9411,"endpoint":"/api/v2/spans"}}}}'  --type=merge

Deploy Sample Application

kubectl create namespace bookstore
kubectl create namespace bookbuyer
kubectl create namespace bookthief
kubectl create namespace bookwarehouse
fsm namespace add bookstore bookbuyer bookthief bookwarehouse
kubectl apply -f https://raw.githubusercontent.com/flomesh-io/fsm-docs/main/manifests/apps/bookbuyer.yaml
kubectl apply -f https://raw.githubusercontent.com/flomesh-io/fsm-docs/main/manifests/apps/bookthief.yaml
kubectl apply -f https://raw.githubusercontent.com/flomesh-io/fsm-docs/main/manifests/apps/bookstore.yaml
kubectl apply -f https://raw.githubusercontent.com/flomesh-io/fsm-docs/main/manifests/apps/bookwarehouse.yaml
kubectl apply -f https://raw.githubusercontent.com/flomesh-io/fsm-docs/main/manifests/apps/mysql.yaml

Check Tracing

Check the Jaeger UI and you will get the tracing data there.

jaeger_pod="$(kubectl get pod -l app=jaeger -o jsonpath='{.items[0].metadata.name}')"                                                                 default ⎈
kubectl port-forward $jaeger_pod 16686:16686 &

Feedback

Was this page helpful?


Last modified April 28, 2024: add new doc for traffic mirroring (#12) (206fabc)