Not receiving traces sent by DAPR + Opentelmetry Xray exporter setup

Question

To collect traces from DAPR we are using Opentelemetry collector and using Zipkin receiver since that's supported by DAPR and using awsxray exporter in open telemetry configuration. We see the traces generated in Opentelemetry log, however we do not see traces collected in AWS Xray. Please let us know what could be the issue

Open telemetry configuration:
    receivers:
      zipkin:
        endpoint: 0.0.0.0:9411
      otlp:
        protocols:
          grpc:
            endpoint: 0.0.0.0:4317
          http:
            endpoint: 0.0.0.0:4318
      awsxray:
      awsxray/udp_endpoint:
          endpoint: "0.0.0.0:5678"
          transport: udp
      awsxray/proxy_server:
          proxy_server:
            endpoint: "0.0.0.0:1234"
            tls:
              insecure: true
            region: ap-south-1

Exporter configuration:

awsxray:
        region: ap-south-1
        index_all_attributes: true
        resource_arn: "arn:aws:iam::215977048474:role/EKS-NodeGroup"
        #role_arn: "arn:aws:iam::215977048474:role/MastBazaar-EKS"
        no_verify_ssl: true

Service configuration:

service:
      extensions: [pprof, zpages, health_check]
      pipelines:
        traces:
          receivers: [zipkin]
          # List your exporter here.
          exporters: [awsxray,logging]

Also added environment variable OTEL_PROPAGATORS: "xray" in deployment as seen below

containers:
      - name: otel-collector
        image: otel/opentelemetry-collector-contrib-dev:latest
        env:
        - name: OTEL_PROPAGATORS
          value: "xray"

Sample log in open telemetry

2022-09-13T18:15:46.106Z        INFO    loggingexporter/logging_exporter.go:42  TracesExporter  {"#spans": 1}
2022-09-13T18:15:46.106Z        DEBUG   loggingexporter/logging_exporter.go:51  ResourceSpans #0
Resource SchemaURL:
Resource labels:
     -> service.name: STRING(multiplyapp)
ScopeSpans #0
ScopeSpans SchemaURL:
InstrumentationScope
Span #0
    Trace ID       : 5db6a8eb0ade84a251ca547034b13c5f
    Parent ID      : 9987294085d81f24
    ID             : f88d24f02301bf28
    Name           : CallLocal/multiplyapp/multiply
    Kind           : SPAN_KIND_SERVER
    Start time     : 2022-09-13 18:15:44.031751 +0000 UTC
    End time       : 2022-09-13 18:15:45.034974 +0000 UTC
    Status code    : STATUS_CODE_UNSET
    Status message :
Attributes:
     -> dapr.api: STRING(/dapr.proto.internals.v1.ServiceInvocation/CallLocal)
     -> dapr.invoke_method: STRING(multiply)
     -> dapr.protocol: STRING(grpc)
     -> rpc.service: STRING(ServiceInvocation)
     -> net.host.ip: STRING(10.0.195.115)

2022-09-13T18:15:54.505Z        INFO    loggingexporter/logging_exporter.go:42  TracesExporter  {"#spans": 1}
2022-09-13T18:15:54.505Z        DEBUG   loggingexporter/logging_exporter.go:51  ResourceSpans #0
Resource SchemaURL:
Resource labels:
     -> service.name: STRING(multiplyapp)
ScopeSpans #0
ScopeSpans SchemaURL:
InstrumentationScope
Span #0
    Trace ID       : c3f98c14589d4618899f4581503e4c3f
    Parent ID      : 642dd665bbc6c3dd
    ID             : 014fec0dfb862b5c
    Name           : CallLocal/multiplyapp/multiply
    Kind           : SPAN_KIND_SERVER
    Start time     : 2022-09-13 18:15:53.489635 +0000 UTC
    End time       : 2022-09-13 18:15:54.492579 +0000 UTC
    Status code    : STATUS_CODE_UNSET
    Status message :
Attributes:
     -> dapr.api: STRING(/dapr.proto.internals.v1.ServiceInvocation/CallLocal)
     -> dapr.invoke_method: STRING(multiply)
     -> dapr.protocol: STRING(grpc)
     -> rpc.service: STRING(ServiceInvocation)
     -> net.host.ip: STRING(10.0.195.115)

Any guidance will be very much appreciated

Answer

Hi, 
Please check whether you have aws credentials configured on OTel collector host? For example, by environment variable https://docs.aws.amazon.com/sdk-for-php/v3/developer-guide/guide_credentials_environment.html

And because XRay does not accept random 128-bits trace id, please ensure configure `xray id generator` in root service.

The env variable `OTEL_PROPAGATORS=xray` is for OTel SDKs. OTel collector does not need it.

Not receiving traces sent by DAPR + Opentelmetry Xray exporter setup

相关内容