(v24) Distributed Tracing for Symphony

Distributed Tracing for Symphony

This applies to: Visual Data Discovery

All Symphony microservices support distributed tracing using the OpenTelemetry (OTel) specifications. Integrate with OTel javaagent or any other OTel compliant agent. Install a tracing server of your choice, an OTel collector, and configure the microservices to trace Symphony front end and back end requests.

Tracing collects information about requests. Use it to associate logs of the other microservices with the collected information.

• Install a Tracing Server

• Install and Configure OTel Collector

• Configure the Collector

• Configure Symphony Microservice Tracing

Install a Tracing Server

Symphony integrates with any tracing server that supports OpenTelemetry (OTel) specifications and its export protocols (otlp, zipkin, jeager).

After you have installed a tracing server, install the otel collector.

Install and Configure OTel Collector

Install an OTel collector to act as a proxy between Symphony microservices and your tracing server, both to simplify configuration and improve security by passing information through the proxy. See https://opentelemetry.io/docs/collector/ and https://opentelemetry.io/docs/collector/getting-started/.

After installation, configure your collector or collectors to export tracing information to the tracing server.

Configure the Collector

Provide the appropriate configuration information for your collector. The configuration information consists of three main sections, receivers, processors, and exporters. See https://opentelemetry.io/docs/collector/configuration/.

receivers: otlp: protocols: grpc: processors: batch: exporters: otlp: endpoint: zoomdata-tempo:4317 tls: insecure: true service: pipelines: traces: receivers: [otlp] processors: [batch] exporters: [otlp]

• Receivers: Configure available protocols and endpoints for the receiver to receive tracing information from Symphony's microservices. insightsoftware recommends you use the OTlp protocol.

• Processor: Enable additional processing of tracing information. In this example, batch processing is enabled to optimize network communication.

• Exporters: Configure export information. Provide a destination URL and other connection parameters to communicate with the tracing server. insightsoftware recommends you use the OTlp protocol.

After you have installed and configured the OTel collector, install and configure a java agent and Symphony's microservices.

Configure Symphony Microservice Tracing

You must install and configure the OTel java agent to trace Symphony microservices. Optionally, next configure the Symphony front end for tracing. After these steps are complete and you've restarted the microservices, you can extract the traceID in a REST request or from a web socket outbound message.

• Install the Java Agent

• Configure the Collector

• Configure the Symphony Front End

• Extract the traceID

Important: insightsoftware recommends you use the OTel java agent.

Install the Java Agent

1. Download the java agent to the server running the Symphony microservice. https://github.com/open-telemetry/opentelemetry-java-instrumentation/releases/latest/download/opentelemetry-javaagent.jar

2. Define the path to the java agent for Symphony. Select the option that works best for your environment.

   1. Specify the path in SERVICE_NAME.jvm (such as zoomdata.jvm or edc-postgresql.jvm) in the /opt/zoomdata/conf/ folder, such as -javaagent:/path/to/java/agent/opentelemetry-javaagent.jar. See Symphony Data Discovery Microservice Name Reference.

   2. Define an environment variable, _JAVA_OPTS, that contains the path, such as _JAVA_OPTIONS="-javaagent:/path/to/java/agent/opentelemetry-javaagent.jar".

Note: Some application performance monitoring tools provide their own java agents that are built on top of the OpenTelemetry java agent, or is compliant with the OTLP protocol. Use at your own discretion; the behavor may be different from the OTel java agent.

Configure the Java Agent

After you've installed the java agent, specify backend configuration properties for the agent using required and optional configuration properties. Include in the SERVICE_NAME.jvm or use environment variables. See https://opentelemetry.io/docs/instrumentation/java/automatic/agent-config/ and https://opentelemetry.io/docs/reference/specification/sdk-environment-variables/.

Required Configuration Properties

Property	Recommended Value	Description
OTEL_TRACES_EXPORTER	otlp	Defines the protocol to use to export tracing information.
OTEL_METRICS_EXPORTER	none	Disable metrics export using opentelemetry.
OTEL_LOGS_EXPORTER	none	Disable logs export using opentelemetry.
OTEL_EXPORTER_OTLP_TRACES_ENDPOINT	http://OTEL_COLLECTOR_HOST:4317	Defines the URL for the otel collector or tracing server.
OTEL_SERVICE_NAME	zoomdata query-engine edc-postgresql edc-mssql screenshot-service data-writer	Define the names of the services.
OTEL_INSTRUMENTATION_SPRING_BOOT_ACTUATOR_AUTOCONFIGURE_ENABLED	false	Required, for now, to enable tracing. This may be updated after future otel releases.

Configure the Symphony Front End

After you've provided back end configuration properties for tracking, provide any needed variables for the Symphony front end as well.

Enable tracing and provide environment variables by specifying the fe.env configuration property for the zoomdata-web microservice using the following format: fe.env=ENV_VARIABLE1=value1;ENV_VARIABLE2=value2;ENV_VARIABLE3=value3.

The default value of OTEL_SDK_DISABLED is false in the OpenTelemetry SDK. When used in your Symphony environment, installation changes this value to true for front end support. To enable tracing, specify OTEL_SDK_DISABLED=false in the environment variable.

Required Configuration Properties

Property	Recommended Value	Description
OTEL_TRACES_EXPORTER	otlp	Defines the protocol to use to export tracing information.
OTEL_METRICS_EXPORTER	none	Disable metrics export using opentelemetry.
OTEL_LOGS_EXPORTER	none	Disable logs export using opentelemetry.
OTEL_EXPORTER_OTLP_TRACES_ENDPOINT	http://OTEL_COLLECTOR_HOST:4317	Defines the URL for the otel collector or tracing server.
OTEL_SERVICE_NAME	zoomdata query-engine edc-postgresql edc-mssql screenshot-service data-writer	Define the names of the services.
OTEL_INSTRUMENTATION_SPRING_BOOT_ACTUATOR_AUTOCONFIGURE_ENABLED	false	Required, for now, to enable tracing. This may be updated after future otel releases.

OpenTelemetry is executed in the user's browser and all tracing information is sent to the collector from the browser. The OTEL_EXPORTER_OTLP_TRACES_ENDPOINT must be reachable by the browser to execute tracing collection. Depending on your environment, you may need to configure cors.

After you have installed and configured the java agent, restart all microservices.

Extract the traceID

When tracing is configured and enabled, you can extract the traceID in a REST request or from a web socket outbound message. traceID is embedded in the traceparent property. The structure is traceparent is {version}-{traceId}-{spanId}-{sampleDecision}.