Helm Chart for Symphony
This applies to: Visual Data Discovery
The Helm chart provided in the Symphony Helm Chart repository is a lightweight way to configure and run Symphony in Kubernetes. It includes sub-charts for use with 3rd-party components such PostgreSQL, Consul, OpenTelemetery Collector, and more.
This Helm chart is publicly available in the Symphony Helm Chart repository, and includes, by default, a 14 day trial Symphony license. Optional components are disabled by default.
This topic covers:
- Prerequisites
- Working With Helm
- Configuring the Chart
Prerequisites
The following prerequisites are required for a successful deployment using Kubernetes and the Helm chart:
A Kubernetes cluster (versions 1.23-1.27)
Install Kubernetes or have access to a cluster.
A local copy of
kubectlconfigured to work with your cluster.
Helm (3.8-3.11). See Installing Helm.
Access to Docker Hub.
This Helm chart was developed with portability in mind. It relies only on Vanilla Kubernetes features and doesn’t use any (cloud) vendor-specific features. Testing is performed in an Amazon EKS cluster and Minikube: you should need to make little to no modifications while working with other flavors of Kubernetes such as Azure AKS or Google GKE.
Required Resources
Symphony is built using the microservices architecture. Since the list of required microservices (e.g. connectors) depends on your usage scenarios, it’s difficult to estimate the exact resource requirements. However, for a deployment to be useful it must include an instance of Composer Web, a Query Engine instance, a Connector instance, and a Consul service for discovery. The usage of autoscaling will obviously impact the resource requirements.
Resource estimates for some common setups:
| Minimal, one replica setup | Standard one replica setup (default) | Advanced, tree replica setup |
|---|---|---|
Mandatory services: Web Server, Query Engine, Consul. Optional services: none. Connectors: one arbitrary connector. PostgreSQL metadata store location: external. Footprint:
| Mandatory services: Web Server, Query Engine, Consul. Optional services: Data Writer. Connectors: PostgreSQL. PostgreSQL metadata store location: internal. Footprint:
| Mandatory services: Web Server, Query Engine, Consul. Optional: Data Writer, Screenshot Service. Connectors: two arbitrary connectors. PostgreSQL metadata store location: external. Footprint:
|
For more exact estimates, you'll need to first determine the set of required components, then sum up their resource limits specified in the chart.
Working With Helm
This section explains the generic usage of Helm and Helm chart lifecycle applied to the Symphony Helm Chart. For a generic introduction to Helm, see the Helm Quickstart Guide.
Obtain the Chart
Add the chart repository:
helm repo add composer https://composer-repo.logianalytics.com/helm-charts/stable
List the charts you can install:
helm repo update composerhelm search repo composer
Install with the Default Config
Run the following command to install the chart with the default config:
helm install <release-name> composer/composer
by running the first command the chart was released. Remember the release name, you will need it to interact with it later. Take a look at the output, it suggests the next steps like retrieving the address for ingress.
helm install --replace.
To keep track of a release's state, or to re-read the next steps, you can use the helm status:
helm status <release-name>
Default Config Caveats
Our default Ingress resource and Ingress Controller’s configuration might not work for you. Follow this guide to re-configure our Ingress.
Customize the Chart Before Installing
Run the following command to list all the configurable parameters of the Symphony chart and their default values:
helm show values composer/composer
Review the list of parameters to decide if anything needs to be changed. Use the guidance provided in Deciding on the Configuration to evaluate the parameters to help you decide if anything has to be changed.
Once you have identified the necessary changes, follow this generic guide on applying the changes to your Helm release.
Applying changes to your Helm release:
Create the
values.yamlfile and add YAML sections that you want to override to it. For example, in the following snippet, the PostgreSQL connector logging level is increased toDEBUG.edc: postgresql: properties: logging.level.com.zoomdata: DEBUGInstall a new release with the override values.
helm upgrade <release-name> composer/composer -f values.yaml
Alternatively, upgrade an existing release.
Upgrade a Release and Recover on Failure
To apply a new configuration to an existing release use helm upgrade:
helm upgrade <release-name> composer/composer -f values.yaml
All affected pods will be automatically restarted.
helm upgrade. If you have such an incompatibility, re-install the chart.
If something went wrong during the upgrade, review the revision history of your release:
helm history <release-name>
Identify the revision you want to rollback to, then run:
helm rollback <release-name> <revision-number>
Uninstall a Release
If a particular release is no longer required, use this command to uninstall:
helm uninstall <release-name> --keep-history
Using --keep-history is optional. If you use it, you can better audit the cluster's history, and even undelete a release using helm rollback.
Persistent Resources Considerations
Running helm uninstall on a release will remove most of the Kubernetes resources from the cluster. Some resources that constitute Composer metadata and re-usable configs are preserved. This allows you to create another Symphony release with the same metadata like Sources, Visuals, Dashboards, etc. The preserved resources are PersistentVolumeClaims (PVC) and corresponding PersistenVolumes (PV) for:
Composer PostgreSQL metadata database folder. The PVC will have a name such as
data-<release-name>-postgresql-0and the capacity of 8Gi. It will beBoundto a PV. These resources will be present only if the release was configured to create an internal PostgreSQL instance (the default setting). If you don't intend to reuse the metadata, you can delete this PVC and PV withkubectl delete. See PostgreSQL Metadata Store Configuration for information about reusing this metadata database in a new release.The volume with connectors' drivers. The PVC will have the name
composer-shared-volume. It will beBoundto the PV with the same name only if you configured Driver Drop-In for your connectors. If you did not configure this, it will be in thePendingstate. If you don't intend to install a new release into the current namespace, you can delete the PVC. Delete thecomposer-shared-volumePV only if you don’t intend to install new Composer charts into this cluster.The volume with the Consul service metadata. The PVC will have a name such as
data-<namespace>-<release-name>-consul-server-0and the capacity of 10Gi. It will beBoundto a PV. Composer can re-create the service discovery information stored in Consul upon a new release, you can safely delete these resources.
Deep Chart Inspection and Customization
Helm pull provides a way to get all source files of the chart. This is useful when you want to perform a deep inspection of the chart. Use the following commands to download the chart from the repository and unpack it in a local directory:
Once you have the source code of the chart, you’ll be able to modify it.
Once you are done with the required modifications, you’ll be able to either repackage the chart into your custom chart or install it from the local directory:
helm install <release-name> . -f values.yaml
Debugging the Chart
Perform a dry run before installing a release:
helm install <release-name> composer/composer -f values.yaml --debug --dry-run
Adding the --debug option causes the server to render your templates. If rendering goes fine, then the resulting manifest file is returned. If not, an error with a stack trace is returned. Attach this error output when reaching out to support.
Helm also allows you to see what templates are installed on the server for a particular (successful) release:
helm get manifest <release-name>
Configuring the Chart
The Symphony platform has lots of configuration options and some optional components. While the out-of-the-box configuration of the Helm chart provides some meaningful defaults, most likely you'll need to customize some important aspects like the list of required connectors or the need for horizontal autoscaling.
The Default Configuration
The default configuration installs a release with the following components:
Enabled Symphony services:
- Mandatory: Web Server, Query Engine, Consul.
- Connectors: PostgreSQL.
- Optional: Data Writer.
- PostgreSQL metadata store location: internal.
Advanced capabilities:
- Default Ingress rule:
ENABLED - Tracing infrastructure:
DISABLED - Horizontal autoscaling:
DISABLED
See the next section to learn how to customize it.
Deciding on the Configuration
This list covers the main decisions and recommended actions that you’ll need to perform to determine your configuration:
Do you want to use an external, managed PostgreSQL instance as the Symphony metadata store (the recommended setup)? Alternatively, you can let the chart install an internal PostgreSQL instance (the default).
Change the default PostgreSQL passwords if using an internal instance.
What is the list of data stores that you need to connect to? By default, only PostgreSQL connector is installed. Customize the list of connectors according to your needs.
Do you want to enable horizontal autoscaling?
Are you going to schedule Dashboard Reports that will deliver you dashboard screenshots periodically? If yes, enable the Screenshot Service component.
Are you going to upload flat files (e.g. CSV, JSON) for further analysis? If not, disable the Data Writer component to save cluster resources.
Do you have special requirements for the Ingress configuration? If yes, reconfigure our default Ingress.
Do you have a license that you want to apply to this deployment? If yes, inject the license during the chart installation.
How do you want to integrate Composer into your observability infrastructure?
Once you decide on your target configuration, prepare corresponding override values for the Helm chart and put them into your values.yaml. Override values fall into two categories:
Symphony application configuration properties injected into services running inside pods/containers:
Regular application properties
Credentials
Configuration for Kubernetes resources governed by Helm.
The next section covers Symphony application configuration.
Injecting Symphony Configuration Properties
Application Properties
There are two categories of Symphony application configuration properties that differ in their sensitivity, hence the way they are specified in the values.yaml:
Regular application configuration properties that don’t contain sensitive data and are not exposed as separate Helm chart parameters.
Sensitive application configuration properties, such as database credentials, that are exposed as Helm chart parameters.
Regular Application Properties
Regular properties for a Symphony service are specified as the properties map within the object representing this service in the values.yaml file. Each key-value pair in this map represents a property name and value.
For example, the following snippet shows a number of regular properties with names starting with mail. that specify mail server configuration for Zoomdata Web component:
Regular properties go to ConfigMaps when Symphony is installed in a Kubernetes cluster and are treated as if they are specified in regular properties files. The names of the properties should have the same names as in regular properties files (usually found under /etc/zoomdata or /opt/zoomdata/conf for Linux-based deployments).
Sensitive Application Properties
Each sensitive property for a Symphony service is specified as a separate parameter in the object representing this service in the values.yaml l file. For example, parameters zoomdataWeb.mailLogin and zoomdataWeb.mailPassword below are sensitive properties that specify, correspondingly, login and password for the mail server configured for Zoomdata Web component:
It is not possible to override sensitive configuration properties with regular ones. For example, in the snippet below, regular properties mail.loginand mail.password will be ignored by the Zoomdata Web component:
Sensitive properties go to Secrets when Symphony is installed in a Kubernetes cluster. They are usually injected into corresponding services through environment variables.
List of Properties Available as Helm Parameters
| Parameter in Values File | Description | Injected As | Corresponding Application Property |
|---|---|---|---|
| Zoomdata Web | |||
zoomdataWeb.contextPath | Web context path | Env variable | server.servlet.context-path |
zoomdataWeb.metadataDbUrl | Metadata database URL | Env variable | spring.datasource.url |
zoomdataWeb.metadataDbUsername | Metadata database username | Secret | spring.datasource.username |
zoomdataWeb.metadataDbPassword | Metadata database password | Secret | spring.datasource.password |
zoomdataWeb.uploadDbUrl | Upload database URL | Env variable | upload.destination.params.jdbc_url |
zoomdataWeb.uploadDbUsername | Upload database username | Secret | upload.destination.params.user_name |
zoomdataWeb.uploadDbPassword | Upload database password | Secret | upload.destination.params.password |
zoomdataWeb.keysetDbUrl | Keyset database URL | Env variable | keyset.destination.params.jdbc_url |
zoomdataWeb.keysetDbUsername | Keyset database username | Secret | keyset.destination.params.user_name |
zoomdataWeb.keysetDbPassword | Keyset database password | Secret | keyset.destination.params.password |
zoomdataWeb.userAuditingDbUrl | User auditing database URL | Env variable | user-auditing.destination.params.jdbc_url |
zoomdataWeb.userAuditingDbUsername | User auditing database username | Secret | user-auditing.destination.params.user_name |
zoomdataWeb.userAuditingDbPassword | User auditing database password | Secret | user-auditing.destination.params.password |
zoomdataWeb.mailLogin | Mail server login | Secret | mail.login |
zoomdataWeb.mailPassword | Mail server password | Secret | mail.password |
zoomdataWeb.adminPassword | Password for the built-in | Secret | admin.password |
zoomdataWeb.supervisorPassword | Password for the built-in | Secret | supervisor.password |
| Query Engine | |||
queryEngine.dbEnabled | Use Query Engine database for storing query results cache when true | ||
queryEngine.dbUrl | Query Engine database URL | Env variable | spring.qe.datasource.jdbcUrl |
queryEngine.dbUsername | Query Engine database username | Secret | spring.qe.datasource.username |
queryEngine.dbPassword | Query Engine database password | Secret | spring.qe.datasource.password |
JVM Properties
There are two categories of properties available for all services:
- Properties to configure Symphony services heap size.
- A catch-all property that allows passing arbitrary Java options to each service.
Heap Size Configuration
The following Helm chart parameters are used to control heap size for Symphony Java services:
Property in Values File | Description | Scope | Java Option |
|---|---|---|---|
heapSizeMin | Initial JVM heap size. | Service | -Xms |
heapSizeMax | Maximum JVM heap size. | Service | -Xmx |
The following snippet shows the default heap size configuration for connectors:
The following snippet shows how to override the default settings for the PostgreSQL connector:
Passing Arbitrary Java Options to Services
A catch-all property for passing arbitrary Java options is called additionalJavaOpts and is supported for each Composer service. For example, this is how to enable garbage collector logging for Query Engine:
queryEngine: additionalJavaOpts: "-verbose:gc"
Injecting Credentials
Some services might need additional credentials provided in separate files, like Java trust stores and Kerberos keytab files. To inject such credentials, you need to use additional Kerberos Secrets and Volumes.
For example, let’s consider how to inject a Java trust store into the Elasticsearch 8 connector:
Create a trust store Secret:
kubectl create secret generic <secret-name> --from-file=truststore.p12
Configure the Elasticsearch 8 connector to mount this Secret as a volume and access the trust store from its file system:
edc: elasticsearch-8.0: enabled: true additionalJavaOpts: "-Djavax.net.ssl.trustStore=/opt/zoomdata/security/truststore.p12 -Djavax.net.ssl.truststoreType=PKCS12 -Djavax.net.ssl.trustStorePassword=<truststore-password>" extraVolumeMounts: - name: truststore mountPath: /opt/zoomdata/security/truststore.p12 subPath: truststore.p12 readOnly: true extraVolumes: - name: truststore secret: secretName: <secret-name>
Comments
0 comments
Please sign in to leave a comment.