(v23) Python Data Generator

Python Data Generator

This applies to: Managed Dashboards, Managed Reports

The Python Data Generator transform lets you generate data by writing scripts using the Python programming language. This is similar to the Python Analysis transform except that it does not accept input from a preceding transform and generates its own output directly from Python.

You can use the Python Data Generator transform to provide data to be used or visualized in Symphony. For example, Python can connect to and manipulate REST API data into a usable format, or generate data for prototyping or developing proof-of-concept dashboards.

To learn more about the Python language, see python.org.

Note: If your organization did not install Symphony, Python transforms may not be available depending on how your instance is hosted.

Setup

The Python programming environment must be installed on the Symphony server for you to use the Python Data Generator transform.

This is normally installed for you automatically as a prerequisite. See Install Python for more details, and for examples of how to install additional Python packages to use in your transforms.

Input

The Python Data Generator transform does not have any inputs. It generates output by running Python scripts.

Add the transform

When creating a new data cube, you can add the Python Data Generator transform to an empty canvas from the toolbar.

You can also create a new data cube from the left menu, and choose Python Data Generator in the dialog of options that appears.

The Python Data Generation transform is added to the data cube and connected to a Process Result transform automatically.

You can also add the Python Data Generator transform from the toolbar to an existing data cube process, and use a Union transform or another transform that combines data from multiple inputs.

Configure the transform

Double-click the Python Data Generation transform, or select Configure from its right-click menu or the toolbar.

In the configuration dialog for the transform, the key task is to enter a Python Script that returns a result. Optionally click Edit script to use the Script Editor window for more space and to access other helpful features.

As a simple example, a script for generating a column of numbers from 1 to 5 looks like this:

return (1,2,3,4,5)

Note: You can use the Print command to print variables to the application logs for debugging. To view these entries, ensure that the Log Filter configuration setting for Data Cubes is set to Information.

Important: The result must be in a format that can be represented as a table, and data cubes will be unable to use Python outputs such as Print or Draw.

In this dialog, you can set up Placeholders to insert into the script that pass in parameter values similar to when using a manual select. Insert the Identifier you entered for the placeholder (beginning and ending with the $ symbol) into your script to access the current parameter value. When typing into the Python Script box shown in the image above, you can type the $ symbol or right-click for a popup list of the current placeholders, then use the keyboard or mouse to select and insert it.

Select the Use Pooling option to reuse Python processes between execution of Python scripts to improve performance compared to starting a new process each time. There are corresponding configuration settings for Symphony administrators in the Python Pool category to manage the number of processes and their memory consumption.

Output

The output of the Python Data Generator depends on the script it is configured with. It can be a single value, a column of values, or multiple columns.

In the case of the simple script for generating numbers from 1 to 5, you can see an output column named f0 in the Data Preview window.

Generate data from a JSON file

Another example Python script for generating data is by connecting to a JSON file. This essentially uses a Python Data Generator transform in a data cube as a JSON data connector.

Setup

This example relies on four packages to be installed in Python on the Symphony server. If needed, open a command prompt as an administrator if the server runs Windows, for example, and use commands like the following for the four packages:

• pip install numpy

• pip install pandas

• pip install jsonlib-python3

• pip install requests

Create the script

To generate the JSON data, configure the Python Data Generator transform and use script like the following, which uses json_normalize:

import json, requests from pandas.io.json import json_normalize url = "http://example.domain.com/data.json" resp = requests.get(url) data = json.loads(resp.text) return json_normalize(data)

This can create a table reflecting all of the data in the referenced JSON file.

Generate data from a REST call

This example relies on the same four packages in Python as the previous section. An administrator with access to the Symphony server may need to install these packages as described above.

This example will log onto Symphony using REST in order to get a session ID. Next, it calls the Symphony file system query API with that session ID to retrieve all the dashboards that exist in a specific project. Finally it logs off, and then returns the results.

Create the script

import json, requests from pandas.io.json import json_normalize host = "<<Insert host address>>" username = "<<Insert user name>>" password = "<<Insert password>>" projectId = "<<Insert project id>>" # authenticate with DBI response = requests.post(host + "/api/logon/", json={"accountName":username, "password":password}) sessionId = response.json()["sessionId"] requestBody = { "queryFileSystemEntriesOptions":{ "entryIds":[ projectId ], "pageNumber":0, "pageSize":0, "queryOptions":"RecursiveQuery", "orderBy":[ { "fileSystemQueryField":"CreatedTime", "sortDirection":"Descending" } ], "filter":[ { "field":"ObjectType", "operator":"Equals", "value":"Dashboard", "options":"None" } ] } } # query the file system api response = requests.post(host + "/api/file/query", json=requestBody, headers={"Authorization":"Bearer " + sessionId}) # logoff requests.delete(host + "/api/session/current", headers={"Authorization":"Bearer " + sessionId}) result = json.loads(response.text) # return the result return json_normalize(result)