Datasource


This page is for data modellers who want to connect to an external database, either to load data or to export processed data back to it.

Overview

A datasource is a modelling object whose properties provide the information needed to connect to an external database. You must have one datasource for each type of database to which you need to connect.  In an analysis model, you can add:

The datasources you create appear in the Basic Settings → Datasource list in collector and exporter properties.

Each datasource must have at least one datasource instance. You can add several instances if you want to configure different settings. For example you may want to connect to the same database using a different username and password for different scheduled tasks. 

In the properties for collectors and exporters, you can optionally specify the datasource instance using the Advanced → Datasource Instance Expression.

To add a new datasource to an analysis model:

  1. Go to the model's toolbar → Create group.
  2. Click  Database to expand the menu.
  3. Drag a  Datasource onto the analysis model.

To add an existing datasource to an analysis model, in the model diagram toolbar:

  1. Go to the model toolbar → List group.
  2. Click  Database to expand the menu.
  3. Click  Datasource to open the list of available datasources.
  4. Drag a datasource into the analysis model.

Property Pane Toolbar

For information about the toolbar options, see the Common Properties page, Toolbars and Controls section.

Datasource  Properties Tab

Parent Details

If this item is within or belongs to another, its parent name is shown here. See the Parent Details section on the Common Properties page for more details.

Basic Settings

In addition to the standard icons, the properties toolbar includes  Show Tables, which opens a list of tables visible for this datasource using the default login details; see the Table Details window.

For a datasource, you must:

  • Enter a Name.
  • Select a Database Driver.
  • Add at least one datasource instance.
  • Tick the Enabled check box when it is ready to use.

We recommend you tick Protect during import to prevent this datasource being updated or deleted during import.

FieldDescription
Name

Enter the name for this datasource. It is helpful to use a name that reflects the type of database to which it will connect. This name appears in the Datasource  drop-down list for database collectors and exporters.

Enabled

 Tick when the configuration is complete and the datasource is ready to be used.

Protect during import

 Untick to allow this datasource to be updated or deleted by import from another PhixFlow instance.

 Tick to retain these settings if this datasource is imported from another PhixFlow instance.

 More ...

When you have several instances of PhixFlow, for example development and production instances, you may want some item properties to be different for each instance. Usually, when you export from a source instance, then import to a target instance, PhixFlow overwrites item properties with those from the source instance. In the target instance, tick this box to prevent the current setting being overwritten by the source item.

Exception: If the datasource parent and the datasource have been deleted in the source instance, then the import deletes the datasource from the target instance. This prevents orphaned datasources persisting in PhixFlow.

Database Driver

Select the database driver for the type of database to which the datasource connects. 

The drop-down list may contain database drivers that are deprecated and do not have the necessary jar files; see Database Driver

When a database driver is selected, to open its properties, click the blue chevron  next to the Database Driver heading.

Datasource Instances 

This section has a toolbar with standard buttons.

The grid contains a list of datasource instances that belong to this datasource. Double-click on a row to open the datasource instance properties. 

To add a datasource to the list, click  Create New . PhixFlow opens the properties for a new datasources instance; see Datasource Instance Properties, below.

To remove a datasource instance from the grid, use the toolbar button  Delete.

Database Collectors

This section has a toolbar with standard buttons.

The grid contains a list of database collectors that use this datasource. Double-click on a row to open the database collector properties. You can change whether or not the collector uses this datasource or whether the collector is enabled.

Any changes you make will affect the analysis models in which the collector is used.

Rate Limiting

This feature is available in PhixFlow version 11.1+.

Rate limiting can be used to limit traffic and help stability. This works by restricting the amount of requests that can be sent.

Field Description
Enable rate limitingToggle on or off to restrict the number of requests.
Requests gained per time period

Available when Enable rate limiting is enabled. 

Number of requests that can be sent in the Time period for gaining new requests.

Time period for gaining new requests (ms)

Available when Enable rate limiting is enabled. 

The time period, in milliseconds, for sending requests.

Maximum rollover requests (Optional)

Available when Enable rate limiting is enabled. 

The maximum number of requests that can be saved, to be sent later.

Defaults to to the requests gained per time period.

Maximum parallel requests (Optional)

Available when Enable rate limiting is enabled. 

The maximum number of requests that can be sent at the same time.

Note: If more than the allowed number of requests are sent, as soon as a request is finished, another is sent.

Rate Limiting Example

Requests gained per time period: 20

Time period for gaining new requests: 30000ms

Maximum rollover requests: 25

If 10 requests are sent, then 30000ms later another 30 requests are sent, 25 requests would be sent straight away (up to the Maximum rollover request), and the other 5 requests would be sent 30000ms later. 

Advanced

FieldDescription
Internal IDRead-only. PhixFlow displays the identifier for this datasource. The PhixFlow support team may ask for this identifier during troubleshooting.

Analysis Models

This section has a toolbar with standard buttons.

The grid contains a list of analysis models that use this datasource. Double-click on a row to open the analysis model properties. 

Description

We recommend that you always enter a description to explain the purpose of this item.

 Audit Tab

Audit Summary

See the Common Properties page, Audit Summary section.

Datasource Instance  Properties Tab

Parent Details

If this item is within or belongs to another, its parent name is shown here. See the Parent Details section on the Common Properties page for more details.

Basic Settings

For a datasource instance, you must:

  • Enter a Name.
  • Enter the URL for the database.
  • Specify the Username and Password.
  • Tick the Enabled check box when it is ready to use.

Other settings are optional.

FieldDescription

Name

Enter the name for this instance. In the properties for collectors and exporters, you can specify this instance by using its name in the Datasource Instance Expression.

Enabled

 Tick when the configuration is complete and the datasource is ready to be used.

URL

Enter the URL for the target database.

Example URLs

  • Oracle
    jdbc:oracle:thin:@hostname:1521:phixflow
  • Oracle for versions above 12c, with PDB containers 
    jdbc:oracle:thin:@hostname:1521/phixflow
  • MySQL 
    jdbc:mysql://hostname/phixflow
  • Big Query
    jdbc:bigquery://@hostname:@port;ProjectId=@Project;#
  • SQL Server 
    jdbc:sqlserver://hostname\myservice;databaseName=phixflow;encrypt=false
Authorisation Type

Select:

  • Local: to provide user credentials from PhixFlow. When you select this option, the Username and Password properties are available.
  • External: to provide user credentials from the secure keystore. When you select this option, the Username Key and Password Key properties are available.
    Use this option for greater security.
UsernameEnter a username.

These options are available when Authorisation Type is Local.

This username and password must match an account that is configured to access the datasource instance. PhixFlow connects to the datasource using this username and password.

PasswordEnter a password.
Username Key

Enter the key that identifies the username in the keystore.

These options are available when Authorisation Type is External.

The keys must match a username and password stored in the keystore and these are configured to access the datasource instance. PhixFlow connects to the datasource using this username and password.


Your system administrator adds usernames, passwords and their aliases (keys) to the keystore. They will notify you of the key to use. See Adding Data to a Keystore.

Password KeyEnter the key that identifies the password in the keystore.

Advanced

Only use these options if you know about database connections and SQL queries. For help with these options, contact support@phixflow.com.

FieldDescription

Check Query

Use this field when connections to the target database might go stale. This can happen if the target database disconnects long-running connections.

Enter a database query that will return a row from the database.

PhixFlow runs the query against the database just before each use of a pooled connection. If the query does not return a row, the connection is discarded and a new connection is created. To avoid any performance impact, the query should execute quickly. 

See the Query String section of the Database Collector page for information about writing SQL queries in PhixFlow.

Datasource Profile

Use this field when the target database has additional requirements not covered by the datasource and datasource instance properties.

Enter the path to an external datasource connection profile.

PhixFlow will use the configuration information in the profile to connect to the target database. 

DescriptionOptionally enter a information about the advanced configuration.

Remember to click  Apply and Close to save and close the  datasource instance properties and return to the datasource properties tab.

 Audit Tab

Audit Summary

See the Common Properties page, Audit Summary section.


Sections on this page

Learn More

For links to all pages in this topic, see Analysis Models for Batch Processing Data.


Terminology changes in progress

As part of the redesign of PhixFlow, we are changing the following terms:

dashboard → screen   
stream → table
stream attributes → attributes
stream item → record
stream set → recordset
stream view → view
stream item action → record-action 
stream action → table-action
driver class → database driver