Versions Compared

Version Old Version 137 New Version 138
Changes made by

Fiona Sargeant (Unlicensed)

Fiona Sargeant (Unlicensed)

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

Insert excerpt
_Banners
_Banners
nameanalysis
nopaneltrue

Overview

Streams

tables are a key model component. They represent a structured store of data within PhixFlow receiving data from one or more components, processing it then storing it.

Insert excerpt
_standard_settings
_standard_settings
nopaneltrue

The tab toolbar has the following additional buttons: 

Insert excerpt
_record_sets_show
_record_sets_show
nopaneltrue
, 
Insert excerpt
_view_show
_view_show
nopaneltrue
, 
Insert excerpt
_run_analysis
_run_analysis
nopaneltrue
.

When you make changes to

streams

tables and

stream

attributes, PhixFlow automatically publishes them

to the stream data tables in the

to the database; see Publishing Streams.

Stream

Insert excerpt
_property_toolbar
_property_toolbar
nopaneltrue

Table Properties

Basic Settings

FieldDescription
NameThe name of the
stream
table.
Model Validation Status
Insert excerpt
_validate
_validate
namestatus
nopaneltrue
Enabled
Insert excerpt
_check_box_tick
_check_box_tick
nopaneltrue
 to include the steam during analysis runs.
Static Data

Insert excerpt
_check_box_tick
_check_box_tick
nopaneltrue
 to set the

stream

table to hold static data. Static data is reference or "look up" data that is used as part of a

stream

table calculation.

Updating a static
stream
table

In general, static

streams

tables are not updated during an analysis run that is initiated on

another stream

another table in the model. The exception is when an incoming push pipe is triggered, as this will force

the stream

the table to update itself.

Instead, to update the data in a static

stream

table you must run analysis directly on the

stream

table via:

  • the model
  • a task plan.
Period

Select the time period for the

stream

table:

Excerpt
  • Transactional: allows multiple users to run independent analysis tasks at the same time.
  • Non-transactional
    • Daily: generate or collect data every day.
    • Monthly: generate or collect data every month.
    • Variable: generate or collect data since the more recent run of the
stream
    • table to the current date.

As

transactional streams

transactional tables need a UID attribute, PhixFlow automatically creates this

stream

attribute if it does not already exist.

Default Data Range

This option affects the following items for which this is the backing

stream

table:

stream
  • views
dashboards
  • screens
  • card
layout
  • components.

Insert excerpt
_data_range
_data_range
nopaneltrue

If no option is is selected, PhixFlow assumes the default data range is All.

You can override this setting using the options Use Custom Data Range and Data Range available:

component

You cannot override this setting for a

stream

view.

Stream
Table Type

Select

the stream type

the table type; see Types of Table. The type indicates the function that operates on the data.

  • Aggregate
Stream
  • Table
  • Calculate
Stream
  • Table
  • CalculateBySet
Stream
  • Table
  • Cartesian
Stream
  • Table
  • Merge
Stream 
  • Table
Supersede Items on Pipe

Select a loop pipe from the list of those available. 

A loop pipe links the

stream

table back to itself. It compares new data records to the existing data records. If there is a repeated record PhixFlow uses the new one in the

stream

table and marks  the old one as superseded.

Audit Manual Changes

This check box is available when the Period is Transactional.

Insert excerpt
_check_box_tick
_check_box_tick
nopaneltrue
 to update and delete initiated by

stream

actions (not those carried out by analysis runs) will automatically mark the existing record as superseded and create a new

streamset

recordset. The new versions of the updated records will be placed in the new

streamset

recordset. Inserts will simply create a new

streamset

recordset, and add the inserted record into that

streamset

recordset.

When Audit Manual Changes is first set, the attributes UpdateAction, UpdatedByName, UpdatedByID and UpdatedTime will be created if they do not already exist. If you do not require these attributes, delete them. For the UpdatedByName attribute, PhixFlow creates a field of 250 characters. (In versions up to 8.0.4, the field was 50 characters long.)

UpdateAction must be set to the type of action, such as INSERT, UPDATE or DELETE. The other attributes will be populated if they exist on the

stream

table:

  • UpdatedByName - the name of the user that performed the update, 
  • UpdatedByID - the internal id of the user that performed the update 
  • UpdatedTime - the date and time the update was made

Attributes

The grid contains a list of the

stream

attributes in the

stream

table. This section has a toolbar with standard buttons and 

Insert excerpt
_table_show
_table_show
nopaneltrue
 and 
Insert excerpt
_file_collector_show
_file_collector_show
nopaneltrue
.

The grid lists the attributes (data columns) in the

stream

table data. To edit the properties of an attribute, double-click the attribute name to open the Attribute properties.

To remove an attribute from the

stream

table, select the attribute and click 

Insert excerpt
_delete
_delete
nopaneltrue
 in the toolbar.

To add a new attribute to the list, click  

Insert excerpt
_new
_new
nopaneltrue
 to open a new Attribute properties tab.

To add an existing attribute to the list:

  1. Click 
    Insert excerpt
    _table_show
    _table_show
    nopaneltrue
     to list the
streams
  1. tables available in the repository.
  2. Expand a
stream
  1. table to show its attributes.
  2. Drag an attribute into this list to add it. 

To edit only the expression:

  1. Right-click an attribute name to display the context menu.
  2. Select Edit the expression field.
  3. PhixFlow opens a simple text editor box
  4. Make changes to the attributes expression.
  5. Click Image Modified to save your changes.

Anchor
multipliers
multipliers
Multipliers and Filters

FieldDescription
Input MultiplierThe input multiplier expression should evaluate to a list of one or more values. For each value in the list, the internal variable _inputMultiplier will be set to that value and the whole
stream
table processing will be repeated i.e. the pull pipes will be read and the data from those pipes processed to generate output
stream items
records to be added to the current
streamset
recordset. For example :

do ( $aRange = [], addElement($aRange, rng.RangeFrom), addElement($aRange, rng.RangeTo), $bRange = [], addElement($bRange, $aRange), $bRange )

Where rng.RangeFrom = 500 and rng.RangeTo = 1000, the above example evaluates to [[500,1000]], which is a list containing 1 element, which is itself a list containing 2 elements. An
Input Multiplier
input multiplier that evaluates to [3,4,7,8] would run the
Stream
Table 4 times.
Because Input Multipliers are evaluated first in the PhixFlow Timing Cycle they are often used to look up values that can be passed to Database or other Collectors.
Log Input Multiplier Messages

Insert excerpt
_check_box_tick
_check_box_tick
nopaneltrue
 to cause PhixFlow to write a message to the log every time an Input Multiplier value changes during an analysis run.

Output MultiplierThis field is an expression which should evaluate to an array of values. A separate output record will be produced for each value in the array and this value is available as _outputMultiplier in each of the
stream
table output attribute expressions (each value in the array is also available through _type, although this is not recommended usage). In effect this will multiply each of the output records by the number of elements from the returned list.
For example :

ifNull(in.ASSET, [1,10,12] , // else do [5,7] )

will create 3 records for every record in the
stream
table if in.ASSET contains a value (setting _type = 1, 10 and 12 in each case). Otherwise it will create 2 records for every record in the
stream
table (and set _type = 5 and 7). An
Output Multiplier
output multiplier may also evaluate to a record, or a group of records. For example an Output Multiplier with the expression:

do( lookup(lkin, $num = in.BNumber), lkin )

will return a list of records which match the lookup on the lkin pipe. In this case the required data can be extracted from the Output Multiplier using the following expression :

do ( $values = _type, $values.account_num )

If the output multiplier expression evaluates to _NULL, an empty list of values or an empty list of records then a single output record will be produced with _type set to _NULL, _NULL or an empty record respectively.

Select Records to Output

You can select a subset of records to output to the next modelling object. You select records using the data in any attribute that is in the

stream

table output. Enter an expression that should evaluate to true or false (equivalently 1 or 0).  

When the expression evaluates to true or 1, the records found by the expression are selected and output to the next modelling object.

For example:

  • A
stream
  • table containing data about accounts has an attribute called "active".
  • You only want to include "active" accounts in the following
streams
  • tables. 
  • Active accounts have the value 1.

To select active accounts, enter the expression  _out.active.
All records that have the active attribute set to any other value are ignored.

Actions

This section has a toolbar with standard buttons. The grid contains a list of actions that operate on the

stream

table.

To edit the properties of an action, double-click its name. To add an action to the list, click  

Insert excerpt
_new
_new
nopaneltrue
. PhixFlow opens the Action properties tab.

To remove an action from the

stream

table, select the action and click 

Insert excerpt
_delete
_delete
nopaneltrue
 in the toolbar.

Stream Item

Record-Actions

This section has a toolbar with standard buttons. The grid contains a list of

stream item

record-actions that can update records in the

stream

table. 

To edit the properties of a

stream item

record-action, double-click its name. To add a

stream item

record-action to the list, click  

Insert excerpt
_new
_new
nopaneltrue
. PhixFlow opens the Record-Action properties tab.

To remove a

stream item

record-action from the

stream

table, select the

stream item

record-action and click 

Insert excerpt
_delete
_delete
nopaneltrue
 in the toolbar.

Views

This section has a toolbar with standard buttons. The grid contains a list of the data views for this

stream

table. 

To edit the properties of an

stream

view, double-click its name. To add a

stream

view to the list, click  

Insert excerpt
_new
_new
nopaneltrue
. PhixFlow opens the View properties tab.

To remove a view from the

stream

table, select the view and click 

Insert excerpt
_delete
_delete
nopaneltrue
 in the toolbar.

Sort Orders

This section has a toolbar with standard buttons. The grid contains a list of the sort orders for this

stream

table. 

To edit the properties of a sort order, double-click its name. To add a sort order to the list, click  

Insert excerpt
_new
_new
nopaneltrue
. PhixFlow opens the Sort Order properties tab.

To remove a sort order from the

stream

table, select the sort order and click 

Insert excerpt
_delete
_delete
nopaneltrue
 in the toolbar.

Errors 
Anchor
error
error

Insert excerpt
_validate
_validate
nameerror
nopaneltrue

Relations

This section has a toolbar with standard buttons. The grid displays a read-only list of the relationships that this

stream

table has. Double-click a relationship to display it's properties; see Relationship.

Filters

This section has a toolbar with standard buttons. The grid contains a list of the filters that can be applied to this

stream and can be added to this stream's views

table. You can add filters to view based on this table. 

All filters in this list are available in the default view for this

stream

table. For

stream

views that you create, in the View properties tab → Filters section, you can add filters. The list of available filters is populated by the filters defined for the

stream

table. 

To edit the properties of a filter, double-click its name. To add a filter to the list, click  

Insert excerpt
_new
_new
nopaneltrue
. PhixFlow opens the Filter properties tab.

To remove a filter from the

stream

table, select the filter and click 

Insert excerpt
_delete
_delete
nopaneltrue
 in the toolbar.

Tip

In a

stream

view grid, the header row has a drop-down list of filters. The drop-down list is populated by the filters that are selected on the

stream

view. The filters must first be defined for the

stream

table here.

This section has a toolbar with 

Insert excerpt
_detailed_list
_detailed_list
nopaneltrue
 and 
Insert excerpt
_refresh
_refresh
nopaneltrue
 buttons; see standard buttons. In the grid, PhixFlow lists all pipes that connect into the

stream

table. You cannot edit this grid. 

Note

This list can include pipes that have no input. This occurs if the source

stream

table has been deleted or if a model has been moved to a different PhixFlow instance (export/import), leaving behind a referenced

stream

table. Any pipes with no input are highlighted in yellow. To resolve pipes with no input you can:

  • recreate the
missing stream
  • missing table 
  • import the
missing stream
  • missing table 
  • keep the connection, if it will be restored when the model is moved to a different PhixFlow instance
  • delete the pipe, if it is no longer required.

Download Limits 
Anchor
download_limits
download_limits

The following properties affect the number of data records a user can download from a

stream

view, such as a grid, chart or card; see Download CSV or Excel Data.

FieldDescription
Stream
Table Download LimitEnter the maximum number of records that a user can download from a
stream view or card view on this stream
view on this table. If no value is set, PhixFlow uses the default limit set in System Configuration→ System Tuning → Default Download File Limit.
Allow Unlimited Downloads

Insert excerpt
_check_box_untick
_check_box_untick
nopaneltrue
 to restrict all users to the 

Stream

Table Download Limit.

Insert excerpt
_check_box_tick
_check_box_tick
nopaneltrue
to display a User Groups section where you can specify user groups whose members can download unlimited data from a

stream

view

or card view

on this

stream

table. For example, members of the Administrators user group may need to download all data.

User Groups

Available when Allow Unlimited Downloads is ticked.

Use this section to select the user groups whose members can download an unlimited number of records.

This section has a toolbar with standard buttons and 

Insert excerpt
_user_group
_user_group
nopaneltrue
.

The grid contains a list of user groups that are exempt from the

Stream

Table Download Limit. To add a user group to the list:

  1. Click
    Insert excerpt
    _user_group
    _user_group
    nopaneltrue
     to list all the user groups in the repository.
  2. Drag user groups into this list to add them to the role. 

To remove user groups, use the

Insert excerpt
_delete
_delete
nopaneltrue
 in the toolbar.


Live Search
spaceKey@self
additionalnone
placeholderSearch all help pages
typepage

Panel
borderColor#00374F
titleColorwhite
titleBGColor#00374F
borderStylesolid
titleSections on this page

Table of Contents
maxLevel3
indent12px
stylenone

   

Learn More


Insert excerpt
_terms_changing
_terms_changing
nopaneltrue


Data Retention Settings
Anchor
data-retain
data-retain

Use the following options to specify how PhixFlow manages old data when a streamtable-data-delete task runs.

Warning

We recommend that you always specify the data retention values for every stream table you create.
PhixFlow checks that a stream table has:

  • either data retention settings (recommended)
  • or Retain All Data selected.

If no data retention option is set, PhixFlow displays an error in the Errors section, above.

FieldDescriptionWhat Data is Deleted

Retain All Data

Insert excerpt
_check_box_untick
_check_box_untick
nopaneltrue
 to specify the criteria for keeping data in the Keep... properties below  (recommended).

 

Insert excerpt
_check_box_tick
_check_box_tick
nopaneltrue
 to allow old data to accumulate in PhixFlow.

Warning

Streams Tables with Retain All Data ticked never have their data deleted and the data will accumulate. This can lead to performance issues.


Keep for N Days

Anchor
keepForXDays
keepForXDays
Enter the number of days to keep data in the streamtable.

The age of data in a streamset recordset is its 'to' date, relative to the 'to' date of the newest valid streamset recordset in the streamtable .

When a streamtable-data-delete task runs for a stream table and:

  • Keep for X Days only is set, PhixFlow deletes stream table data that is older than N days.
  • Keep for Y StreamsetsRecordsets only is set, PhixFlow keeps the most recent X streamsets  recordsets and deletes streamsets recordsets that exceed X.
  • both Keep for X Days and Keep for Y StreamsetsRecordsets are set, PhixFlow deletes the stream table data that meets both conditions.
  • neither Keep for N Days nor Keep for X Streamsets  Recordsets are set, PhixFlow keeps stream table data indefinitely. This can lead to performance issues.
Keep for X Streamsets Recordsets

The number of streamsets recordsets to keep in the streamtable.


Keep Superseded for N Days

Anchor
keepSupersededForXDays
keepSupersededForXDays
Available when Track Superseded Data is selected.

Enter the number of days to keep superseded records in the stream table data. 

Tip

Remember to keep superseded records for as long as you want the option to roll back the data.

For example, when Keep Superseded for N Days is set to 4, you can run roll back on data that was superseded up to 4 days ago.

A streamtable-data-delete task removes superseded records older than 4 days, so PhixFlow no longer has the data to be able to roll it back.


In a stream a table where the superseded date is tracked, the stream the table data contains a mixture of active records and records that have been superseded.

When a streamtable-data-delete task runs for a stream a table and:

  • Keep Superseded for N Days only is set, PhixFlow deletes data records that were superseded more than N days ago.
  • Keep Superseded for X Streamsets Recordsets only is set, PhixFlow  deletes superseded records from the streamsets recordsets that exceed X.
  • both Keep Superseded for N Days and Keep Superseded for X StreamsetsRecordsets are set, PhixFlow deletes the superseded records that meet both conditions.
  • neither Keep Superseded for N Days nor Keep Superseded for X Streamsets Recordsets are set, PhixFlow keeps superseded records indefinitely. This can lead to performance issues.
Keep Superseded for X Streamsets Tablesets

Available when Track Superseded Data is selected.

Enter the number of recent streamsets recordsets that will retain their superseded data records.


Apply Delete Filter

: 

Insert excerpt
_check_box_tick
_check_box_tick
nopaneltrue
 to open a filter tab where you can set filter conditions; see Filter. PhixFlow applies this filter when it runs a streamtable-data-delete task. Only the records which match the filter are deleted.




Store

Stream

Views

This section has a toolbar with standard buttons. In the grid PhixFlow displays a list of the stream views to store for this streamtable. 

Analysis Models

A list of the analysis models that this stream this table appears on.

Advanced 

Only change the advanced properties with advice by PhixFlow support.

FieldDescription
Advanced PropertiesOnly use this field with advice from PhixFlow support. 
Database View

Use this property if your database administrator wants to query stream query table data in the PhixFlow database.

Enter a string in the format v_xxxx. When the stream the table is published to the database, PhixFlow creates a  stream view. If the stream the table is deleted, the next system task that runs will clear the associated stream view from the database.

By default, this field is blank and no stream view is created in the database.

See also Configuring Access to Database Views.

Storage Options
Index Scheme 
Anchor
index_scheme
index_scheme
Select an option to determine how indexes on the stream the table are organised. 
  • All: indexes on the stream the table are optimised for selecting from all streamsets recordsets (non-historied reads).
  • Latest: indexes on the stream the table are optimised for selecting from the latest streamset recordset (i.e. for historied reads).
  • Superseded: indexes on the stream the table are optimised for self-updating streams tables which have a mostly superseded records.
  • None: no indexes are created on the streamtable.
  • Partitioned: this index scheme is specific to partitioned streamstables; see Storage Type below. Partitioned streams tables must use this index scheme, and only partitioned streams tables can use it.
Storage TypeSelect an option to specify how data for the stream the table should be stored:
  • Database: Store the data in a regular table within the PhixFlow database. This is the most common option
  • Database (Partitioned): Store the data in a partitioned table within the PhixFlow database. This option provides improved performance for rollback and retaining very large streamsetsrecordsets. The option is only available if "partitioning" is available within your database installation.
  • In Memory: Data for the Stream table will not be written to the database. This option can be used (for example) when you want to aggregate large amounts of unsorted data which can then be written to a stored Streamtable.
Data Generation Options 
Anchor
data-gen
data-gen
Start DateEnter the date on which this stream this table starts or click
Insert excerpt
_calendar
_calendar
nopaneltrue
 to select a date. Data will be populated into the stream the table from this date onwards.
Primary Key Generator

This option relates to an actionflow with a save node that inserts a new stream-item/ record into a stream/ table. Each new record requires a unique value for the record's primary key. Use this option to specify how PhixFlow generates the primary key value.

  • Internal: PhixFlow generates unique primary key values automatically. This is equivalent to setting the stream attribute expression to _NULL. Check the stream attr expression doc 
  • Sequence: display the Primary Key Generator Sequence option, in which you can select a sequence of values.
Primary Key Generator Sequence

Available when Primary Key Generator is Sequence.

Select a sequence to assign a new value to the primary key attribute of a new stream-item/ record; see Sequence.

The new

stream item

record takes the next primary key value in a sequence. This is equivalent to setting the

stream

attribute expression to nextValue("sequenceName".

  • Sequence
    • In Memory Cache SizeEnter the size of the cache that will be maintained when you are using an in-memory streamtable. See Manage Unsorted Data Using an In-Memory Stream.
    Allow Partial Set Processing

    Insert excerpt
    _check_box_untick
    _check_box_untick
    nopaneltrue
     to restrict all users to the Stream Download Table Download Limit.

    Insert excerpt
    _check_box_tick
    _check_box_tick
    nopaneltrue
     so that, when analysis reaches the end of a buffer block, it submits the candidate set for processing, even if the next buffer block has a different key.

    Prevent Parallel Processing

    Available when Period is Transactional.

    Insert excerpt
    _check_box_tick
    _check_box_tick
    nopaneltrue
     to ensure only a single streamset recordset can be generated at a time even if the stream table receives several concurrent requests to generate data.

    This can be useful where you want to make sure that two analysis runs do not attempt to update the same records at the same time e.g. as a result of two people selecting the same records in a view and then hitting the same action button at the same time to process those records.

    Run Alone

    Insert excerpt
    _check_box_tick
    _check_box_tick
    nopaneltrue
     so that, whenever the analysis engine needs to generate data for this streamtable, it will first wait for all running tasks to complete before it starts.

    Any additional analysis tasks submitted while this stream this table is waiting to start, or while it is generating data, will wait until this stream this table has completed its analysis before they start.

    Key TolerancesWhen building a candidate set, data can be grouped together using a specific key value, e.g. Account Number. If the key value is a number, enter a key tolerance to identify numbers within the given tolerance as the same key value.
    Default ViewSelect the default view for the streamtable; see  Views for details of creating views on streamstables.
    Last Run Date (Read only)PhixFlow displays the date and time that analysis was last run for this streamtable. This date is taken from the "to date" of the most recent streamset recordset for this streamtable.
    Last Run ByPhixFlow displays the user that last ran analysis on this streamtable.


    Access Permissions

    By default, the permissions to access stream access table data is set in System Configuration →  Allow access to data by default.

    You can add specific user groups to restrict access to the streamtable; see Common Properties → Access Permissions.

    Insert excerpt
    _audit
    _audit
    nopaneltrue