Versions Compared

Version Old Version 76 New Version 77
Changes made by

Fiona Sargeant

Fiona Sargeant

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

Streams are a key model component. They represent a structured store of data within PhixFlow receiving data from one or more components, processing it then storing it. Insert excerpt_standard_settings_standard_settingsnopaneltrueThe tab toolbar has the following additional buttons:  Insert excerpt_stream_sets_show_stream_sets_shownopaneltrue,  Insert excerpt_stream_views_stream_viewsnopaneltrue,  Insert excerpt_run_analysis_run_analysisnopaneltrue.

Basic Settings

...

Updating a static stream

In general, static streams are not updated during an analysis run that is initiated on another stream in the model. The exception is when an incoming push pipe is triggered, as this will force the stream to update itself.

Instead, to update the data in a static stream you must run analysis directly on the stream via:

  • the model
  • a task plan.

...

  • Transactional: allows multiple users to run independent analysis tasks at the same time.
  • Daily: generate or collect data every day.
  • Monthly: generate or collect data every month.
  • Variable: generate or collect data since the more recent run of the stream to the current date.

As transactional streams need a UID attribute, PhixFlow automatically creates this stream attribute if it does not already exist.

...

Select the stream type. The type indicates the function that operates on the data.

  • Aggregate Stream
  • Calculate Stream
  • CalculateBySet Stream
  • Cartesian Stream
  • Merge Stream 

...

Select a loop pipe from the list of those available. 

A loop pipe links the stream back to itself. It compares new data records to the existing data records. If there is a repeated record PhixFlow uses the new one in the stream and marks  the old one as superseded.

...

This check box is available when the Period is Transactional.

...

When Audit Manual Changes is first set, the attributes UpdateAction, UpdatedByName, UpdatedByID and UpdatedTime will be created if they do not already exist. If you do not require these attributes, delete them. For the UpdatedByName attribute, PhixFlow creates a field of 250 characters. (In versions up to 8.0.4, the field was 50 characters long.)

UpdateAction must be set to the type of action, such as INSERT, UPDATE or DELETE. The other attributes will be populated if they exist on the stream:

  • UpdatedByName - the name of the user that performed the update, 
  • UpdatedByID - the internal id of the user that performed the update 
  • UpdatedTime - the date and time the update was made

Attributes

...

The grid lists the attributes (data columns) in the stream data. To edit the properties of an attribute, double-click the attribute name to open the Stream Attributes properties.

...

To add an existing attribute to the list:

...

To edit only the expression:

  1. Right-click an attribute name to display the context menu.
  2. Select Edit the expression field.
  3. PhixFlow opens a simple text editor box
  4. Make changes to the attributes expression.
  5. Click Image Removed to save your changes.

...

do ( $aRange = [], addElement($aRange, rng.RangeFrom), addElement($aRange, rng.RangeTo), $bRange = [], addElement($bRange, $aRange), $bRange )

...

ifNull(in.ASSET, [1,10,12] , // else do [5,7] )

...

do( lookup(lkin, $num = in.BNumber), lkin )

will return a list of records which match the lookup on the lkin pipe. In this case the required data can be extracted from the Output Multiplier using the following expression :

do ( $values = _type, $values.account_num )

If the output multiplier expression evaluates to _NULL, an empty list of values or an empty list of records then a single output record will be produced with _type set to _NULL, _NULL or an empty record respectively.

...

Select Records to Output

...

You can select a subset of records to output to the next modelling object. You select records using the data in any attribute that is in the stream output. Enter an expression that should evaluate to true or false (equivalently 1 or 0).  

When the expression evaluates to true or 1, the records found by the expression are selected and output to the next modelling object.

For example:

  • A stream containing data about accounts has an attribute called "active".
  • You only want to include "active" accounts in the following streams. 
  • Active accounts have the value 1.

To select active accounts, enter the expression  _out.active.
All records that have the active attribute set to any other value are ignored.

Actions

The grid contains a list of actions that use the stream. This section has a toolbar with standard buttons.

...

Stream Item Actions

The grid contains a list of stream item actions that can update the stream. This section has a toolbar with standard buttons.

...

Views

The grid contains a list of the data views for this stream. This section has a toolbar with standard buttons.

o edit the properties of an stream view, double-click the attribute name to open its Stream View properties tab.

...

First pass draft done to here...

Sort Orders

A list of the sort orders on the stream. See Stream View Order.

Filters

A list of the filters on the stream. See Filters on Data Views.

Any filter defined on the stream may appear in the dropdown list of filters accessible from the header of each stream view. To make a filter available in a view, the filter must be added to the list of filters for that view. See Stream View for details.

All filters defined on this tab will be available on the system generated Default View for this Stream.

A list of pipes into the stream.

...

It is possible for this list to include pipes that have no input. This occurs if the source stream has been deleted, or if a model has been moved to a different PhixFlow instance (export/import), leaving behind a referenced stream. Any pipes with no input are highlighted in yellow. To resolve pipes with no input you can:

...

Streams are a key model component. They represent a structured store of data within PhixFlow receiving data from one or more components, processing it then storing it.

Insert excerpt
_standard_settings
_standard_settings
nopaneltrue

The tab toolbar has the following additional buttons: 

Insert excerpt
_stream_sets_show
_stream_sets_show
nopaneltrue
, 
Insert excerpt
_stream_views
_stream_views
nopaneltrue
, 
Insert excerpt
_run_analysis
_run_analysis
nopaneltrue
.

Basic Settings

FieldDescription
NameThe name of the stream.
Enabled
Insert excerpt
_check_box_ticked
_check_box_ticked
nopaneltrue
 to include the steam during analysis runs.
Static Data

Insert excerpt
_check_box_ticked
_check_box_ticked
nopaneltrue
 to set the stream to hold static data. Static data is reference or "look up" data that is used as part of a stream calculation.

Updating a static stream

In general, static streams are not updated during an analysis run that is initiated on another stream in the model. The exception is when an incoming push pipe is triggered, as this will force the stream to update itself.

Instead, to update the data in a static stream you must run analysis directly on the stream via:

  • the model
  • a task plan.
PeriodSelect the time period for the stream. This can be either a regular period, or variable.
  • Transactional: allows multiple users to run independent analysis tasks at the same time.
  • Daily: generate or collect data every day.
  • Monthly: generate or collect data every month.
  • Variable: generate or collect data since the more recent run of the stream to the current date.

As transactional streams need a UID attribute, PhixFlow automatically creates this stream attribute if it does not already exist.

Stream Type

Select the stream type. The type indicates the function that operates on the data.

  • Aggregate Stream
  • Calculate Stream
  • CalculateBySet Stream
  • Cartesian Stream
  • Merge Stream 
Supersede Items on Pipe

Select a loop pipe from the list of those available. 

A loop pipe links the stream back to itself. It compares new data records to the existing data records. If there is a repeated record PhixFlow uses the new one in the stream and marks  the old one as superseded.

Audit Manual Changes

This check box is available when the Period is Transactional.

Insert excerpt
_check_box_ticked
_check_box_ticked
nopaneltrue
 to update and delete initiated by stream actions (not those carried out by analysis runs) will automatically mark the existing record as superseded and create a new stream set. The new versions of the updated records will be placed in the new stream set. Inserts will simply create a new stream set, and add the inserted record into that stream set.

When Audit Manual Changes is first set, the attributes UpdateAction, UpdatedByName, UpdatedByID and UpdatedTime will be created if they do not already exist. If you do not require these attributes, delete them. For the UpdatedByName attribute, PhixFlow creates a field of 250 characters. (In versions up to 8.0.4, the field was 50 characters long.)

UpdateAction must be set to the type of action, such as INSERT, UPDATE or DELETE. The other attributes will be populated if they exist on the stream:

  • UpdatedByName - the name of the user that performed the update, 
  • UpdatedByID - the internal id of the user that performed the update 
  • UpdatedTime - the date and time the update was made


Attributes

The grid contains a list of the stream attributes in the stream. This section has a toolbar with standard buttons and 

Insert excerpt
_stream_show
_stream_show
nopaneltrue
 and 
Insert excerpt
_file_collector_show
_file_collector_show
nopaneltrue
.

The grid lists the attributes (data columns) in the stream data. To edit the properties of an attribute, double-click the attribute name to open the Stream Attributes properties.

To remove an attribute from the stream, select the attribute and click 

Insert excerpt
_delete
_delete
nopaneltrue
 in the toolbar.

To add a new attribute to the list, click  

Insert excerpt
_add
_add
nopaneltrue
 to open a new Stream Attributes properties tab.

To add an existing attribute to the list:

  1. Click 
    Insert excerpt
    _stream_show
    _stream_show
    nopaneltrue
     to list the streams available in the repository.
  2. Expand a stream to show its attributes.
  3. Drag an attribute into this list to add it. 

To edit only the expression:

  1. Right-click an attribute name to display the context menu.
  2. Select Edit the expression field.
  3. PhixFlow opens a simple text editor box
  4. Make changes to the attributes expression.
  5. Click Image Added to save your changes.

Anchor
multipliers
multipliers
Multipliers and Filters

FieldDescription
Input MultiplierThe input multiplier expression should evaluate to a list of one or more values. For each value in the list, the internal variable _inputMultiplier will be set to that value and the whole stream processing will be repeated i.e. the pull pipes will be read and the data from those pipes processed to generate output stream items to be added to the current stream set. For example :

do ( $aRange = [], addElement($aRange, rng.RangeFrom), addElement($aRange, rng.RangeTo), $bRange = [], addElement($bRange, $aRange), $bRange )

Where rng.RangeFrom = 500 and rng.RangeTo = 1000, the above example evaluates to [[500,1000]], which is a list containing 1 element, which is itself a list containing 2 elements. An Input Multiplier that evaluates to [3,4,7,8] would run the Stream 4 times.
Because Input Multipliers are evaluated first in the PhixFlow Timing Cycle they are often used to look up values that can be passed to Database or other Collectors.
Output MultiplierThis field is an expression which should evaluate to an array of values. A separate output record will be produced for each value in the array and this value is available as _outputMultiplier in each of the stream output attribute expressions (each value in the array is also available through _type, although this is not recommended usage). In effect this will multiply each of the output records by the number of elements from the returned list.
For example :

ifNull(in.ASSET, [1,10,12] , // else do [5,7] )

will create 3 records for every record in the stream if in.ASSET contains a value (setting _type = 1, 10 and 12 in each case). Otherwise it will create 2 records for every record in the stream (and set _type = 5 and 7). An Output Multiplier may also evaluate to a record, or a group of records. For example an Output Multiplier with the expression:

do( lookup(lkin, $num = in.BNumber), lkin )

will return a list of records which match the lookup on the lkin pipe. In this case the required data can be extracted from the Output Multiplier using the following expression :

do ( $values = _type, $values.account_num )

If the output multiplier expression evaluates to _NULL, an empty list of values or an empty list of records then a single output record will be produced with _type set to _NULL, _NULL or an empty record respectively.

Select Records to Output

You can select a subset of records to output to the next modelling object. You select records using the data in any attribute that is in the stream output. Enter an expression that should evaluate to true or false (equivalently 1 or 0).  

When the expression evaluates to true or 1, the records found by the expression are selected and output to the next modelling object.

For example:

  • A stream containing data about accounts has an attribute called "active".
  • You only want to include "active" accounts in the following streams. 
  • Active accounts have the value 1.

To select active accounts, enter the expression  _out.active.
All records that have the active attribute set to any other value are ignored.


Actions

The grid contains a list of actions that use the stream. This section has a toolbar with standard buttons.

To edit the properties of an action, double-click the attribute name. To add an action to the list, click  

Insert excerpt
_add
_add
nopaneltrue
. PhixFlow opens a new Action properties tab.

To remove an action from the stream, select the action and click 

Insert excerpt
_delete
_delete
nopaneltrue
 in the toolbar.

Stream Item Actions

The grid contains a list of stream item actions that can update the stream. This section has a toolbar with standard buttons.

To edit the properties of an action, double-click the attribute name. To add an action to the list, click  

Insert excerpt
_add
_add
nopaneltrue
. PhixFlow opens a new to open the Stream Item Actions properties tab.

To remove an action from the stream, select the action and click 

Insert excerpt
_delete
_delete
nopaneltrue
 in the toolbar.

Views

The grid contains a list of the data views for this stream. This section has a toolbar with standard buttons.

o edit the properties of an stream view, double-click the attribute name to open its Stream View properties tab.

To add an stream view to the list, click  

Insert excerpt
_add
_add
nopaneltrue
 to open a new its Stream View properties tab.

To remove a view from the stream, select the view and click 

Insert excerpt
_delete
_delete
nopaneltrue
 in the toolbar.


...

First pass draft done to here...

Sort Orders

A list of the sort orders on the stream. See Stream View Order.

Filters

A list of the filters on the stream. See Filters on Data Views.

Any filter defined on the stream may appear in the dropdown list of filters accessible from the header of each stream view. To make a filter available in a view, the filter must be added to the list of filters for that view. See Stream View for details.

All filters defined on this tab will be available on the system generated Default View for this Stream.

A list of pipes into the stream.

Note

It is possible for this list to include pipes that have no input. This occurs if the source stream has been deleted, or if a model has been moved to a different PhixFlow instance (export/import), leaving behind a referenced stream. Any pipes with no input are highlighted in yellow. To resolve pipes with no input you can:

  • recreate the missing stream
  • import the missing stream
  • keep the connection, if it will be restored when the model is moved to a different PhixFlow instance
  • delete the pipe, if it is no longer required.

Download Limits

FieldDescription
Stream Download LimitThe maximum number of records that a user can download from a stream view or card view on this stream. If no value is set, PhixFlow uses the default limit set in /wiki/spaces/HELP81/pages/1570899052→ System Tuning → Default Stream View Download Limit.
Allow Unlimited Downloads

Insert excerpt
_check_box_untick
_check_box_untick
nopaneltrue
 to restrict all users to the Stream Download Limit.

Insert excerpt
_check_box_ticked
_check_box_ticked
nopaneltrue
so display a User Groups section. Specify user groups whose members can download unlimited data from a stream view or card view on this stream. For example, members of the Administrators user group may need to download all data.

User Groups

Available when Allow Unlimited Downloads is ticked.

Use this section to select the user groups whose members can download an unlimited number of records.

This section has a toolbar with standard buttons and 

Insert excerpt
_user_group
_user_group
nopaneltrue
.

The grid contains a list of user groups that are exempt from the Stream Download Limit. To add a user group to the list:

  1. Click
    Insert excerpt
    _user_group
    _user_group
    nopaneltrue
     to list all the user groups in the repository.
  2. Drag user groups into this list to add them to the role. 

To remove user groups, use the

Insert excerpt
_delete
_delete
nopaneltrue
 in the toolbar.

Action Flows

To do

Anchor
archive
archive
Archive Settings

...

Store Stream Views

To do

Access Permissions

By default, the permissions to access stream data is set in System Configuration →  Allow access to data by default. You can add specific user groups to restrict access to the stream; see Common Properties → Access Permissions.

...

FieldDescription
Advanced PropertiesOnly use this field with advice from PhixFlow support. 
Database View

Use this option if your database administrator wants to query stream data in the PhixFlow database. Enter a string in the format v_xxxx. When the stream is published to the database, PhixFlow creates a  stream view. 

By default, this field is blank and no stream view is created in the database.

If the stream is deleted, the next system task that runs will clear the associated stream view from the database.

Storage Options
Index Scheme 
Anchor
index_scheme
index_scheme
This determines how indexes on the Stream are organised. 
  • All: indexes on the Stream are optimised for selecting from all stream sets (non-historied reads).
  • Latest: indexes on the Stream are optimised for selecting from the latest stream set (i.e. for historied reads).
  • Superseded: indexes on the Stream are optimised for self-updating streams which have a mostly superseded records.
  • None: no indexes are created on the Stream.
Storage TypeSpecifies how data for the Stream should be stored:
  • Database: Store the data in a regular table within the PhixFlow database. This is the most common option
  • Database (Partitioned): Store the data in a partitioned table within the PhixFlow database. This option provides improved performance for rollback and archiving of very large Streamsets. The option is only available if "partitioning" is available within your database installation.
  • In Memory: Data for the Stream will not be written to the database. This option can be used (for example) when you want to aggregate large amounts of unsorted data which can then be written to a stored Stream.
Data Generation Options
Start DateThe date that this stream starts. Data will be populated into the stream from this date onwards.
In Memory Cache SizeThe size of the cache that will be maintained when you are using an in-memory stream. See How To: Summarise unsorted data with In Memory Streams.
Allow Partial Set ProcessingIf ticked, when analysis reaches the end of a buffer block it submits the candidate set for processing, even if the next buffer block has a different key.
Prevent Parallel Processing

This field only appears if the Period is set to Transactional. If ticked, it ensures that only a single stream set can be generated at a time even if the stream receives several concurrent requests to generate data.

This can be useful where you want to make sure that two analysis runs don't attempt to update the same records at the same time e.g. as a result of two people selecting the same records in a view and then hitting the same action button at the same time to process those records.

Run Alone

If this flag is ticked then whenever the analysis engine needs to generate data for this stream it will first wait for all running tasks to complete before it starts.

Any additional analysis tasks submitted while this stream is waiting to start, or while it is generating data, will wait until this stream has completed its analysis before they start.

Key TolerancesWhen building a candidate set, data can be grouped together using a specific key value, e.g. Account Number. If the key value is a number, setting a key tolerance will identify numbers within the given tolerance as the the same key value.
Write in Single TransactionIf this is ticked, all of the records for a Stream Set will be written to the database in a single transaction.
Maximum Records to WriteThe maximum number of records per Stream Set if Write in Single Transaction is ticked.
Default ViewThe default view selected for the stream. See help on Views for details of creating views on streams.
Last Run Date (Read only)The date and time that analysis was last run for this stream. This date is taken from the "to date" of the most recent stream set for this stream.
Last Run ByThe user that last ran this stream.

...