Overview

Streams are a key model component. They represent a structured store of data within PhixFlow receiving data from one or more components, processing it then storing it.

For information about the properties toolbar, and about the sections Parent Details, Analysis Models, Description and Audit Summary, see Common Properties. For a full list of all the PhixFlow property tabs and windows, see Property Tabs.

The tab toolbar has the following additional buttons: Show Stream Sets, Stream Views, Run Analysis.

When you make changes to streams and stream attributes, PhixFlow automatically publishes them to the stream data tables in the database; see Publishing Streams.

Stream Properties

Basic Settings

Field	Description
Name	The name of the stream.
Enabled	Tick to include the steam during analysis runs.
Static Data	Tick to set the stream to hold static data. Static data is reference or "look up" data that is used as part of a stream calculation. Updating a static stream In general, static streams are not updated during an analysis run that is initiated on another stream in the model. The exception is when an incoming push pipe is triggered, as this will force the stream to update itself. Instead, to update the data in a static stream you must run analysis directly on the stream via: the model a task plan.
Period	Select the time period for the stream. This can be either a regular period, or variable. Transactional: allows multiple users to run independent analysis tasks at the same time. Daily: generate or collect data every day. Monthly: generate or collect data every month. Variable: generate or collect data since the more recent run of the stream to the current date. As transactional streams need a `UID` attribute, PhixFlow automatically creates this stream attribute if it does not already exist.
Default Data Range	This option affects the following items for which this is the backing stream: stream views dashboards card layout components. Select which stream set data is displayed. Latest: displays the records from the latest stream set only. Recommended when stream sets are updated by batch process, and therefore have a Period of Daily or Monthly. All: displays the records from all stream sets. Recommended when stream sets are updated incrementally and therefore have a Period of Transactional or Variable. If no option is is selected, PhixFlow assumes the default data range is All. You can override this setting using the options Use Custom Data Range and Data Range available: on a Dashboard. on a card component; see Layout. You cannot override this setting for a stream view.
Stream Type	Select the stream type. The type indicates the function that operates on the data. Aggregate Stream Calculate Stream CalculateBySet Stream Cartesian Stream Merge Stream
Supersede Items on Pipe	Select a loop pipe from the list of those available. A loop pipe links the stream back to itself. It compares new data records to the existing data records. If there is a repeated record PhixFlow uses the new one in the stream and marks the old one as `superseded`.
Audit Manual Changes	This check box is available when the Period is Transactional. Tick to update and delete initiated by stream actions (not those carried out by analysis runs) will automatically mark the existing record as superseded and create a new stream set. The new versions of the updated records will be placed in the new stream set. Inserts will simply create a new stream set, and add the inserted record into that stream set. When Audit Manual Changes is first set, the attributes `UpdateAction`, `UpdatedByName`, `UpdatedByID` and `UpdatedTime` will be created if they do not already exist. If you do not require these attributes, delete them. For the `UpdatedByName` attribute, PhixFlow creates a field of 250 characters. (In versions up to 8.0.4, the field was 50 characters long.) `UpdateAction` must be set to the type of action, such as INSERT, UPDATE or DELETE. The other attributes will be populated if they exist on the stream: `UpdatedByName` - the name of the user that performed the update, `UpdatedByID` - the internal id of the user that performed the update `UpdatedTime` - the date and time the update was made

Attributes

The grid contains a list of the stream attributes in the stream. This section has a toolbar with standard buttons and Show Streams and Show File Collectors.

The grid lists the attributes (data columns) in the stream data. To edit the properties of an attribute, double-click the attribute name to open the Stream Attributes properties.

To remove an attribute from the stream, select the attribute and click Permanently Delete in the toolbar.

To add a new attribute to the list, click Add to open a new Stream Attributes properties tab.

To add an existing attribute to the list:

Click Show Streams to list the streams available in the repository.
Expand a stream to show its attributes.
Drag an attribute into this list to add it.

To edit only the expression:

Right-click an attribute name to display the context menu.
Select Edit the expression field.
PhixFlow opens a simple text editor box
Make changes to the attributes expression.
Click to save your changes.

Multipliers and Filters

Field	Description
Input Multiplier	The input multiplier expression should evaluate to a list of one or more values. For each value in the list, the internal variable _inputMultiplier will be set to that value and the whole stream processing will be repeated i.e. the pull pipes will be read and the data from those pipes processed to generate output stream items to be added to the current stream set. For example : do ( $aRange = [], addElement($aRange, rng.RangeFrom), addElement($aRange, rng.RangeTo), $bRange = [], addElement($bRange, $aRange), $bRange ) Where rng.RangeFrom = 500 and rng.RangeTo = 1000, the above example evaluates to [[500,1000]], which is a list containing 1 element, which is itself a list containing 2 elements. An Input Multiplier that evaluates to [3,4,7,8] would run the Stream 4 times. Because Input Multipliers are evaluated first in the PhixFlow Timing Cycle they are often used to look up values that can be passed to Database or other Collectors.
Log Input Multiplier Messages	Tick to cause PhixFlow to write a message to the log every time an Input Multiplier value changes during an analysis run.
Output Multiplier	This field is an expression which should evaluate to an array of values. A separate output record will be produced for each value in the array and this value is available as _outputMultiplier in each of the stream output attribute expressions (each value in the array is also available through _type, although this is not recommended usage). In effect this will multiply each of the output records by the number of elements from the returned list. For example : ifNull(in.ASSET, [1,10,12] , // else do [5,7] ) will create 3 records for every record in the stream if in.ASSET contains a value (setting _type = 1, 10 and 12 in each case). Otherwise it will create 2 records for every record in the stream (and set _type = 5 and 7). An Output Multiplier may also evaluate to a record, or a group of records. For example an Output Multiplier with the expression: do( lookup(lkin, $num = in.BNumber), lkin ) will return a list of records which match the lookup on the lkin pipe. In this case the required data can be extracted from the Output Multiplier using the following expression : do ( $values = _type, $values.account_num ) If the output multiplier expression evaluates to _NULL, an empty list of values or an empty list of records then a single output record will be produced with _type set to _NULL, _NULL or an empty record respectively.
Select Records to Output	You can select a subset of records to output to the next modelling object. You select records using the data in any attribute that is in the stream output. Enter an expression that should evaluate to true or false (equivalently 1 or 0). When the expression evaluates to true or 1, the records found by the expression are selected and output to the next modelling object. For example: A stream containing data about accounts has an attribute called "active". You only want to include "active" accounts in the following streams. Active accounts have the value 1. To select active accounts, enter the expression `_out.active`. All records that have the active attribute set to any other value are ignored.

Actions

This section has a toolbar with standard buttons. The grid contains a list of actions that operate on the stream.

To edit the properties of an action, double-click the attribute name. To add an action to the list, click Add. PhixFlow opens the Action properties tab.

To remove an action from the stream, select the action and click Permanently Delete in the toolbar.

Stream Item Actions

This section has a toolbar with standard buttons. The grid contains a list of stream item actions that can update records in the stream.

To edit the properties of a stream item action, double-click the attribute name. To add a stream item action to the list, click Add. PhixFlow opens the Stream Item Actions properties tab.

To remove a stream item action from the stream, select the stream item action and click Permanently Delete in the toolbar.

Views

This section has a toolbar with standard buttons. The grid contains a list of the data views for this stream.

To edit the properties of an stream view, double-click the attribute name. To add a stream view to the list, click Add. PhixFlow opens the Stream View properties tab.

To remove a view from the stream, select the view and click Permanently Delete in the toolbar.

Sort Orders

This section has a toolbar with standard buttons. The grid contains a list of the sort orders for this stream.

To edit the properties of a sort order, double-click its name. To add a sort order to the list, click Add. PhixFlow opens the Sort Order properties tab.

To remove a sort order from the stream, select the sort order and click Permanently Delete in the toolbar.

Filters

This section has a toolbar with standard buttons. The grid contains a list of the filters that can be applied to this stream and can be added to this stream's views.

All filters in this list are available in the default view for this stream. For stream views that you create, in the Stream View properties tab → Filters section, you can add filters. The list of available filters is populated by the filters defined for the stream.

To edit the properties of a filter, double-click its name. To add a filter to the list, click Add. PhixFlow opens the Filter properties tab.

To remove a filter from the stream, select the filter and click Permanently Delete in the toolbar.

In a stream view grid, the header row has a drop-down list of filters. The drop-down list is populated by the filters that are selected on the stream view. The filters must first be defined for the stream here.

Inputs

This section has a toolbar with Show detailed list and Refresh buttons; see standard buttons. In the grid, PhixFlow lists all pipes that connect into the stream. You cannot edit this grid.

This list can include pipes that have no input. This occurs if the source stream has been deleted or if a model has been moved to a different PhixFlow instance (export/import), leaving behind a referenced stream. Any pipes with no input are highlighted in yellow. To resolve pipes with no input you can:

recreate the missing stream
import the missing stream
keep the connection, if it will be restored when the model is moved to a different PhixFlow instance
delete the pipe, if it is no longer required.

Download Limits

The following properties affect the number of data records a user can download from a stream view, such as a grid, chart or card; see Download CSV or Excel Data.

Field	Description
Stream Download Limit	Enter the maximum number of records that a user can download from a stream view or card view on this stream. If no value is set, PhixFlow uses the default limit set in System Configuration→ System Tuning → Default Download File Limit.
Allow Unlimited Downloads	Untick to restrict all users to the Stream Download Limit. Tick to display a User Groups section where you can specify user groups whose members can download unlimited data from a stream view or card view on this stream. For example, members of the Administrators user group may need to download all data.
User Groups	Available when Allow Unlimited Downloads is ticked. Use this section to select the user groups whose members can download an unlimited number of records. This section has a toolbar with standard buttons and User Groups. The grid contains a list of user groups that are exempt from the Stream Download Limit. To add a user group to the list: Click User Groups to list all the user groups in the repository. Drag user groups into this list to add them to the role. To remove user groups, use the Permanently Delete in the toolbar.

Data Retention Settings

Always specify the data retention values for every stream you create.

Streams that have no data retention settings will never have their data deleted. Stream data will accumulate, leading to space and performance issues.

Store Stream Views

This section has a toolbar with standard buttons. In the grid PhixFlow displays a list of the stream views to store for this stream.

Access Permissions

By default, the permissions to access stream data is set in System Configuration → Allow access to data by default.

You can add specific user groups to restrict access to the stream; see Common Properties → Access Permissions.

Analysis Models

A list of the analysis models that this stream appears on.

Advanced

Only change the advanced properties with advice by PhixFlow support.

Field	Description
Advanced Properties	Only use this field with advice from PhixFlow support.
Database View	Use this property if your database administrator wants to query stream data in the PhixFlow database. Enter a string in the format `v_xxxx`. When the stream is published to the database, PhixFlow creates a stream view. If the stream is deleted, the next system task that runs will clear the associated stream view from the database. By default, this field is blank and no stream view is created in the database. See also Configuring Access to Database Views.
Storage Options
Index Scheme	Select an option to determine how indexes on the stream are organised. All: indexes on the stream are optimised for selecting from all stream sets (non-historied reads). Latest: indexes on the stream are optimised for selecting from the latest stream set (i.e. for historied reads). Superseded: indexes on the stream are optimised for self-updating streams which have a mostly superseded records. None: no indexes are created on the stream. Partitioned: this index scheme is specific to partitioned streams; see Storage Type below. Partitioned streams must use this index scheme, and only partitioned streams can use it.
Storage Type	Select an option to specify how data for the stream should be stored: Database: Store the data in a regular table within the PhixFlow database. This is the most common option Database (Partitioned): Store the data in a partitioned table within the PhixFlow database. This option provides improved performance for rollback and retaining very large stream sets. The option is only available if "partitioning" is available within your database installation. In Memory: Data for the Stream will not be written to the database. This option can be used (for example) when you want to aggregate large amounts of unsorted data which can then be written to a stored Stream.
Data Generation Options
Start Date	Enter the date on which this stream starts. Data will be populated into the stream from this date onwards.
In Memory Cache Size	Enter the size of the cache that will be maintained when you are using an in-memory stream. See Manage Unsorted Data Using an In-Memory Stream.
Allow Partial Set Processing	Untick to restrict all users to the Stream Download Limit. Tick so that, when analysis reaches the end of a buffer block, it submits the candidate set for processing, even if the next buffer block has a different key.
Prevent Parallel Processing	Available when Period is Transactional. Tick to ensure only a single stream set can be generated at a time even if the stream receives several concurrent requests to generate data. This can be useful where you want to make sure that two analysis runs do not attempt to update the same records at the same time e.g. as a result of two people selecting the same records in a view and then hitting the same action button at the same time to process those records.
Run Alone	Tick so that, whenever the analysis engine needs to generate data for this stream, it will first wait for all running tasks to complete before it starts. Any additional analysis tasks submitted while this stream is waiting to start, or while it is generating data, will wait until this stream has completed its analysis before they start.
Key Tolerances	When building a candidate set, data can be grouped together using a specific key value, e.g. Account Number. If the key value is a number, enter a key tolerance to identify numbers within the given tolerance as the same key value.
Default View	Select the default view for the stream; see Views for details of creating views on streams.
Last Run Date (Read only)	PhixFlow displays the date and time that analysis was last run for this stream. This date is taken from the "to date" of the most recent stream set for this stream.
Last Run By	PhixFlow displays the user that last ran analysis on this stream.

Field	Description	What Data is Deleted
Keep for N Days	Enter the number of days to keep data in the stream. The age of data in a stream set is its 'to' date, relative to the 'to' date of the newest valid stream set in the stream.	When a stream-data-delete task runs for a stream and: Keep for X Days only is set, PhixFlow deletes stream data that is older than N days. Keep for Y Stream Sets only is set, PhixFlow keeps the most recent X stream sets and deletes stream sets that exceed X. both Keep for X Days and Keep for Y Stream Sets are set, PhixFlow deletes the stream data that meets both conditions. neither Keep for N Days nor Keep for X StreamSets are set, PhixFlow keeps stream data indefinitely. This can lead to performance issues.
Keep for X StreamSets	The number of stream sets to keep in the stream.
Keep Superseded for N Days	Available when Track Superseded Data is selected. Enter the number of days to keep superseded records in the stream data. Remember to keep superseded records for as long as you want the option to roll back the data. For example, when Keep Superseded for N Days is set to 4, you can run roll back on data that was superseded up to 4 days ago. A stream-data-delete task removes superseded records older than 4 days, so PhixFlow no longer has the data to be able to roll it back.	In a stream where the superseded date is tracked, the stream data contains a mixture of active records and records that have been superseded. When a stream-data-delete task runs for a stream and: Keep Superseded for N Days only is set, PhixFlow deletes data records that were superseded more than N days ago. Keep Superseded for X StreamSets only is set, PhixFlow deletes superseded records from the stream sets that exceed X. both Keep Superseded for N Days and Keep Superseded for X Stream Sets are set, PhixFlow deletes the superseded records that meet both conditions. neither Keep Superseded for N Days nor Keep Superseded for X Stream Sets are set, PhixFlow keeps superseded records indefinitely. This can lead to performance issues.
Keep Superseded for X StreamSets	Available when Track Superseded Data is selected. Enter the number of recent stream sets that will retain their superseded data records.
Apply Delete Filter	: Tick to open a filter tab where you can set filter conditions; see Filter. PhixFlow applies this filter when it runs a stream-data-delete task. Only the records which match the filter are deleted.

Stream