Forms: Stream
Streams are a key model component. They represent a structured store of data within PhixFlow receiving data from one or more components, processing it then storing it.
The following are the top-level configuration parameters for streams:
Form: Stream Details
The following fields are configured on the Details tab:
| Field | Description |
|---|---|
| Name | The name of the stream. |
| Stream Function | The type of function used to generate this stream - see stream functions . If this stream is to be populated through collection, then no function is selected. |
| Enabled | Whether or not the stream is enabled to run. |
| Run Alone |
If this flag is ticked then whenever the analysis engine needs to generate data for this stream it will first wait for all running tasks to complete before it starts. Any additional analysis tasks submitted while this stream is waiting to start, or while it is generating data, will wait until this stream has completed its analysis before they start. |
| Start Date | The date that this stream starts. Data will be populated into the stream from this date onwards. |
| Last Run Date | The date and time that analysis was last run for this stream. This date is taken from the "to date" of the most recent streamset for this stream. |
| Storage Type |
Specifies how data for the Stream should be stored:
|
| Index Scheme |
This determines how indexes on the Stream are organised.
There are two possible settings:
Note: it is not possible to change this setting if the Stream contains data. |
| Period |
This is the period of the stream. This can be either a regular period, or variable.
There are three possible settings:
|
| Prevent Parallel Processing |
This field only appears if the Period is set to Transactional. If ticked, it ensures that only a single stream set can be generated
at a time even if the stream receives several concurrent requests to generate data. This can be useful where you want to make sure that two analysis runs don't attempt to update the same records at the same time e.g. as a result of two people selecting the same records in a view and then hitting the same action button at the same time to process those records. |
| All Users Can View Data | If checked, this specifies that all users can view this data by default (provided they have the basic privilege to view streams). If this field is not checked, then access to the underlying data is controlled by dropping user groups onto the stream's "User Group" tab. Note that the default setting for this field on streams is controlled by the system parameter allowAccessToDataByDefault. |
| Default View | The default view selected for the stream. See help on Views for details of creating views on streams. |
| Keep for X Days |
The number of days data to keep in the stream. When an archive task runs for a stream, all stream data is deleted if it is at least Keep for X Days old or if it is older than the Keep for Y Stream Sets most recent valid stream sets. If both Keep for X Days and Keep for Y Stream Sets are set, stream data will be deleted only if it meets both conditions. If neither are set, stream data is kept indefinitely. If Save Archive to File is ticked, deleted items are first saved to archive files. The age of data in a stream set is its 'to' date relative to the 'to' date of the newest valid stream set in the stream. See here for how to set up and schedule an Archive Task. Please see the section below on href="#archiveExamples">Archiving Examples to see how this value can be used within Archiving strategies. |
| Keep for Y StreamSets |
The number of stream sets data to keep in the stream. See href="#keepForXDays">Keep for X Days for the main description of archiving. |
| Apply Archive Filter | If this flag is ticked then a dialog box appears within which a filter can be created. This filter will be applied during archiving and only the records which match the filter will be archived and deleted. |
| Save Archive to File | If Save Archive to File is set, archived data will be written to compressed archive files before being deleted. |
If the stream is linked into itself with a pipe, the following additional fields are available on the Details tab of the stream editor:
| Field | Description |
|---|---|
| Track Superseded Date | If ticked, new records will be compared to existing records and if a repeated record is found, the old one will be marked as 'superseded'. |
| Using Pipe | Name of the pipe used to retrieve existing records from the stream. This must be a pipe linking the stream back into itself. |
| Keep Superseded for X Days |
The number of days for which to keep superseded data in the stream. If Track Superseded Data is ticked, then this field will become visible/enabled. In a stream where the superseded date is tracked, the stream data will contain a mixture of superseded records and "active" records - that is, records that have not been superseded. When an archive task runs for a stream, records that were marked as superseded more than Keep Superseded for X Days days or more than Keep Superseded for Y Stream Sets stream sets ago are deleted. If both Keep Superseded for X Days and Keep Superseded for Y Stream Sets are set, superseded records will be deleted only if they meet both conditions. If neither are set, superseded records are not deleted. This means, for example, that if you have set Keep Superseded for X Days to 4, you will be able to roll back 3 days, making the 4th day the latest valid day. If Save Archive to File is ticked, deleted items are first saved to archive files. Please see the section below on href="#archiveSupercededExamples">Archiving Superceded Examples to see how this value can be used within Archiving strategies. |
| Keep Superseded for Y StreamSets |
The number of stream sets for which to keep superseded data in the stream. If Track Superseded Data is ticked, then this field will become visible/enabled. See href="#keepSupersededForXDays">Keep Superseded for X Days for the main description of archiving superseded records. |
If the stream period is Variable an additional field is available:
| Field | Description |
|---|---|
| Static Data | If ticked, this stream can be used to hold static data. This is usually reference, or "look up" data used as part of a stream calculation. In technical terms, setting this means that the stream will only update itself when the user requests an analysis run on the stream directly, either ad-hoc through the user interface, or via a task plan. When this stream is part of a larger analysis run, unlike all other types of stream the stream will not attempt to update itself. |
The following fields are configured on the Stream Attributes tab:
| Field | Description |
|---|---|
| Stream Attributes | See the Stream Attribute form. |
The following fields are configured on the Function Properties tab:
| Field | Description |
|---|---|
| Input Multiplier |
The input multiplier expression should evaluate to a list of one
or more values. For each value in the list, the internal variable
_inputMultiplier will be set to that value and the whole stream
processing will be repeated i.e. the pull pipes will be read and
the data from those pipes processed to generate output stream
items to be added to the current stream set.
For example :
do ( $aRange = [], addElement($aRange, rng.RangeFrom), addElement($aRange, rng.RangeTo), $bRange = [], addElement($bRange, $aRange), $bRange ) Where rng.RangeFrom = 500 and rng.RangeTo = 1000, the above example evaluates to [[500,1000]], which is a list containing 1 element, which is itself a list containing 2 elements.An Input Multiplier that evaluates to [3,4,7,8] would run the Stream 4 times. Because Input Multipliers are evaluated first in the PhixFlow Timing Cycle they are often used to look up values that can be passed to Database or other Collectors. |
| Output Multiplier |
This field is an
expression
which should evaluate to an array of values. A separate output record will be produced
for each value in the array and this value is available as _type in
each of the stream output attribute expressions. In effect this will multiply each
of the output records by the number of elements the returned list. For example : ifNull(in.ASSET, [1,10,12] , // else do [5,7] ) will create 3 records for every record in the stream if in.ASSET contains a value (setting _type = 1, 10 and 12 in each case). Otherwise it will create 2 records for every record in the stream (and set _type = 5 and 7).An Output Multiplier may also evaluate to a record, or a group of records. For example an Output Multiplier with the expression: do( lookup(lkin, $num = in.BNumber), lkin ) will return a list of records which match the lookup on the lkin pipe. In this case the required data can be extracted from the Output Multiplier using the following expression : do ( $values = _type, $values.account_num ) If the output multiplier expression evaluates to _NULL, an empty list of values or an empty list of records then a single output record will be produced with _type set to _NULL, _NULL or an empty record respectively. |
| Output Filter | Records created as output from the stream function can be filtered before they are written to the stream. Any attribute of the output filter can be used in the expression. |
| Key Tolerances | When building a candidate set, data can be grouped together using a specific key value, e.g. Account Number. If the key value is a number, setting a key tolerance will identify numbers within the given tolerance as the the same key value. |
| Allow Partial Set Processing | If ticked, when analysis reaches the end of a buffer block it submits the candidate set for processing, even if the next buffer block has a different key. |
The following fields are configured on the Views tab:
| Field | Description |
|---|---|
| Stream Views | See the Stream Views form. |
The following fields are configured on the Actions tab:
| Field | Description |
|---|---|
| Stream Actions | See the Stream Actions form. |
The following fields are configured on the Sort Orders tab:
| Field | Description |
|---|---|
| Sort Orders | See the Stream View Sort Orders form. |
The Filters tab allows you to define filters which may appear in the dropdown list of filters accessible from the header of each StreamView. To make a filter available to a user defined view then the filter must be added to the list of filters for that StreamView on the Filters tab of the StreamView editor form. However, all filters defined on this tab are available on the system generated Default View for this Stream. The following fields are configured on the Filters tab:
| Field | Description |
|---|---|
| Filters | See the Filter Condition Editor form. |
The following fields are configured through separate tabs on the form:
| Field | Description |
|---|---|
| Description | Description of the stream. |
Archiving Examples for Full Stream Sets
The table below assumes the stream to be archived currently contains 8 stream sets. Two from the current day and one from each of the previous 6 days.
In the table below the value null refers to the fact that no value has been entered into this field.
Note that archiving will always retain the maximum active stream sets in the data such that no conflicting stream sets will be archived.
| Archive After X Days | Keeping Latest Y StreamSets | Resulting Streams Archived/Retained |
|---|---|---|
| null | null | No stream sets will be archived. |
| 0 | null | All stream sets will be archived |
| 1 | null | The last day of valid steam sets will be retained. All earlier stream sets will be archived. In our example the 2 latest stream sets will be retained with the earliest 6 stream sets archived. |
| X | null | All stream sets which are older than X days before the latest valid stream set will be archived. |
| null | 0 | All stream sets will be archived |
| null | 1 | The last valid stream set will be retained, all other stream sets will be archived |
| null | Y | The most recent Y valid stream sets will be retained, all others stream sets will be archived. |
| 0 | 0 | All stream sets will be archived |
| 0 | 1 | The last valid stream set will be retained, all other stream sets will be archived |
| 1 | 0 | The last day of valid steam sets will be retained. All earlier stream sets will be archived. |
| 1 | 1 | The last day of valid steam sets will be retained regardless of if there are more than 1. If there are no stream sets in the last day then the first previous stream set will be retained instead. |
| X | Y | Will retain the maximum active stream sets in the data such that no conflicting stream sets will be archived. If X=3,Y=6 then although X says only archive stream sets more than 3 days old, we must keep a Y minimum of 6 stream sets. Hence the earliest 2 stream sets will be archived and the 6 latest retained. If X=3,Y=1 then although Y says only retain 1 stream set, we must retain all stream sets less than X (3) days old. Hence the earliest 4 stream sets will be archived and the 4 latest retained. |
Archiving Examples for Superceded Stream Sets
In the case where only the Keep Superseded for X Days and Keep Superseded for Y StreamSets fields are populated, the same logic in the table above will apply to the superseded records. Note that again archiving will always retain the maximum superseded stream sets in the data such that no conflicting stream sets will be archived.
In the cases where a mixture of the full archive fields Keep for X Days, Keep for Y StreamSets' and the superseded archive fields Keep Superseded for X Days, Keep Superseded for Y StreamSets are populated, then the full archive values will be first applied and the resultant stream item records will be archived and deleted. Only then will the remaining stream sets use the Keep Superseded ... values to apply a further condition to archive and delete any remaining non qualifying superseded records.
Form Icons
The form provides the standard form icons.
The form also provides the following icons on the Stream Attributes tab:
 |
Shows the list of File Collectors. |
 |
Shows the list of Streams. |
 |
Deletes the selected object from the list. |
 |
Adds a new Stream Attribute. |