Overview
A pipe is a connector that links two elements in a PhixFlow model and sends data from the input to the output. Pipes allows you to control which attributes and which records from the input are delivered by to the output, although in most cases - with minimal configuration - you will get all columns and the records from the current run.
...
| Panel | ||||||
|---|---|---|---|---|---|---|
| ||||||
Some of the stream sets on the input stream will have dates in the future relative to some of the stream sets you are rebuilding. By default, the Read Future Data checkbox is not ticked. This means pipes ignore any stream sets with dates in the future relative to the stream set you are generating. You want to ignore future stream sets when you rebuild an old stream set, because you want the pipe to retrieve the same data on the rerun as it retrieved when the stream set was first built. When you run analysis on a stream with a transactional period, it is possible that as your analysis is still running, a different run can start and complete. This run can generate additional stream sets on the input stream with a future data relative to the date of the stream set you are generating. For transactional input streams, you want the pipe to use these future streams. To do this, tick the Read Future Data checkbox. |
Filter
...
...
Filter
...
Filters are made up of a set of clauses; each clause in turn contains a number of conditions. These conditions must be satisfied for data to be passed through the pipe.
Form Icons
The form provides the following buttons:
...
...
Filter on Current User
Sometimes when running analysis you want to select, from the source, only records belonging to the currently logged in user. To set a filter where, say, an attribute in the source Owner equals the current logged in user, add a condition to the filter like this:
Owner Equals _user.name fx
Enter a list of values for an "Is In" or "Is Not In" filter
If you want to based on a list of values, use the Is in or Is not in comparators, then type the list of values into the comparison field as a comma separated list like this:
Country Is in England, France, Germany ABC
In this case you must NOT click the ABC icon to convert the value to an fx, because this will indicate that the value is a formula; it must be left as a literal value. If you do click the ABC icon, then the value must be entered like this:
Country Is in ["England","France","Germany"] fx
...
...
...
Select one of the options
- Where ALL...
- Where ANY...
...
Hover your mouse pointer over the Condition field to display this button.
Add another condition to your filter.
...
Select an option from the list. PhixFlow adds more fields where you can:
- select how the filter matches (for example,
equals,contains,is null) - enter a string that the filter uses to match the data. The string can be an expression or a literal string.
...
Hover your mouse pointer over a filter clause to display this button.
Delete the selected clause or condition from the filter.
...
Indicates the value entered is a literal value. Click this icon to treat the value as an expression.
...
Indicates the value entered is an expression. Click this icon to treat the value to a literal string.
Note: ["123", "234", "345"] looks like a literal value but it can be evaluated as an expression.
...
A cache extraction filter allows you to further filter the data retrieved by a pipe. These are not commonly used, but are sometimes helpful when either:
- Optimising performance on a lookup pipe when for a set of records, the record you require from the lookup depends on non-key data, e.g. the date
- When getting data from a pull pipe when the filter requires that you compare one value in each record with another; this is not possible within a standard filter.
For case 1, when using a lookup pipe, data retrieved is stored in a cache. See cache size for details. The cache extraction filter allows you, as you are processing a set of output records, to use different cached entries from the lookup for each of the records are you are processing. This is very fast compared to looking up from the source (i.e. going back to an external DB table or even another PhixFlow stream) for each output record.
E.g. you want to look up the credit rating for a customer for a set of transactions - in the output, each transaction is represented by a single output record. You create an indexed lookup pipe using CustNo as the key for the index. This means that for each new CustNo you encounter in the data, all the credit rating entries for that CustNo would be retrieved by the pipe and placed into the cache. The credit rating for each customer is fully historied, so you get a number of entries for each CustNo. To get the relevant lookup entry for each output report (each transaction), you need to compare the transaction date of the output record to the dates of credit rating entries in the cache. So to extract the relevant record, you include a cache extraction filter in the form:
| Code Block |
|---|
StartDate >= _out.TransDate && (EndDate <= _out.TransDate || EndDate == _NULL) |
Cache extraction filters are entered free hand.
The attribute names referenced must exist in a stream. This means that the each attribute must be one of:
- an attribute in a source stream, if you are reading from a stream
...
Field | Description | ||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Include History Records |
| ||||||||||||||||
| Condition | Select one of the options
To add more conditions, hover your mouse pointer over this field to display the
| ||||||||||||||||
Hover your mouse pointer over the Condition field to display this button. | |||||||||||||||||
| Clause | Select an option from the list. PhixFlow adds more fields where you can:
| ||||||||||||||||
Hover your mouse pointer over a filter clause to display this button. | |||||||||||||||||
Indicates the value entered is a literal value. Click this icon to treat the value as an expression. | |||||||||||||||||
Indicates the value entered is an expression. Click this icon to treat the value to a literal string. Note: ["123", "234", "345"] looks like a literal value but it can be evaluated as an expression. | |||||||||||||||||
| Open the expression in a larger editor. | ||||||||||||||||
| Cache Extraction Filter | A cache extraction filter allows you to further filter the data retrieved by a pipe. These are not commonly used, but are sometimes helpful when either:
For case 1, when using a lookup pipe, data retrieved is stored in a cache. See cache size for details. The cache extraction filter allows you, as you are processing a set of output records, to use different cached entries from the lookup for each of the records are you are processing. This is very fast compared to looking up from the source (i.e. going back to an external DB table or even another PhixFlow stream) for each output record. E.g. you want to look up the credit rating for a customer for a set of transactions - in the output, each transaction is represented by a single output record. You create an indexed lookup pipe using CustNo as the key for the index. This means that for each new CustNo you encounter in the data, all the credit rating entries for that CustNo would be retrieved by the pipe and placed into the cache. The credit rating for each customer is fully historied, so you get a number of entries for each CustNo. To get the relevant lookup entry for each output report (each transaction), you need to compare the transaction date of the output record to the dates of credit rating entries in the cache. So to extract the relevant record, you include a cache extraction filter in the form:
Cache extraction filters are entered free hand. The attribute names referenced must exist in a stream. This means that the each attribute must be one of:
|
Filter Examples
Filter on Current User
Sometimes when running analysis you want to select, from the source, only records belonging to the currently logged in user. To set a filter where, say, an attribute in the source Owner equals the current logged in user, add a condition to the filter like this:
Owner Equals _user.name fx
Enter a list of values for an "Is In" or "Is Not In" filter
If you want to based on a list of values, use the Is in or Is not in comparators, then type the list of values into the comparison field as a comma separated list like this:
Country Is in England, France, Germany ABC
In this case you must NOT click the ABC icon to convert the value to an fx, because this will indicate that the value is a formula; it must be left as a literal value. If you do click the ABC icon, then the value must be entered like this:
Country Is in ["England","France","Germany"] fx
Sort/Group
This section has a toolbar with standard buttons.
The grid contains a list of attributes from the input stream. You can group and sort the data as it comes through the pipe, using this grid.
To add attributes to the list, click
to open a New Group By Attribute property tab.. Insert excerpt _add _add nopanel true
To remove an attribute, use 
Permanently Delete in the toolbar.
A list of the stream attributes used to sort the data.
To add attributes,
click
| Insert excerpt | ||||||
|---|---|---|---|---|---|---|
|
drag a stream attribute into the grid.
To set the sort or group properties for an attribute, double-click it to open its properties. following fields:
OrderThe order that the sorting will be appliedNameThe name of the stream attribute - in the sort order attribute editor, this is read-only.Direction
One of:
(A-Z) - ascending, e.g. A to Z, 1 to 9, earliest to latest date
(Z-A) - descending, e.g. Z to A, 9 to 1, latest to earliest date
DescriptionDescription of the sort order attribute.Access Permissions
| Field | Description |
|---|---|
| Filter | Allows the user to set up a filter on the pipe. Also allows to set the flag to Include Audit Records. If not set, superseded records will be filtered out. |
| Sort/Group | Specify the group/ order by attributes on the pipe. |
| Aggregate attributes | Specify any aggregate attributes on the pipe. |
| Advanced | Configure advanced features on the pipe. |
...