...
| Panel | ||||
|---|---|---|---|---|
| ||||
A pipe joining a datasource to a data collector has no details to edit. All the configuration for the output data set occurs in the collector - either a database collector for a database datasource, or an HTTP collector for an HTTP datasource. |
Basic Settings
| Field | Description | ||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Name | Enter a name. The name is used to refer to the pipe in other model elements. Pipe names default to The name:
| ||||||||||||||||
| Enabled |
|
to indicate the pipe properties are complete and the pipe is ready to be used. | |||||||||
| Static | Normally when a pipe requests data from a non-static input stream, that stream will first attempt to bring itself up to date, generating new stream sets as necessary, before supplying the data requested. However, if this field is ticked, the input stream will not run. Pipes from collectors cannot be marked as static.
|
when the pipe requests data from a non-static input stream, that stream will first attempt to bring itself up to date, generating new stream sets as necessary, before supplying the data requested.
|
to prevent the input stream from updating itself. The pipe will pull the existing data from the input stream. Pipes from collectors cannot be marked as static. | |||||||||||||||||
| Mandatory |
|
Select:
- Pull: pull pipes are the most common type in PhixFlow - they "pull" data from the input to the output. Pull pipes are shown as solid arrows on models.
- Look-up: look-up pipes are used to enrich data. Typically, you will have one of more pull pipes to supply the base data for an output, and if needed one or more look-up pipes to enrich the base data with values from additional inputs. Look-up pipes are shown as dashed lines on models.
- Push: data is "pushed" rather than "pulled" into the output stream. Push pipes are most commonly used when sending data from streams to exporters (File Exporters, Database Exporters, HTTP Exporters). Push pipes are shown as dotted lines on models.
Select the type of input data to use.
- Latest: supply data from the current run (the latest stream set). This is the mostly commonly used option.
- Previous: supply data from the previous run (the previous stream set). This is used when you are comparing data for the current run with data from the previous run, for example, today's data with yesterday's.
- All: supply data from all runs (all stream sets).
- All Previous: supply data from all runs except the current run (all stream sets except the latest stream set).
- Same Run: this option should only be used where the input and output streams are set to Period: Transactional. The pipe will only collect data from inputs in the same analysis run. This configuration support several analysis runs going on at the same time without interfering with each other.
- Custom: select this option to display additional settings, described in Advanced Pipe Configuration, below. We recommend that you only use the custom settings when directed to by PhixFlow consultants or support.
Data To Read
...
to indicate that, when multiple streams are being merged, there must be an input record from this pipe for an output record to be generated by the output stream. If this is a push pipe with positive offsets and this option is ticked then the notification to create another stream set will only be pushed along the pipe if the last stream set created contains at least one record. This causes the pipe to present each candidate set to the output stream in a different way than usual. | |||||||||||||||||
| Multiplier |
| ||||||||||||||||
| Type | Select:
| ||||||||||||||||
| Data to Read | Select the type of input data to use.
|
...
| |||||||||||||||||
| Read Future Data | Use this option to exclude or include input streams sets that have future dates relative to the stream set you are generating. For details about how future stream sets occur, see Managing Future Stream Sets, above.
| ||||||||||||||||
| Output |
| Panel | ||||||
|---|---|---|---|---|---|---|
| ||||||
Some of the stream sets on the input stream will have dates in the future relative to some of the stream sets you are rebuilding.By default, the Read Future Data checkbox is not ticked. This means pipes ignore any stream sets with dates in the future relative to some of the stream set sets you are generating. You want to ignore future stream sets when you rebuild an old stream set, because you want the pipe to retrieve the same data on the rerun as it retrieved when the stream set was first built.When you run analysis on a stream with a transactional period, it is possible that as your analysis is still running, a different run can start and complete. This run can generate additional stream sets on the input stream with a future data relative to the date of the rebuilding. By default, the Read Future Data checkbox is not ticked. This means pipes ignore any stream sets with dates in the future relative to the stream set you are generating. For transactional input streams, You want to ignore future stream sets when you rebuild an old stream set, because you want the pipe to use these future streams. To do this, tick the Read Future Data checkbox. |
Static
Normally when a pipe requests data from a non-static input stream, that stream will first attempt to bring itself up to date, generating new stream sets as necessary, before supplying the data requested. However, if this field is ticked, the input stream will not run. Pipes from collectors cannot be marked as static.
Mandatory
If ticked, when multiple Streams are being merged then there must be an input record from this Pipe for an output record to be generated by the output Stream.
| Panel | ||||||
|---|---|---|---|---|---|---|
| ||||||
If this is a push pipe with positive offsets and this flag is ticked then the notification to create another stream set will only be pushed along the pipe if the last stream set created contains at least one record. |
Multiplier
This causes the pipe to present each candidate set to the output stream in a different way than usual. The multiplier flag is on the Advanced tab of the form.
For each output record generated by a stream, the stream will get a set of records from each of its input pipes. If the multiplier flag is ticked on one of these, then the stream will generate an output record for each record from the set of records provided by the multiplier pipe. For each output record, each of the other input pipes will provide the same set of records as normal.
Filters, sorting and grouping, aggregating
Filters, sorting and grouping, and aggregating are configured through their own sections on the form:
...
The data being delivered by a pipe can be filtered.
Filters are made up of a set of clauses; each clause in turn contains a number of conditions. These conditions must be satisfied for data to be passed through the pipe.
Form Icons
The form provides the following buttons:
...
...
Add a clause or condition.
...
...
Delete a clause or condition.
...
...
Specifies that the value entered is a literal value. Click this icon to change this - to specify that the value entered be evaluated as an expression.
...
...
Specifies that the value entered is a evaluated as an PhixFlow expression. Click this icon to change this - to specify that the value entered be treated as a literal. Note : ["123", "234", "345"] looks like a literal value but it can be evaluated as an expression.
...
...
Filter on Current User
Sometimes when running analysis you want to select, from the source, only records belonging to the currently logged in user. To set a filter where, say, an attribute in the source Owner equals the current logged in user, add a condition to the filter like this:
Owner Equals _user.name fx
Enter a list of values for an "Is In" or "Is Not In" filter
If you want to based on a list of values, use the Is in or Is not in comparators, then type the list of values into the comparison field as a comma separated list like this:
Country Is in England, France, Germany ABC
In this case you must NOT click the ABC icon to convert the value to an fx, because this will indicate that the value is a formula; it must be left as a literal value. If you do click the ABC icon, then the value must be entered like this:
Country Is in ["England","France","Germany"] fx
...
retrieve the same data on the rerun as it retrieved when the stream set was first built. When you run analysis on a stream with a transactional period, it is possible that as your analysis is still running, a different run can start and complete. This run can generate additional stream sets on the input stream with a future data relative to the date of the stream set you are generating. For transactional input streams, you want the pipe to use these future streams. To do this, tick the Read Future Data checkbox. |
Filter
| Anchor | ||||
|---|---|---|---|---|
|
The data being delivered by a pipe can be filtered.
Filters are made up of a set of clauses; each clause in turn contains a number of conditions. These conditions must be satisfied for data to be passed through the pipe.
Form Icons
The form provides the following buttons:
| Open the expression in a larger editor. |
Filter on Current User
Sometimes when running analysis you want to select, from the source, only records belonging to the currently logged in user. To set a filter where, say, an attribute in the source Owner equals the current logged in user, add a condition to the filter like this:
Owner Equals _user.name fx
Enter a list of values for an "Is In" or "Is Not In" filter
If you want to based on a list of values, use the Is in or Is not in comparators, then type the list of values into the comparison field as a comma separated list like this:
Country Is in England, France, Germany ABC
In this case you must NOT click the ABC icon to convert the value to an fx, because this will indicate that the value is a formula; it must be left as a literal value. If you do click the ABC icon, then the value must be entered like this:
Country Is in ["England","France","Germany"] fx
Field | Description | ||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Include History Records |
| ||||||||||||||||
| Condition | Select one of the options
To add more conditions, hover your mouse pointer over this field to display the
| ||||||||||||||||
Hover your mouse pointer over the Condition field to display this button. | |||||||||||||||||
| Clause | Select an option from the list. PhixFlow adds more fields where you can:
| ||||||||||||||||
Hover your mouse pointer over a filter clause to display this button. | |||||||||||||||||
Indicates the value entered is a literal value. Click this icon to treat the value as an expression. | |||||||||||||||||
Indicates the value entered is an expression. Click this icon to treat the value to a literal string. Note: ["123", "234", "345"] looks like a literal value but it can be evaluated as an expression. | |||||||||||||||||
| |||||||||||||||||
| Cache Extraction Filter | Click  to open the expression in a larger editor.A cache extraction filter allows you to further filter the data retrieved by a pipe. These are not commonly used, but are sometimes helpful when either:
For case 1, when using a lookup pipe, data retrieved is stored in a cache. See cache size for details. The cache extraction filter allows you, as you are processing a set of output records, to use different cached entries from the lookup for each of the records are you are processing. This is very fast compared to looking up from the source (i.e. going back to an external DB table or even another PhixFlow stream) for each output record. E.g. you want to look up the credit rating for a customer for a set of transactions - in the output, each transaction is represented by a single output record. You create an indexed lookup pipe using CustNo as the key for the index. This means that for each new CustNo you encounter in the data, all the credit rating entries for that CustNo would be |
...
| Code Block |
|---|
StartDate >= _out.TransDate && (EndDate <= _out.TransDate || EndDate == _NULL) |
Cache extraction filters are entered free hand.
The attribute names referenced must exist in a stream. This means that the each attribute must be one of:
...
retrieved by the pipe and placed into the cache. The credit rating for each customer is fully historied, so you get a number of entries for each CustNo. To get the relevant lookup entry for each output report (each transaction), you need to compare the transaction date of the output record to the dates of credit rating entries in the cache. So to extract the relevant record, you include a cache extraction filter in the form:
Cache extraction filters are entered free hand. The attribute names referenced must exist in a stream. This means that the each attribute must be one of:
|
Sort/Group
| Field | Description |
|---|---|
| Filter | Allows the user to set up a filter on the pipe. Also allows to set the flag to Include Audit Records. If not set, superseded records will be filtered out. |
| Sort/Group | Specify the group/ order by attributes on the pipe. |
| Aggregate attributes | Specify any aggregate attributes on the pipe. |
| Advanced | Configure advanced features on the pipe. |
Aggregate Attributes
- if you are reading from an external database table, one of the fields returned by the database collector AND an attribute in the output stream - i.e. to use an attribute with the source as a database collector, there must be an attribute of matching name in the output stream
- an attribute in the destination stream, in which case you will refer to it using the format
_out.AttributeName
...