...
For advanced configuration, see Advanced Pipe Configuration.
| Panel | ||||
|---|---|---|---|---|
| ||||
A pipe joining a datasource to a data collector has no details to edit. All the configuration for the output data set occurs in the collector - either a database collector for a database datasource, or an HTTP collector for an HTTP datasource. |
...
- Latest: supply data from the current run (the latest stream set). This is the mostly commonly used option.
- Previous: supply data from the previous run (the previous stream set). This is used when you are comparing data for the current run with data from the previous run, for example, today's data with yesterday's.
- All: supply data from all runs (all stream sets).
- Custom: If you are setting up a transactional model, choose this option so that you can select Only collector from same run. There are other fields revealed by selecting this option (see Advanced Pipe Configuration), but you are recommended not to update these unless directed to by PhixFlow consultants or support.
...
| Field | Description |
|---|---|
| Filter | Allows the user to set up a filter on the pipe. Also allows to set the flag to Include Audit Records. If not set, superseded records will be filtered out. |
| Sort/Group | Specify the group/ order by attributes on the pipe. |
| Aggregate attributes | Specify any aggregate attributes on the pipe. |
| Advanced | Configure advanced features on the pipe. |
| Anchor | ||||
|---|---|---|---|---|
|
...
For case 1, when using a lookup pipe, data retrieved is stored in a cache. See cache size for details. The cache extraction filter allows you, as you are processing a set of output records, to use different cached entries from the lookup for each of the records are you are processing. This is very fast compared to looking up from the source (i.e. going back to an external DB table or even another PhixFlow stream) for each output record.
...
| Anchor | ||||
|---|---|---|---|---|
|
A Pipe can be grouped and sorted by attributes of the input stream. These are set up in the Group/Order section of the Pipe form. In fact, this section is called Sort/Group for pull and push pipes, and Order/Index for lookup pipes.
...
Possible aggregate values are counts, summations, averages and maximum or minimum values of Stream Items grouped in the Group/Order tab of the Pipe.
| Field | Description |
|---|---|
| Stream Function | The Aggregate Function e.g. Count or Sum. |
| Attribute | The name of the Stream Attribute to be aggregated. Note that the value in this field is not used if the Aggregate Function is Count. |
| Name | A new name for the aggregated attribute. Note that this can be the same as the original Attribute. |
| Order | The order of the aggregate attribute. |
...
| Field | Description | |||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Data Expected | This field is available when the Pipe Type is Push or Pull. This flag allows the user to specify that the pipe is expecting to receive data. If ticked but no data is received this is treated as an error. | |||||||||||||||||||||||||||||||
| Allow Incomplete Stream Sets | Normally, when a pipe tries to read from an input stream that contains an incomplete stream set, PhixFlow will attempt to complete the stream set before passing data down the pipe. However if the stream is static (i.e. the stream has its 'static' flag ticked) or is effectively static (i.e. all of the pipes reading from it in this analysis run are static) then, instead of completing the stream set, an error message is produced indicating that you cannot read from this stream because it contains an incomplete stream set. If you do not want this error message to be produced when reading from static (or effectively static) streams, but would instead prefer PhixFlow to ignore the incomplete stream sets, then you must tick this box on all pipes that will read from the input stream in this analysis run. If there are multiple pipes that read from the input stream during this analysis run and even one of the pipes does not have this box ticked then you will not be allowed to read from the stream and the error message will be produced. Pipes which are not used in the current analysis run (for example where they lead to streams on branches of the model which are not run by the current task plan) have no effect on whether or not the error message is produced. | |||||||||||||||||||||||||||||||
| The cache is used when carrying out lookups from streams or database collectors. When doing a lookup, there are two common scenarios:
In case 2, the results returned are typically based on a key value, e.g. an account number. This will be used in the filter of the pipe, if you are reading from a stream, or in the query, if you are reading from a database collector. For example, the query in a database collector will include the condition:
For efficiency, the records are cached (stored temporarily in memory) so that if the same set of records need to be looked up again they are readily available without going back to the database. This field allows you to set a limit on the size of the cache. Setting a limit is important because if you do not, the cache can become very large and consume a lot of memory, which can lead to a slow down in both your tasks and those of other users of PhixFlow. To set the cache size, try to estimate the largest number of records that the lookup pipe will return on a single read. If you do not set a limit, it will default to the system-wide default, specified in the Maximum Pipe Cache Size in the System Tuning tab of the System Configuration.
| |||||||||||||||||||||||||||||||
| Buffer Size | The buffer size used to perform the stream calculation. If a large amount of data is being processed, then setting a large buffer size will give better performance. | |||||||||||||||||||||||||||||||
| Pipe View | The pipe view is used to limit which fields are retrieved down the pipe and in what order, and in some circumstances how each field is to be formatted. You can select from any of the views that have been configured on the source stream. Please note that any sorting or filtering of records will have to be applied directly on the pipe, and will not be inherited from the pipe view. The pipe view is used in two contexts. During lookupsPipe views can also be used on lookup pipes to limit the fields that are returned by the lookup request. This is most useful in the scenario where the you want to read and cache data on a lookup pipe from a stream that has lots of attributes but where only a small number of attributes are actually required. You can simply create a new view on the source stream listing only the attributes needed, then specify it as the pipe view on the lookup pipe. Only those attributes specified on the view will then be loaded. During File ExportWhen sending data to a file exporter only those fields specified on the pipe view will be exported. If no pipe view is supplied then all fields will be exported. | |||||||||||||||||||||||||||||||
Max Records To Read | The maximum number of records that should be read down this pipe. The pipe may read more than this number of records if it is configured to carry out multiple reads simultaneously e.g. if it is connected to a File Collector which has been configured to read multiple files simultaneously or if this pipe strategy is "Directed" with multiple workers. | |||||||||||||||||||||||||||||||
| Strategy | The Execution Strategy determines how this pipe should be implemented. See the section on Directed Merge Strategy | |||||||||||||||||||||||||||||||
Max Workers | This field is only available if Strategy = Directed The maximum number of concurrent worker tasks. If blank, this defaults to 1. | |||||||||||||||||||||||||||||||
| Worker Size | This field is only available if Strategy = Directed The number of key values to read for a single worker task (which runs a single select statement). If blank, this defaults to 1000. This is the maximum value that can be used when reading from an Oracle database. |
...