Overview
A pipe is a connector that links two elements in a PhixFlow model and sends data from the input to the output. Pipes allows you to control which attributes and which records from the input are delivered by to the output, although in most cases - with minimal configuration - you will get all columns and the records from the current run.
...
| Field | Description |
|---|
| Name | Enter a name. The name is used to refer to the pipe in other model elements. Pipe names default to in. The name: - must start with a letter
- must not be an Attribute Function name.
- must not include special characters, except underscore _
|
| Enabled | | Insert excerpt |
|---|
| _check_box_untick |
|---|
| _check_box_untick |
|---|
| nopanel | true |
|---|
|
to prevent the pipe from being used during an analysis run. | Insert excerpt |
|---|
| _check_box_ticked |
|---|
| _check_box_ticked |
|---|
| nopanel | true |
|---|
|
to indicate the pipe properties are complete and the pipe is ready to be used. |
| Static | Normally when a pipe requests data from a non-static input stream, that stream will first attempt to bring itself up to date, generating new stream sets as necessary, before supplying the data requested. However, if this field is ticked, the input stream will not run. Pipes from collectors cannot be marked as static. | Insert excerpt |
|---|
| _check_box_untick |
|---|
| _check_box_untick |
|---|
| nopanel | true |
|---|
|
when the pipe requests data from a non-static input stream, that stream will first attempt to bring itself up to date, generating new stream sets as necessary, before supplying the data requested. | Insert excerpt |
|---|
| _check_box_ticked |
|---|
| _check_box_ticked |
|---|
| nopanel | true |
|---|
|
to prevent the input stream from updating itself. The pipe will pull the existing data from the input stream. Pipes from collectors cannot be marked as static. |
| Mandatory | | Insert excerpt |
|---|
| _check_box_untick |
|---|
| _check_box_untick |
|---|
| nopanel | true |
|---|
|
| Insert excerpt |
|---|
| _check_box_ticked |
|---|
| _check_box_ticked |
|---|
| nopanel | true |
|---|
|
to indicate that, when multiple streams are being merged, there must be an input record from this pipe for an output record to be generated by the output stream. If this is a push pipe with positive offsets and this option is ticked then the notification to create another stream set will only be pushed along the pipe if the last stream set created contains at least one record. This causes the pipe to present each candidate set to the output stream in a different way than usual. |
| Multiplier | | Insert excerpt |
|---|
| _check_box_untick |
|---|
| _check_box_untick |
|---|
| nopanel | true |
|---|
|
is the default. | Insert excerpt |
|---|
| _check_box_ticked |
|---|
| _check_box_ticked |
|---|
| nopanel | true |
|---|
|
so that, for each output record generated by a stream, the stream will get a set of records from each of its input pipes. If the multiplier flag is ticked on one of these, then the stream will generate an output record for each record from the set of records provided by the multiplier pipe. For each output record, each of the other input pipes will provide the same set of records as normal. |
| Type | Select: - Pull: pull pipes are the most common type in PhixFlow - they "pull" data from the input to the output. Pull pipes are shown as solid arrows on models.
- Look-up: look-up pipes are used to enrich data. Typically, you will have one of more pull pipes to supply the base data for an output, and if needed one or more look-up pipes to enrich the base data with values from additional inputs. Look-up pipes are shown as dashed lines on models.
- Push: data is "pushed" rather than "pulled" into the output stream. Push pipes are most commonly used when sending data from streams to exporters (File Exporters, Database Exporters, HTTP Exporters). Push pipes are shown as dotted lines on models.
|
| Data to Read | Select the type of input data to use. - Latest: supply data from the current run (the latest stream set). This is the mostly commonly used option.
- Previous: supply data from the previous run (the previous stream set). This is used when you are comparing data for the current run with data from the previous run, for example, today's data with yesterday's.
- All: supply data from all runs (all stream sets).
- All Previous: supply data from all runs except the current run (all stream sets except the latest stream set).
- Same Run: this option should only be used where the input and output streams are set to Period: Transactional. The pipe will only collect data from inputs in the same analysis run. This configuration support several analysis runs going on at the same time without interfering with each other.
- Custom: select this option to display additional settings, described in the the Custom Data to Read section, below. We recommend that you only use the custom settings when directed to by PhixFlow consultants or support.
|
| Read Future Data | Use this option to exclude or include input streams sets that have future dates relative to the stream set you are generating. For details about how future stream sets occur, see Managing Future Stream Sets, above. | Insert excerpt |
|---|
| _check_box_untick |
|---|
| _check_box_untick |
|---|
| nopanel | true |
|---|
|
to exclude future stream sets from this analysis run. This is the default.| Insert excerpt |
|---|
| _check_box_ticked |
|---|
| _check_box_ticked |
|---|
| nopanel | true |
|---|
|
to include future stream sets in this analysis run. For example, for a stream with Period: Transactional , you will want to include new streams sets that are being added to the input stream after your analysis run starts. |
| Output |
|
...
Country Is in ["England","France","Germany"] fx
Sort/Group
Use this section to group and sort data as it comes through the pipe. for lookup pipes, this section is called Order/Index. This section has:
- a toolbar with standard buttons
- a grid that lists the attributes that you want to sort or use to group
- below the grid are the following options:
...
| Note |
|---|
You cannot aggregate data from attributes if the pipe's input is from: If you need to aggregate data from a database collector, you can use an SQL query. |
This section has:
- a toolbar with standard buttons
- a grid that lists the attributes that you want to aggregate.
...
| Field | Description |
|---|
| Data Expected | This checkbox is available when the pipe Type is Push or Pull. | Insert excerpt |
|---|
| _check_box_untick |
|---|
| _check_box_untick |
|---|
| nopanel | true |
|---|
|
means the pipe may receive no data from its input during an analysis run. | Insert excerpt |
|---|
| _check_box_ticked |
|---|
| _check_box_ticked |
|---|
| nopanel | true |
|---|
|
means PhixFlow reports an error if the pipe receives no data from the input datasource, collector or stream during an analysis run. |
| Allow Incomplete Stream Sets | | Insert excerpt |
|---|
| _check_box_untick |
|---|
| _check_box_untick |
|---|
| nopanel | true |
|---|
|
to complete a stream set before passing the data to the output stream. During the analysis run, PhixFlow pulls data into the input stream until the stream set is complete. If it cannot complete the stream set, PhixFlow reports an error message. PhixFlow cannot complete a stream set if: - either the input stream is set to be
| Insert excerpt |
|---|
| _static |
|---|
| _static |
|---|
| nopanel | true |
|---|
|
- or all of the pipes reading from the stream are static.
| Insert excerpt |
|---|
| _check_box_ticked |
|---|
| _check_box_ticked |
|---|
| nopanel | true |
|---|
|
PhixFlow ignores incomplete stream sets in a static input stream and does not report an error. | Tip |
|---|
You must tick this checkbox on all the pipes that will read from a static (or effectively static) input stream in the analysis run. PhixFlow will report an error if there is any pipe trying to complete the stream set during the analysis run. |
Pipes that are not used in the analysis run do not try to complete a stream set, so will not report an error. (Unused pipes can occur if they lead to streams on branches of the model that are not being run.) |
| | Anchor |
|---|
| pipeCacheSize |
|---|
| pipeCacheSize |
|---|
|
This option is useful for lookup pipes. PhixFlow uses the pipe cache when it looks-up data from streams or database collectors. For efficiency, the records are cached (stored temporarily in memory) so that if the same set of records need to be looked up again they are readily available without going back to the database.Enter a number to set a limit on the data cache size available for the pipe. You need to estimate the largest number of records that the lookup pipe will return on a single read. Check whether PhixFlow is looking up: - many records
The pipe does a single lookup onto a stream or database table to get a large number of records in one go, for example 10,000 records. - few records may times
The pipe does many lookups, getting a small number of records for each lookup, for example, 10 records at a time. In this case, PhixFlow is usually using a key value, such as an account number, to get the data. The key value is:- for a stream - the attribute used to filter the pipe
- for a database collector - a condition in the database query. For example:
| Code Block |
|---|
WHERE AccountNumber = _out.AccountNum |
If you do not set a limit for the cache, PhixFlow uses the system default set in System Configuration → System Tuning → Maximum Pipe Cache Size. | Expand |
|---|
| In the log for an analysis run, which is available in the system console, PhixFlow reports warnings when a single read returns: - over 90% of the specified cache size
- 100% or more of the cache size.
PhixFlow reports an error and stops the analysis run when: - a single read returns 100% or more of the cache size
- and the System Configuration → Enforce Cache Size limit flag is ticked.
| Code Block |
|---|
| title | Error Message: Cache Size Limit Exceeded |
|---|
| The Pipe "stream_name.lookup_pipe_name" cache is 100% full (the cache size is 10). |
|
| Expand |
|---|
| Every time the lookup pipe is referenced, PhixFlow calculates the values of all of the variable elements of the query or pipe filter, and checks if it already has a set of data in the cache retrieved using this set of variable values. If so the data is immediately returned from the cache. Otherwise, a new set of data is read from the stream of collector. If adding the new records to the cache would cause it to exceed the maximum cache size, previously cached results are removed to make enough room for the new results. |
|
| Buffer Size | Enter a number for the buffer size used to perform the stream calculation. If a large amount of data is being processed, then setting a large buffer size will give better performance. |
| Pipe View | Use this option to look up data from attributes that are present in a stream view on the input stream Select a stream view from the list. If the input stream has no views, the list will be empty. | Note |
|---|
Sorting or filtering of records must be set directly on the pipe. It is not inherited from the pipe view. |
Use the pipe view to limit the attributes that the pipe reads when: - a stream has lots of attributes containing many data records
- but you only need data from a few attributes.
Only the data for the attributes in the stream view are sent to the output stream. Pipe views are very useful: - during lookups a pipe loads and caches all of the attributes from the stream. This can use a lot of memory, especially when there are many records.
- during file export all data records from all attributes are exported.
- Create a new view on the source stream. In the view, only add the attributes you need.
- In the pipe Pipe View option, select the pipe view.
- Run analysis. PhixFlow only looks-up or exports data from the attributes specified on the view.
| Expand |
|---|
| title | Using pipe views with file exporters |
|---|
| If a File Exporter is configured to export to Excel or to HTML: - and it has an Excel template, PhixFlow uses this template and does not use any template specified in the pipe view.
- but it does not have an Excel template, PhixFlow will use the Excel template specified in the pipe view (if there is one)
- and the pipe view is a chart view, PhixFlow will export a PNG picture of the chart.
|
|
Max Records To Read
| Enter a number for the maximum number of records that should be read down this pipe. The pipe may read more than this number of records if it is configured to carry out multiple reads simultaneously. For example: - a pipe can be connected to a File Collector that reads multiple files simultaneously
- the pipe strategy is Directed with multiple workers.
|
| Strategy | Select an option to specify how this pipe should be implemented. See the section on Directed Merge Strategy |
Max Workers | This field is available when Strategy is Directed Enter the maximum number of concurrent worker tasks. When no value is specified, this defaults to 1. |
| Worker Size | This field is available when Strategy is Directed Enter the number of key values to read for a single worker task, which runs a single select statement. When no value is specified, this defaults to 1000. This is the maximum value that can be used when reading from an Oracle database. |
Custom Data to
...
Read
The following fields properties are available in the Basic Settings section if when you set Data To Read = to Custom:.
| Field | Description |
|---|
| From Offset | The Enter the offset applied to the start of the collection period, relative to the period in the output stream that requires populating. |
| To Offset | The Enter the offset applied to the end of the collection period, relative to the period in the output stream that requires populating. |
| Max Stream Sets | In almost all cases this specifies Enter the number of stream sets to be retrieved from the input stream. However, if this is For a push pipe with positive offsets this value indicates . enter the maximum number of stream sets that can be created i.e. the maximum number of cycles this pipe can initiate. |
| Only collect from same run | Every time the analysis engine runs, all of the stream sets that are created by all of the streams affected by that analysis run are given the same Run ID. | Insert excerpt |
|---|
| _check_box_untick |
|---|
| _check_box_untick |
|---|
| nopanel | true |
|---|
|
so the pipe can collect stream sets with different Run IDs.| Insert excerpt |
|---|
| _check_box_ticked |
|---|
| _check_box_ticked |
|---|
| nopanel | true |
|---|
|
so that the pipe will only collect stream sets from the input stream that have the same Run ID as the stream set currently being created by the output stream. You should only checkbox if both the input and output streams have Period set to Transactional. |
| Historied | If ticked, the pipe will collect data | Insert excerpt |
|---|
| _check_box_untick |
|---|
| _check_box_untick |
|---|
| nopanel | true |
|---|
|
so that all data will be collected from the input stream, regardless of period. In this case, any From Offset or To Offset values determine whether the required data periods in the input stream exist before the stream calculation can be carried out. | Insert excerpt |
|---|
| _check_box_ticked |
|---|
| _check_box_ticked |
|---|
| nopanel | true |
|---|
|
so that the pipe will collect data from the input stream by period.For example, if: - the from and to offsets are both 0.0
- and the output stream requires stream generation for the period 17/10/07 - 18/10/07
the pipe reads data from the input stream for the period 17/10/07 - 18/10/07. | Read Future Data | Use this option to exclude or include input streams sets that have future dates relative to the stream set you are generating. For details about how future stream sets occur, see Managing Future Stream Sets, above. | Insert excerpt |
|---|
_check_box_untick | _check_box_untick | | nopanel | true |
|---|
to exclude future stream sets from this analysis run. This is the default.| Insert excerpt |
|---|
_check_box_ticked | _check_box_ticked | | nopanel | true |
|---|
to include future stream sets in this analysis run. For example, for a stream with Period: Transactional , you will want to include new streams sets that are being added to the input stream after your analysis run starts. |