Forms: Pipe

A pipe is a connector that links two elements in a PhixFlow model. A pipe joining a datasource to a data collector has no editable details.

Form: Pipe Details

The following fields are configured on the Details tab:

Field	Description
Name	The name of the pipe. The name is automatically generated when the pipe is first created to be 'in' (or the equivalent word in the user's selected language). If the name 'in' has already been used then a number will be added to the name to make it unique, e.g. in2
Input	The name of the pipe. In particular, this name will be used in any attribute expressions defined for the stream, when referring to attributes of the input stream. Please note Pipes cannot have the same name as any of the PhixFlow Attribute Functions, otherwise Expressions, using them will not function correctly
Output	Name of the output stream, that is, the stream at the end (the arrow end) of the pipe. This cannot be edited through the dialogue; if you wish to update this, you will need to go back into the model view, delete the existing pipe and create a new pipe between the two streams as needed.
Type	There are 3 options available: Push: data is 'pushed' rather than 'pulled' into the output stream. Every time data is written to the input stream, the pipe 'informs' the output stream and the Analysis Engine will attempt to run stream generation on the output stream. Push pipes are shown as blue lines on modelling screens. Pull: This type of pipe will pass data along as it is calculated which may be before the streamset is complete or any of the records have been committed to the database. This may improve performance over other pipes which wait until the input streams have committed their data to the database before reading the data back in again (which allows the database to apply the filtering and aggregation rules). Because PUSH pipes don't get their data from the database, but instead receive it direct from the input stream, any filters or aggregation specified on the pipe will be ignored. Pull pipes are shown as blue lines on modelling screens. Look-up: used to reference data, see lookup for more details. Look-up pipes are shown as dotted lines on modelling screens.
DataToRead	This field is used to determine which Streamsets to read from the input Stream. There are 4 options available: Latest: This will cause the pipe to read only the latest Streamset from the input Stream. Previous: This will cause the pipe to read the Streamset before the latest Streamset on the input Stream. All: This will cause the pipe to read all Streamsets from the input Stream. However, In some circumstances the input Stream may have Streamsets that have dates in the future relative to the Streamset the output Stream is creating. This may happen for example if you have rolled back a number of Streamsets on the output Stream but have not rolled back the corresponding Streamsets on the input Stream and have then requested that one of the rolled back Streamsets be rebuilt. Some of the Streamsets on the input Stream will then have dates in the future relative to the Streamset you are rebuilding. By default the pipe will ignore any Streamsets with dates in the future relative to the Streamset you are generating so that if you are rebuilding an old Streamset the pipe will retrieve the same data on the rerun as it retrieved when the Streamset was first built. Similarly if you are running a Transactional Stream then it is possible that while your analysis run is taking place, other analysis runs which started after yours may have managed to complete before yours thereby generating additional Streamsets on the input Stream with a future data relative to the date of the Streamset you are generating. For Transactional input Streams it is possible to tell the pipe not to ignore these future Streamsets by ticking the Read Future Data tickbox on the Advanced tab. Custom: If this option is selected then you need to specify which input Streamsets to read by configuring combinations of the following fields on the Advanced tab: Read Future Data Only collect from the same run Max Stream Sets Historied From Data Offset To Date Offset
Static	Normally when a Pipe requests data from a non-static input Stream then that Stream will first attempt to bring itself up to date, generating new Streamsets as necessary, before supplying the data requests. However, if this field is ticked, the input Stream will not attempt to do this.
Enabled	If this flag is not ticked then it is an indication to PhixFlow that the Stream is not ready to be used during any analysis runs and should be therefore be ignored.
Mandatory	If ticked, when multiple Streams are being merged then there must be an input record from this Pipe for an output record to be generated by the output Stream. If this is a push pipe with positive offsets and this flag is ticked then the notification to create another stream set will only be pushed along the pipe if the last stream set created contains at least one record.
Multiplier	When processing data, a Stream first constructs CandidateSets ready for processing from its (non-Multiplier) input Pipes. For each Multiplier Pipe, the Stream then multiplies each CandidateSet by creating a copy of the original CandidateSet for each row returned from the Multiplier Pipe. Note that each resulting CandidateSet contains the original CandidateSet plus one row from the Multiplier Pipe.
Index Type	Look-up pipes can be configured for fast "indexed" access to cached data collected from external tables, files or from other streams. Indexed access is controlled through configuring a pipe with an index and setting index expressions on grouping attributes. If the Type field on the Pipe is set to 'Look-up' then the field "Index Type" becomes available. This can have the value "None" meaning that there are no index keys or "Exact Match", "Best Match" or "Near Match" as described below: Exact Match: The pipe will retrieve data from its cache based on an exact match look-up with the values provided after evaluating the index expressions on the "Group By" attributes. Best Match: The pipe will retrieve data from its cache based on a "Best Match" look-up after evaluating the index expressions on the "Group By" attributes. Note that the last Group By Attribute with a key expression is used for the best match lookup. The index keys on any Group By attributes with a lower sequence number are used as an initial "Exact Match" to find the set of data on which to do the "Best Match". The "Best Match" is defined as the longest key value which matches the evaluated index expression. Near Match: The pipe will retrieve data from its cache based on a "Near Match" look-up after evaluating the index expressions on the "Group By" attributes. Note that the last Group By Attribute with a key expression is used for the near match lookup. The index keys on any Group By attributes with a lower sequence number are used as an initial "Exact Match" to find the set of data on which to do the "Best Match". When "Near Match" is selected, an additional field appears where you can enter an expression which should evaluate to a number representing the allowed number of edits (e.g. deletions, insertions, substitutions and transpositions) which can be made when comparing the result of the index expression to the index key in order to achieve a match. For example if the index key is "Smyhte" and the result of the index expression is "Smith" this would still be a match providing that the allowed number of edits is 3 or more (i.e. substitute the 'i' for a 'y', transpose the 't' and the 'h' and then insert an 'e' at the end).

The following fields are configured on the Advanced tab:

...

The Execution Strategy determines how this pipe should be implemented.

Where this pipe is a push or pull pipe into a Merge Stream, the Default Execution Strategy is to select all stream items from the input Stream sorted by the Group By attributes, then to read items from all input pipes simultaneously, constructing candidate sets from items with matching key values.

Where the Directed Execution Strategy is applied to a specific non-mandatory pipe, the other pipes with the Default Strategy operate as above, each being sorted then merged to generate a sequence of candidate sets; the Directed pipe then runs worker tasks to select the additional items by matching key value. These selects are batched up so that each worker reads items for many key values in a single select (see Worker Size), and many workers are run in parallel (see Max Workers).

In general, the Directed strategy should only be used where

the number of items in the source Stream is so large that the sorting phase of the Default strategy takes too long, or
only a small subset of the items are needed (the majority being discarded because they have key values that don't match the key values read on one of the other mandatory pipes).

Changing the Execution Strategy will make the Merge faster or slower depending on the input data and the details of the input and output Streams, but will not change the business logic of the Merge (i.e. which input items are grouped into candidate sets).

If the input to the pipe is a Collector, the list of key values is made available as _keyList. This will typically be used with an in clause in the Collector query. For example:

select * from customer where account_num in ({_keyList}) order by account_num

...

The maximum number of concurrent worker tasks.

If blank, this defaults to 1.

...

The number of key values to read for a single worker task (which runs a single select statement).

If blank, this defaults to 1000. This is the maximum value that can be used when reading from an Oracle database.

...

The offset applied to the start of the collection period, relative to the period in the output stream that requires populating.

The units are the period of the output stream, that is, if the output stream has a daily period, then setting from date offset = -1.0 means that the start of the collection period will be 1 day earlier than the start of the period in the output stream that is being calculated.

If this is a push pipe then a positive offset can be input. This will tell the stream to run again and generate another stream set.

...

The offset applied to the end of the collection period, relative to the period in the output stream that requires populating.

The units are the period of the output stream, that is, if the output stream has a daily period, then setting to date offset = -1.0 means that the end of the collection period will be 1 day earlier than the end of the period in the output stream that is being calculated.

If this is a push pipe then a positive offset can be input. This will tell the stream to run again and generate another stream set.

...

Only collect from the same run
Max Stream Sets (this may also be set to zero)
Historied

...

During Drill Down

...

During Look Ups

...

During File Export

...

The following fields are configured through separate tabs on the form:

Field	Description
Filter	Allows the user to set up a filter on the pipe.
Sort/Group	See the Order By Attribute form.
Aggregate Attributes	See the Aggregate Attribute form.

Form Icons

The form provides the standard form icons.

The form also provides the following icons on the Filter tab:

Image Removed	Adds a clause to the filter.
Image Removed	Deletes the selected clause or condition from the filter.
Image Removed	Adds a condition to a clause of the filter.

The form also provides the following icons on both the Sort/Group and Aggregate Attributes tabs:

Image Removed	Shows the list of attributes that can be added as sort/group or aggregate attributes.
Image Removed	Deletes the selected object from the list.
Image Removed	Adds an object to the list.

Overview

A pipe is a connector that links two elements in a PhixFlow model and sends data from the input to the output. Pipes allows you to control which attributes and which records from the input are delivered by to the output, although in most cases - with minimal configuration - you will get all columns and the records from the current run.

The pipe must be enabled to make it active.

For advanced configuration, see Advanced Pipe Configuration.

Insert excerpt

	_standard_settings
	_standard_settings
nopanel	true

Panel

bgColor	#e6f0ff
titleBGColor	#99c2ff

A pipe joining a datasource to a data collector has no details to edit. All the configuration for the output data set occurs in the collector - either a database collector for a database datasource, or an HTTP collector for an HTTP datasource.

Basic Settings

Field Description

Name

Enter a name. The name is used to refer to the pipe in other model elements. Pipe names default to in.

The name:

must start with a letter
must not be an Attribute Function name.
must not include special characters, except underscore _

Enabled

Insert excerpt

	_check_box_untick
	_check_box_untick
nopanel	true

to prevent the pipe from being used during an analysis run.

Insert excerpt

	_check_box_ticked
	_check_box_ticked
nopanel	true

to indicate the pipe properties are complete and the pipe is ready to be used.

Static

Normally when a pipe requests data from a non-static input stream, that stream will first attempt to bring itself up to date, generating new stream sets as necessary, before supplying the data requested. However, if this field is ticked, the input stream will not run. Pipes from collectors cannot be marked as static.

Insert excerpt

	_check_box_untick
	_check_box_untick
nopanel	true

when the pipe requests data from a non-static input stream, that stream will first attempt to bring itself up to date, generating new stream sets as necessary, before supplying the data requested.

Insert excerpt

	_check_box_ticked
	_check_box_ticked
nopanel	true

to prevent the input stream from updating itself. The pipe will pull the existing data from the input stream.

Pipes from collectors cannot be marked as static.

Mandatory

Insert excerpt

	_check_box_untick
	_check_box_untick
nopanel	true

Insert excerpt

	_check_box_ticked
	_check_box_ticked
nopanel	true

to indicate that, when multiple streams are being merged, there must be an input record from this pipe for an output record to be generated by the output stream.

If this is a push pipe with positive offsets and this option is ticked then the notification to create another stream set will only be pushed along the pipe if the last stream set created contains at least one record. This causes the pipe to present each candidate set to the output stream in a different way than usual.

Multiplier

Insert excerpt

	_check_box_untick
	_check_box_untick
nopanel	true

is the default.

Insert excerpt

	_check_box_ticked
	_check_box_ticked
nopanel	true

so that, for each output record generated by a stream, the stream will get a set of records from each of its input pipes. If the multiplier flag is ticked on one of these, then the stream will generate an output record for each record from the set of records provided by the multiplier pipe. For each output record, each of the other input pipes will provide the same set of records as normal.

Type

Select:

Pull: pull pipes are the most common type in PhixFlow - they "pull" data from the input to the output. Pull pipes are shown as solid arrows on models.
Look-up: look-up pipes are used to enrich data. Typically, you will have one of more pull pipes to supply the base data for an output, and if needed one or more look-up pipes to enrich the base data with values from additional inputs. Look-up pipes are shown as dashed lines on models.
Push: data is "pushed" rather than "pulled" into the output stream. Push pipes are most commonly used when sending data from streams to exporters (File Exporters, Database Exporters, HTTP Exporters). Push pipes are shown as dotted lines on models.

Data to Read

Select the type of input data to use.

Latest: supply data from the current run (the latest stream set). This is the mostly commonly used option.
Previous: supply data from the previous run (the previous stream set). This is used when you are comparing data for the current run with data from the previous run, for example, today's data with yesterday's.
All: supply data from all runs (all stream sets).
All Previous: supply data from all runs except the current run (all stream sets except the latest stream set).
Same Run: this option should only be used where the input and output streams are set to Period: Transactional. The pipe will only collect data from inputs in the same analysis run. This configuration support several analysis runs going on at the same time without interfering with each other.
Custom: select this option to display additional settings, described in the Custom Data to Read section, below. We recommend that you only use the custom settings when directed to by PhixFlow consultants or support.

Read Future Data

Use this option to exclude or include input streams sets that have future dates relative to the stream set you are generating. For details about how future stream sets occur, see Managing Future Stream Sets, above.

Insert excerpt

	_check_box_untick
	_check_box_untick
nopanel	true

to exclude future stream sets from this analysis run. This is the default.

Insert excerpt

	_check_box_ticked
	_check_box_ticked
nopanel	true

to include future stream sets in this analysis run. For example, for a stream with Period: Transactional , you will want to include new streams sets that are being added to the input stream after your analysis run starts.

Input The name of the object at the start of the pipe.

Output The name of the stream at the end of the pipe.

Anchor
future
future
Managing Future Stream Sets

In some circumstances the input stream may have stream sets that have dates in the future relative to the stream set being generated for the output stream. This may happen, for example, if:

you roll-back some stream sets on the output stream
but do not roll-back the corresponding stream sets on the input stream
and then request that the output stream is brought up to date.

Some of the stream sets on the input stream will have dates in the future relative to some of the stream sets you are rebuilding.

By default, the Read Future Data check box is not ticked. This means pipes ignore any stream sets with dates in the future relative to the stream set you are generating. You want to ignore future stream sets when you rebuild an old stream set, because you want the pipe to retrieve the same data on the rerun as it retrieved when the stream set was first built.

When you run analysis on a stream with a transactional period, it is possible that as your analysis is still running, a different run can start and complete. This run can generate additional stream sets on the input stream with a future data relative to the date of the stream set you are generating. For transactional input streams, you want the pipe to use these future streams. To do this, tick the Read Future Data check box.

Filter

Filters are made up of a set of clauses; each clause in turn contains a number of conditions. These conditions must be satisfied for data to be passed through the pipe.

Tip
When the pipe Type is Lookup, the filter controls which data will be cached in memory.

Field

Description

Cache Size

Anchor

	pipeCacheSize
	pipeCacheSize

Available when Type is Lookup.

For lookup pipes, PhixFlow uses the pipe cache when it looks-up data from streams or database collectors. For efficiency, the records are cached (stored temporarily in memory) so that if the same set of records need to be looked up again they are readily available without going back to the database.

Enter a number to set a limit on the data cache size available for the pipe. You need to estimate the largest number of records that the lookup pipe will return on a single read. Check whether PhixFlow is looking up:

many records
The pipe does a single lookup onto a stream or database table to get a large number of records in one go, for example 10,000 records.
few records may times
The pipe does many lookups, getting a small number of records for each lookup, for example, 10 records at a time. In this case, PhixFlow is usually using a key value, such as an account number, to get the data. The key value is:
- for a stream - the attribute used to filter the pipe
- for a database collector - a condition in the database query. For example:

Code Block
WHERE AccountNumber = _out.AccountNum

If you do not set a limit for the cache, PhixFlow uses the system default set in System Configuration → System Tuning → Maximum Pipe Cache Size.

Expand

title	Warnings and Errors

In the log for an analysis run, which is available in the system console, PhixFlow reports warnings when a single read returns:

over 90% of the specified cache size
100% or more of the cache size.

PhixFlow reports an error and stops the analysis run when:

a single read returns 100% or more of the cache size
and the System Configuration → Enforce Cache Size limit flag is ticked.

Code Block

title	Error Message: Cache Size Limit Exceeded

The Pipe "stream_name.lookup_pipe_name" cache is 100% full (the cache size is 10).

Expand

title	More cache details

Every time the lookup pipe is referenced, PhixFlow calculates the values of all of the variable elements of the query or pipe filter, and checks if it already has a set of data in the cache retrieved using this set of variable values. If so the data is immediately returned from the cache. Otherwise, a new set of data is read from the stream of collector. If adding the new records to the cache would cause it to exceed the maximum cache size, previously cached results are removed to make enough room for the new results.

Pipe View

Use this option to look up data from attributes that are present in a stream view on the input stream.

Select a stream view from the list. If the input stream has no views, the list will be empty.

Note
Sorting or filtering of records must be set directly on the pipe. It is not inherited from the pipe view.

Use the pipe view to limit the attributes that the pipe reads when a stream has lots of attributes containing many data records but you only need data from a few attributes. Only the data for the attributes in the stream view are sent to the output stream.

Pipe views are very useful:

during lookups a pipe loads and caches all of the attributes from the stream. This can use a lot of memory, especially when there are many records.
during file export all data records from all attributes are exported.

To set up a pipe view:

Create a new view on the source stream. In the view, only add the attributes you need.
In the pipe Pipe View option, select the pipe view.
Run analysis. PhixFlow only looks-up or exports data from the attributes specified on the view.

Expand

title	Using pipe views with file exporters

If a File Exporter is configured to export to Excel or to HTML:

and it has an Excel template, PhixFlow uses this template and does not use any template specified in the pipe view.
but it does not have an Excel template, PhixFlow will use the Excel template specified in the pipe view (if there is one)
and the pipe view is a chart view, PhixFlow will export a PNG picture of the chart.

Include History Records

Insert excerpt

	_check_box_untick
	_check_box_untick
nopanel	true

to filter out superseded records.

Insert excerpt

	_check_box_ticked
	_check_box_ticked
nopanel	true

to include superseded records.

Condition

Select one of the options

Where ALL...
Where ANY...

To add more conditions, hover your mouse pointer over this field to display the button and add another condition to your filter.

Clause

Select an option from the list. PhixFlow adds more fields where you can:

select how the filter matches (for example, equals, contains, is null)
enter a string that the filter uses to match the data. The string can be an expression or a literal string.

Filter Icons

Hover your mouse pointer over conditions or clauses to display:

Insert excerpt
_add
_add
nopanel true
another condition or clause to the filter
Insert excerpt
_delete
_delete
nopanel true
the selected clause or condition from the filter.
Insert excerpt
_filter_literal
_filter_literal
nopanel true
indicates the value entered is a literal value. Click this icon to treat the value as an expression.
Insert excerpt
_filter_expression
_filter_expression
nopanel true
indicates the value entered is an expression. Click this icon to treat the value to a literal string.
Note: ["123", "234", "345"] looks like a literal value but it can be evaluated as an expression.
Image AddedOpen the expression in a larger editor.

Cache Extraction Filter

A cache extraction filter allows you to further filter the data retrieved by a pipe. These are not commonly used, but are sometimes helpful when either:

Optimising performance on a lookup pipe when for a set of records, the record you require from the lookup depends on non-key data, e.g. the date
When getting data from a pull pipe when the filter requires that you compare one value in each record with another; this is not possible within a standard filter.

For case 1, when using a lookup pipe, data retrieved is stored in a cache. See cache size for details. The cache extraction filter allows you, as you are processing a set of output records, to use different cached entries from the lookup for each of the records are you are processing. This is very fast compared to looking up from the source (i.e. going back to an external DB table or even another PhixFlow stream) for each output record.

E.g. you want to look up the credit rating for a customer for a set of transactions - in the output, each transaction is represented by a single output record. You create an indexed lookup pipe using CustNo as the key for the index. This means that for each new CustNo you encounter in the data, all the credit rating entries for that CustNo would be retrieved by the pipe and placed into the cache. The credit rating for each customer is fully historied, so you get a number of entries for each CustNo. To get the relevant lookup entry for each output report (each transaction), you need to compare the transaction date of the output record to the dates of credit rating entries in the cache. So to extract the relevant record, you include a cache extraction filter in the form:

Code Block
StartDate >= _out.TransDate && (EndDate <= _out.TransDate \|\| EndDate == _NULL)

Cache extraction filters are entered free hand.

The attribute names referenced must exist in a stream. This means that the each attribute must be one of:

an attribute in a source stream, if you are reading from a stream
if you are reading from an external database table, one of the fields returned by the database collector AND an attribute in the output stream. This means to use an attribute with the source as a database collector, there must be an attribute of matching name in the output stream
an attribute in the destination stream, in which case you will refer to it using the format _out.AttributeName

Filter Examples

Filter on Current User

Sometimes when running analysis you want to select, from the source, only records belonging to the currently logged in user. To set a filter where, say, an attribute in the source Owner equals the current logged in user, add a condition to the filter like this:

Owner Equals _user.name fx

Enter a list of values for an "Is In" or "Is Not In" filter

If you want to based on a list of values, use the Is in or Is not in comparators, then type the list of values into the comparison field as a comma separated list like this:

Country Is in England, France, Germany ABC

In this case you must NOT click the ABC icon to convert the value to an fx, because this will indicate that the value is a formula; it must be left as a literal value. If you do click the ABC icon, then the value must be entered like this:

Country Is in ["England","France","Germany"] fx

Sort/Group

For lookup pipes this section is called Order/Index.

Use this section to group and sort data as it comes through the pipe. This section has:

a toolbar with standard buttons
a grid that lists the attributes that you want to sort or use to group
below the grid are the following options:

Field

Description

Maximum Number of records per Group

Enter an upper limit for grouped records.

When collating the input records into groups, PhixFlow uses the specified sort order. When it has added the maximum number of records, any more records for the group are ignored.

This can be useful if you want the most recent record for an attribute that has many records.

Tick the Group check box for the attribute you want to use for grouping.
On an appropriate date attribute, apply a (Z-A) sort order.
Set the Maximum Number of Records to 1.

Index Type

Available when Type is Look-up.

Look-up pipes can be configured for fast "indexed" access to cached data collected from external tables, files or from other streams. Indexed access is controlled through configuring a pipe with an index and setting index expressions on grouping attributes. If the Type field on the Pipe is set to 'Look-up' then the field "Index Type" becomes available. This can have the value "None" meaning that there are no index keys or "Exact Match", "Best Match" or "Near Match" as described below:

Exact Match: The pipe retrieves data from its cache based on an exact match look-up with the values provided after evaluating the index expressions on the "Group By" attributes.
Best Match: The pipe retrieves data from its cache based on a "Best Match" look-up after evaluating the index expressions on the "Group By" attributes.
Note: The last Group By Attribute with a key expression is used for the best match lookup. The index keys on any Group By attributes with a lower sequence number are used as an initial "Exact Match" to find the set of data on which to do the "Best Match". The "Best Match" is defined as the longest key value which matches the evaluated index expression.
Near Match: The pipe will retrieve data from its cache based on a "Near Match" look-up after evaluating the index expressions on the "Group By" attributes.
Note: the last Group By Attribute with a key expression is used for the near match lookup. The index keys on any Group By attributes with a lower sequence number are used as an initial "Exact Match" to find the set of data on which to do the "Best Match". When "Near Match" is selected, an additional field appears where you can enter an expression which should evaluate to a number representing the allowed number of edits (e.g. deletions, insertions, substitutions and transpositions) which can be made when comparing the result of the index expression to the index key in order to achieve a match. For example if the index key is "Smyhte" and the result of the index expression is "Smith" this would still be a match providing that the allowed number of edits is 3 or more (i.e. substitute the 'i' for a 'y', transpose the 't' and the 'h' and then insert an 'e' at the end).

Using the Sort/Group Grid

To add a stream attribute to the list:

click
Insert excerpt
_attributes_show
_attributes_show
nopanel true
to open the list of attributes in the input stream
drag a stream attribute into the grid.

To remove an attribute, click

Insert excerpt

	_delete
	_delete
nopanel	true

in the toolbar.

To set the sort or group properties for an attribute, double-click its name in the grid. If you want to create a new attribute that is not present in the input stream, in the section toolbar, click

Insert excerpt

	_add
	_add
nopanel	true

. PhixFlow opens the attribute's sort properties:

Field

Description

Attribute

For input stream attributes, PhixFlow displays the attribute name. (Read-only)

For a new attribute, enter a name.

Order

Enter the number for the order the attribute appears in the grid and the order in which it is processed. Other attributes are renumbered.

Direction

Select the sort order

(A-Z) to sort data records in ascending order, e.g. A to Z, 1 to 9, earliest to latest date.
(Z-A) to sort data records in descending alpha-numeric order, e.g. Z to A, 9 to 1, latest to earliest date.

Group

Insert excerpt

	_check_box_untick
	_check_box_untick
nopanel	true

by default, data is not grouped.

Insert excerpt

	_check_box_ticked
	_check_box_ticked
nopanel	true

to group data records by the value in this attribute.

If this attribute is part of the candidate key set, you must tick the Group check box. Otherwise, the attributes will be used only to sort the data in the candidate set.

Index Expression

This field is available for lookup pipes with an Index Type option selected.

Look-up pipes can be configured for fast "indexed" access to cached data. This data is collected from external tables, files or from other streams. Indexed access is controlled through configuring a pipe with an index and setting index expressions on "Group By" attributes here.

Audit Summary

See Common Properties.

Tip
In some cases, you may have a pipe connected to a database collector, which pulls data from an external database table. In these cases, the fields in the database must have matching attribute names in the output stream. You can refer to it using the format `_out.AttributeName`

Aggregate Attributes

Use this section to define the properties of data that you want to combine as it comes through the pipe.

Note

You cannot aggregate data from attributes if the pipe's input is from:

If you need to aggregate data from a database collector, you can use an SQL query.

This section has:

a toolbar with standard buttons
a grid that lists the attributes that you want to aggregate.

Using the Aggregate Attributes Grid

To add a stream attribute to the list:

click
Insert excerpt
_attributes_show
_attributes_show
nopanel true
to open the list of attributes in the input stream
drag a stream attribute into the grid.

To remove an attribute, click

Insert excerpt

	_delete
	_delete
nopanel	true

in the toolbar.

To set the properties for an aggregate attribute, double-click its name in the grid. PhixFlow opens the attribute's sort properties:

Field	Description
Stream Function	Select an function to combine stream data records. Average Count Distinct Maximum Minimum Sum See Aggregate Function for details. Make sure the function matches the data in the attribute. For example, you cannot Sum text.
Attribute	Select the stream attribute aggregated from the list of attributes in the input stream. PhixFlow does not use the value in this field if the Aggregate Function is Count.
Name	Enter the name for the aggregated attribute. This can be the same as the original attribute.
Order	The order of the aggregate attribute in the output stream.

Advanced

The following options are available when the pipe Type is Push or Pull. Allow Incomplete Stream Sets is also available when the pipe Type is Lookup.

Field

Description

Data Expected

Insert excerpt

	_check_box_untick
	_check_box_untick
nopanel	true

means the pipe may receive no data from its input during an analysis run.

Insert excerpt

	_check_box_ticked
	_check_box_ticked
nopanel	true

means PhixFlow reports an error if the pipe receives no data from the input datasource, collector or stream during an analysis run.

Allow Incomplete Stream Sets

Insert excerpt

	_check_box_untick
	_check_box_untick
nopanel	true

to complete a stream set before passing the data to the output stream. During the analysis run, PhixFlow pulls data into the input stream until the stream set is complete. If it cannot complete the stream set, PhixFlow reports an error message.

PhixFlow cannot complete a stream set if:

either the input stream is set to be
Insert excerpt
_static
_static
nopanel true
or all of the pipes reading from the stream are static.

Insert excerpt

	_check_box_ticked
	_check_box_ticked
nopanel	true

PhixFlow ignores incomplete stream sets in a static input stream and does not report an error.

Tip
You must tick this check box on all the pipes that will read from a static (or effectively static) input stream in the analysis run. PhixFlow will report an error if there is any pipe trying to complete the stream set during the analysis run.

Pipes that are not used in the analysis run do not try to complete a stream set, so will not report an error. (Unused pipes can occur if they lead to streams on branches of the model that are not being run.)

Buffer Size

Enter a number for the buffer size used to perform the stream calculation. If a large amount of data is being processed, then setting a large buffer size will give better performance.

Max Records To Read

Enter a number for the maximum number of records that should be read down this pipe. The pipe may read more than this number of records if it is configured to carry out multiple reads simultaneously. For example:

a pipe can be connected to a File Collector that reads multiple files simultaneously
the pipe strategy is Directed with multiple workers.

Strategy

Select an option to specify how this pipe should be implemented. See the section on Directed Merge Strategy

Default: this is the way pipes usually works
Directed: when reading from a large data set; see Using Directed Pipes to Read from Large Data Sets

Max Workers

This field is available when Strategy is Directed

Enter the maximum number of concurrent worker tasks. When no value is specified, this defaults to 1.

Worker Size

This field is available when Strategy is Directed

Enter the number of key values to read for a single worker task, which runs a single select statement.

When no value is specified, this defaults to 1000. This is the maximum value that can be used when reading from an Oracle database.

Log Traffic

Insert excerpt

	_log_traffic2
	_log_traffic2
nopanel	true

Pipe Logging: when ticked, PhixFlow always logs the number of records returned by this pipe, whatever is set here.

Insert excerpt

	_log_traffic1
	_log_traffic1
nopanel	true

Custom Data to Read
Anchor
custom
custom

The following properties are available in Basic Settings when you set Data To Read to Custom.

Field

Description

Only collect from same run

Every time the analysis engine runs, all of the stream sets that are created by all of the streams affected by that analysis run are given the same Run ID.

Insert excerpt

	_check_box_untick
	_check_box_untick
nopanel	true

so the pipe can collect stream sets with different Run IDs.

Insert excerpt

	_check_box_ticked
	_check_box_ticked
nopanel	true

so that the pipe will only collect stream sets from the input stream that have the same Run ID as the stream set currently being created by the output stream. You should only tick this check box if both the input and output streams have Period set to Transactional.

From Offset

Enter the offset applied to the start of the collection period, relative to the period in the output stream that requires populating.

To Offset

Enter the offset applied to the end of the collection period, relative to the period in the output stream that requires populating.

Max Stream Sets

Enter the number of stream sets to be retrieved from the input stream.

For a push pipe with positive offsets. enter the maximum number of stream sets that can be created i.e. the maximum number of cycles this pipe can initiate.

Historied

Insert excerpt

	_check_box_untick
	_check_box_untick
nopanel	true

so that all data will be collected from the input stream, regardless of period. In this case, any From Offset or To Offset values determine whether the required data periods in the input stream exist before the stream calculation can be carried out.

Insert excerpt

	_check_box_ticked
	_check_box_ticked
nopanel	true

so that the pipe will collect data from the input stream by period.

For example, if:

the from and to offsets are both 0.0
and the output stream requires stream generation for the period 17/10/07 - 18/10/07

the pipe reads data from the input stream for the period 17/10/07 - 18/10/07.

Versions Compared

Old Version 4

New Version Current

Key

Forms: Pipe

Form: Pipe Details

During Drill Down

During Look Ups

During File Export

Form Icons

See Also

Overview

Basic Settings

Anchor
future
future
Managing Future Stream Sets

Filter

Filter Examples

Filter on Current User

Enter a list of values for an "Is In" or "Is Not In" filter

Sort/Group

Using the Sort/Group Grid

Using the Aggregate Attributes Grid

Advanced

Custom Data to Read
Anchor
custom
custom

Page Comparison

Versions Compared

Old Version 4

New Version Current

Key

Forms: Pipe

Form: Pipe Details

During Drill Down

During Look Ups

During File Export

Form Icons

See Also

Basic Settings

AnchorfuturefutureManaging Future Stream Sets

Filter

Filter Examples

Filter on Current User

Enter a list of values for an "Is In" or "Is Not In" filter

Sort/Group

Using the Sort/Group Grid

Using the Aggregate Attributes Grid

Advanced

Custom Data to Read Anchorcustomcustom

Anchor
future
future
Managing Future Stream Sets

Custom Data to Read
Anchor
custom
custom