• Push: data is 'pushed' rather than 'pulled' into the output stream. Every time data is written to the input stream, the pipe 'informs' the output stream and the Analysis Engine will attempt to run stream generation on the output stream. Push pipes are shown as blue lines on modelling screens.
• Pull: This type of pipe will pass data along as it is calculated which may be before the streamset is complete or any of the records have been committed to the database. This may improve performance over other pipes which wait until the input streams have committed their data to the database before reading the data back in again (which allows the database to apply the filtering and aggregation rules). Because PUSH pipes don't get their data from the database, but instead receive it direct from the input stream, any filters or aggregation specified on the pipe will be ignored. Pull pipes are shown as blue lines on modelling screens.
• Look-up: used to reference data, see lookup for more details. Look-up pipes are shown as dotted lines on modelling screens.

This field is used to determine which Streamsets to read from the input Stream. There are 4 options available:

• Latest: This will cause the pipe to read only the latest Streamset from the input Stream.
• Previous: This will cause the pipe to read the Streamset before the latest Streamset on the input Stream.
• All: This will cause the pipe to read all Streamsets from the input Stream. However, In some circumstances the input Stream may have Streamsets that have dates in the future relative to the Streamset the output Stream is creating. This may happen for example if you have rolled back a number of Streamsets on the output Stream but have not rolled back the corresponding Streamsets on the input Stream and have then requested that one of the rolled back Streamsets be rebuilt. Some of the Streamsets on the input Stream will then have dates in the future relative to the Streamset you are rebuilding.
  By default the pipe will ignore any Streamsets with dates in the future relative to the Streamset you are generating so that if you are rebuilding an old Streamset the pipe will retrieve the same data on the rerun as it retrieved when the Streamset was first built.
  Similarly if you are running a Transactional Stream then it is possible that while your analysis run is taking place, other analysis runs which started after yours may have managed to complete before yours thereby generating additional Streamsets on the input Stream with a future data relative to the date of the Streamset you are generating.
  For Transactional input Streams it is possible to tell the pipe not to ignore these future Streamsets by ticking the Read Future Data tickbox on the Advanced tab.
• Custom: If this option is selected then you need to specify which input Streamsets to read by configuring combinations of the following fields on the Advanced tab:
  • Read Future Data
  • Only collect from the same run
  • Max Stream Sets
  • Historied
  • From Data Offset
  • To Date Offset

Normally when a Pipe requests data from a non-static input Stream then that Stream will first attempt to bring itself up to date, generating new Streamsets as necessary, before supplying the data requests. However, if this field is ticked, the input Stream will not attempt to do this.

If this flag is not ticked then it is an indication to PhixFlow that the Stream is not ready to be used during any analysis runs and should be therefore be ignored.

If ticked, when multiple Streams are being merged then there must be an input record from this Pipe for an output record to be generated by the output Stream.

If this is a push pipe with positive offsets and this flag is ticked then the notification to create another stream set will only be pushed along the pipe if the last stream set created contains at least one record.

• Exact Match: The pipe will retrieve data from its cache based on an exact match look-up with the values provided after evaluating the index expressions on the "Group By" attributes.
• Best Match: The pipe will retrieve data from its cache based on a "Best Match" look-up after evaluating the index expressions on the "Group By" attributes. Note that the last Group By Attribute with a key expression is used for the best match lookup. The index keys on any Group By attributes with a lower sequence number are used as an initial "Exact Match" to find the set of data on which to do the "Best Match". The "Best Match" is defined as the longest key value which matches the evaluated index expression.
• Near Match: The pipe will retrieve data from its cache based on a "Near Match" look-up after evaluating the index expressions on the "Group By" attributes. Note that the last Group By Attribute with a key expression is used for the near match lookup. The index keys on any Group By attributes with a lower sequence number are used as an initial "Exact Match" to find the set of data on which to do the "Best Match". When "Near Match" is selected, an additional field appears where you can enter an expression which should evaluate to a number representing the allowed number of edits (e.g. deletions, insertions, substitutions and transpositions) which can be made when comparing the result of the index expression to the index key in order to achieve a match.
  For example if the index key is "Smyhte" and the result of the index expression is "Smith" this would still be a match providing that the allowed number of edits is 3 or more (i.e. substitute the 'i' for a 'y', transpose the 't' and the 'h' and then insert an 'e' at the end).

The Execution Strategy determines how this pipe should be implemented.

Where this pipe is a push or pull pipe into a Merge Stream, the Default Execution Strategy is to select all stream items from the input Stream sorted by the Group By attributes, then to read items from all input pipes simultaneously, constructing candidate sets from items with matching key values.

Where the Directed Execution Strategy is applied to a specific non-mandatory pipe, the other pipes with the Default Strategy operate as above, each being sorted then merged to generate a sequence of candidate sets; the Directed pipe then runs worker tasks to select the additional items by matching key value. These selects are batched up so that each worker reads items for many key values in a single select (see Worker Size), and many workers are run in parallel (see Max Workers).

In general, the Directed strategy should only be used where

• the number of items in the source Stream is so large that the sorting phase of the Default strategy takes too long, or
• only a small subset of the items are needed (the majority being discarded because they have key values that don't match the key values read on one of the other mandatory pipes).

Changing the Execution Strategy will make the Merge faster or slower depending on the input data and the details of the input and output Streams, but will not change the business logic of the Merge (i.e. which input items are grouped into candidate sets).

If the input to the pipe is a Collector, the list of key values is made available as _keyList. This will typically be used with an in clause in the Collector query. For example:

select * from customer where account_num in ({_keyList}) order by account_num

The maximum number of concurrent worker tasks.

If blank, this defaults to 1.

The number of key values to read for a single worker task (which runs a single select statement).

If blank, this defaults to 1000. This is the maximum value that can be used when reading from an Oracle database.

Whenever data is requested down a lookup pipe, PhixFlow remembers all of the variable elements of the query i.e. the variables within any pipe filter, if the pipe is connected to a stream or file collector, or the variables within the SQL if the pipe is attached to a database collector. Any data returned by the query is then cached using the set of current values for each of these variables as the key to the cache.

When a subequent request is made to the pipe, PhixFlow again gathers the values of all of the variable elements of the query and checks if it already has a set of data in the cache retrieved using this set of variable values. If so the data is immediately returned from the cache otherwise a new set of data is read from the database. If adding the new records to the cache would cause it to exceed the maximum cache size, previously cached results are removed.

A warning is generated when the number of cached records approaches the maximum, and if the number exceeds the maximum, the operation is aborted.

If no maximum cache size is specified the System Configuration Default Pipe Cache Size is used. If there are likely to be a great many database lookups with different sets of variable values then the cache may end up containing a lot of data and consuming a great deal of memory. The maximum cache size can be specified to overcome this by defining the maximum number of records that the cache may contain.

The offset applied to the start of the collection period, relative to the period in the output stream that requires populating.

The units are the period of the output stream, that is, if the output stream has a daily period, then setting from date offset = -1.0 means that the start of the collection period will be 1 day earlier than the start of the period in the output stream that is being calculated.

If this is a push pipe then a positive offset can be input. This will tell the stream to run again and generate another stream set.

The offset applied to the end of the collection period, relative to the period in the output stream that requires populating.

The units are the period of the output stream, that is, if the output stream has a daily period, then setting to date offset = -1.0 means that the end of the collection period will be 1 day earlier than the end of the period in the output stream that is being calculated.

If this is a push pipe then a positive offset can be input. This will tell the stream to run again and generate another stream set.

If you are running a Transactional Stream then it is possible that while your analysis run is taking place, other analysis runs which started after yours may have managed to complete before yours thereby generating additional Streamsets on the input Stream. These additional Streamsets will then have a future data relative to the date of the Streamset you are generating. By default PhixFlow will ignore input Streamsets that have a date in the future relative to the Streamset being generated.

However, for transactional streams it is possible to tell the pipe to include future Streamsets by ticking this box.

This field is only available if the input Stream is Transactional and the following fields are empty:

• Only collect from the same run
• Max Stream Sets (this may also be set to zero)
• Historied

During Drill Down

During Look Ups

During File Export