Page Comparison

If ticked, when multiple Streams are being merged then there must be an input record from this Pipe for an output record to be generated by the output Stream.

If this is a push pipe with positive offsets and this flag is ticked then the notification to create another stream set will only be pushed along the pipe if the last stream set created contains at least one record.

The Execution Strategy determines how this pipe should be implemented.

Where this pipe is a push or pull pipe into a Merge Stream, the Default Execution Strategy is to select all stream items from the input Stream sorted by the Group By attributes, then to read items from all input pipes simultaneously, constructing candidate sets from items with matching key values.

Where the Directed Execution Strategy is applied to a pipe (the pipe must not be mandatory Mandatory), the other pipes with the Default Strategy operate as above, each being sorted then merged to generate a sequence of candidate sets; the Directed pipe then runs worker tasks to select the additional items by matching key value. These selects are batched up so that each worker reads items for many key values in a single select (see Worker Size), and many workers are run in parallel (see Max Workers).

In general, the Directed strategy should only be used where

• the number of items in the source Stream is so large that the sorting phase of the Default strategy takes too long, or
• only a small subset of the items are needed (the majority being discarded because they have key values that don't match the key values read on one of the other mandatory pipes).

Changing the Execution Strategy will make the Merge faster or slower depending on the input data and the details of the input and output Streams, but will not change the business logic of the Merge (i.e. which input items are grouped into candidate sets).

If the input to the pipe is a Collector, the list of key values is made available as _keyList. This will typically be used with an in clause in the Collector query. For example:

select * from customer where account_num in ({_keyList}) order by account_num

The maximum number of concurrent worker tasks.

If blank, this defaults to 1.

The number of key values to read for a single worker task (which runs a single select statement).

If blank, this defaults to 1000. This is the maximum value that can be used when reading from an Oracle database.

Whenever data is requested down a lookup pipe, PhixFlow remembers all of the variable elements of the query i.e. the variables within any pipe filter, if the pipe is connected to a stream or file collector, or the variables within the SQL if the pipe is attached to a database collector. Any data returned by the query is then cached using the set of current values for each of these variables as the key to the cache.

When a subequent request is made to the pipe, PhixFlow again gathers the values of all of the variable elements of the query and checks if it already has a set of data in the cache retrieved using this set of variable values. If so the data is immediately returned from the cache otherwise a new set of data is read from the database. If adding the new records to the cache would cause it to exceed the maximum cache size, previously cached results are removed.

A warning is generated when the number of cached records approaches the maximum, and if the number exceeds the maximum, the operation is aborted.

If no maximum cache size is specified the System Configuration Default Pipe Cache Size is used. If there are likely to be a great many database lookups with different sets of variable values then the cache may end up containing a lot of data and consuming a great deal of memory. The maximum cache size can be specified to overcome this by defining the maximum number of records that the cache may contain.

The offset applied to the start of the collection period, relative to the period in the output stream that requires populating.

The units are the period of the output stream, that is, if the output stream has a daily period, then setting from date offset = -1.0 means that the start of the collection period will be 1 day earlier than the start of the period in the output stream that is being calculated.

If this is a push pipe then a positive offset can be input. This will tell the stream to run again and generate another stream set.

The offset applied to the end of the collection period, relative to the period in the output stream that requires populating.

The units are the period of the output stream, that is, if the output stream has a daily period, then setting to date offset = -1.0 means that the end of the collection period will be 1 day earlier than the end of the period in the output stream that is being calculated.

If this is a push pipe then a positive offset can be input. This will tell the stream to run again and generate another stream set.

If you are running a Transactional Stream then it is possible that while your analysis run is taking place, other analysis runs which started after yours may have managed to complete before yours thereby generating additional Streamsets on the input Stream. These additional Streamsets will then have a future data relative to the date of the Streamset you are generating. By default PhixFlow will ignore input Streamsets that have a date in the future relative to the Streamset being generated.

However, for transactional streams it is possible to tell the pipe to include future Streamsets by ticking this box.

This field is only available if the input Stream is Transactional and the following fields are empty:

• Only collect from the same run
• Max Stream Sets (this may also be set to zero)
• Historied

During Drill Down

During Look Ups

During File Export