PhixFlow Help

Skip to end of metadata
Go to start of metadata

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 13 Next »

Forms: Pipe

A pipe is a connector that links two elements in a PhixFlow model. A pipe joining a datasource to a data collector has no editable details.

Form: Pipe Details

The following fields are configured on the Details tab:

FieldDescription
NameThe name of the pipe. The name is automatically generated when the pipe is first created to be 'in' (or the equivalent word in the user's selected language). If the name 'in' has already been used then a number will be added to the name to make it unique, e.g. in2
InputThe name of the pipe. In particular, this name will be used in any attribute expressions defined for the stream, when referring to attributes of the input stream. Please note Pipes cannot have the same name as any of the PhixFlow Attribute Functions, otherwise Expressions, using them will not function correctly
OutputName of the output stream, that is, the stream at the end (the arrow end) of the pipe. This cannot be edited through the dialogue; if you wish to update this, you will need to go back into the model view, delete the existing pipe and create a new pipe between the two streams as needed.
TypeThere are 3 options available:
  • Push: data is 'pushed' rather than 'pulled' into the output stream. Every time data is written to the input stream, the pipe 'informs' the output stream and the Analysis Engine will attempt to run stream generation on the output stream. Push pipes are shown as blue lines on modelling screens.
  • Pull: This type of pipe will pass data along as it is calculated which may be before the streamset is complete or any of the records have been committed to the database. This may improve performance over other pipes which wait until the input streams have committed their data to the database before reading the data back in again (which allows the database to apply the filtering and aggregation rules). Because PUSH pipes don't get their data from the database, but instead receive it direct from the input stream, any filters or aggregation specified on the pipe will be ignored. Pull pipes are shown as blue lines on modelling screens.
  • Look-up: used to reference data, see lookup for more details. Look-up pipes are shown as dotted lines on modelling screens.
DataToRead

This field is used to determine which Streamsets to read from the input Stream. There are 4 options available:

  • Latest: This will cause the pipe to read only the latest Streamset from the input Stream.
  • Previous: This will cause the pipe to read the Streamset before the latest Streamset on the input Stream.
  • All: This will cause the pipe to read all Streamsets from the input Stream. However, In some circumstances the input Stream may have Streamsets that have dates in the future relative to the Streamset the output Stream is creating. This may happen for example if you have rolled back a number of Streamsets on the output Stream but have not rolled back the corresponding Streamsets on the input Stream and have then requested that one of the rolled back Streamsets be rebuilt. Some of the Streamsets on the input Stream will then have dates in the future relative to the Streamset you are rebuilding.
    By default the pipe will ignore any Streamsets with dates in the future relative to the Streamset you are generating so that if you are rebuilding an old Streamset the pipe will retrieve the same data on the rerun as it retrieved when the Streamset was first built.
    Similarly if you are running a Transactional Stream then it is possible that while your analysis run is taking place, other analysis runs which started after yours may have managed to complete before yours thereby generating additional Streamsets on the input Stream with a future data relative to the date of the Streamset you are generating.
    For Transactional input Streams it is possible to tell the pipe not to ignore these future Streamsets by ticking the Read Future Data tickbox on the Advanced tab.
  • Custom: If this option is selected then you need to specify which input Streamsets to read by configuring combinations of the following fields on the Advanced tab:
    • Read Future Data
    • Only collect from the same run
    • Max Stream Sets
    • Historied
    • From Data Offset
    • To Date Offset
Static

Normally when a Pipe requests data from a non-static input Stream then that Stream will first attempt to bring itself up to date, generating new Streamsets as necessary, before supplying the data requests. However, if this field is ticked, the input Stream will not attempt to do this.

Enabled

If this flag is not ticked then it is an indication to PhixFlow that the Stream is not ready to be used during any analysis runs and should be therefore be ignored.

The following fields are configured on the Advanced tab:

FieldDescription

Mandatory

If ticked, when multiple Streams are being merged then there must be an input record from this Pipe for an output record to be generated by the output Stream.

If this is a push pipe with positive offsets and this flag is ticked then the notification to create another stream set will only be pushed along the pipe if the last stream set created contains at least one record.

MultiplierWhen processing data, a Stream first constructs CandidateSets ready for processing from its (non-Multiplier) input Pipes. For each Multiplier Pipe, the Stream then multiplies each CandidateSet by creating a copy of the original CandidateSet for each row returned from the Multiplier Pipe. Note that each resulting CandidateSet contains the original CandidateSet plus one row from the Multiplier Pipe.
Execution Strategy

The Execution Strategy determines how this pipe should be implemented.

Where this pipe is a push or pull pipe into a Merge Stream, the Default Execution Strategy is to select all stream items from the input Stream sorted by the Group By attributes, then to read items from all input pipes simultaneously, constructing candidate sets from items with matching key values.

Where the Directed Execution Strategy is applied to a pipe (the pipe must not be mandatory), the other pipes with the Default Strategy operate as above, each being sorted then merged to generate a sequence of candidate sets; the Directed pipe then runs worker tasks to select the additional items by matching key value. These selects are batched up so that each worker reads items for many key values in a single select (see Worker Size), and many workers are run in parallel (see Max Workers).

In general, the Directed strategy should only be used where

  • the number of items in the source Stream is so large that the sorting phase of the Default strategy takes too long, or
  • only a small subset of the items are needed (the majority being discarded because they have key values that don't match the key values read on one of the other mandatory pipes).

 

Changing the Execution Strategy will make the Merge faster or slower depending on the input data and the details of the input and output Streams, but will not change the business logic of the Merge (i.e. which input items are grouped into candidate sets).

If the input to the pipe is a Collector, the list of key values is made available as _keyList. This will typically be used with an in clause in the Collector query. For example:

select * from customer where account_num in ({_keyList}) order by account_num

Max Workers

The maximum number of concurrent worker tasks.

If blank, this defaults to 1.

Worker Size

The number of key values to read for a single worker task (which runs a single select statement).

If blank, this defaults to 1000. This is the maximum value that can be used when reading from an Oracle database.

Cache Size

Whenever data is requested down a lookup pipe, PhixFlow remembers all of the variable elements of the query i.e. the variables within any pipe filter, if the pipe is connected to a stream or file collector, or the variables within the SQL if the pipe is attached to a database collector. Any data returned by the query is then cached using the set of current values for each of these variables as the key to the cache.

When a subequent request is made to the pipe, PhixFlow again gathers the values of all of the variable elements of the query and checks if it already has a set of data in the cache retrieved using this set of variable values. If so the data is immediately returned from the cache otherwise a new set of data is read from the database. If adding the new records to the cache would cause it to exceed the maximum cache size, previously cached results are removed.

A warning is generated when the number of cached records approaches the maximum, and if the number exceeds the maximum, the operation is aborted.

If no maximum cache size is specified the System Configuration Default Pipe Cache Size is used. If there are likely to be a great many database lookups with different sets of variable values then the cache may end up containing a lot of data and consuming a great deal of memory. The maximum cache size can be specified to overcome this by defining the maximum number of records that the cache may contain.

Buffer SizeThe buffer size used to perform the stream calculation. If a large amount of data is being processed, then setting a large buffer size will give better performance.
Allow Incomplete Stream SetsNormally, when a pipe tries to read from an input stream that contains an incomplete stream set, PhixFlow will attempt to complete the stream set before passing data down the pipe. However if the stream is static (i.e. the stream has its 'static' flag ticked) or is effectively static (i.e. all of the pipes reading from it in this analysis run are static) then, instead of completing the stream set, an error message is produced indicating that you cannot read from this stream because it contains an incomplete stream set.

If you do not want this error message to be produced when reading from static (or effectively static) streams, but would instead prefer PhixFlow to ignore the incomplete stream sets, then you must tick this box on all pipes that will read from the input stream in this analysis run. If there are multiple pipes that read from the input stream during this analysis run and even one of the pipes does not have this box ticked then you will not be allowed to read from the stream and the error message will be produced.

Pipes which are not used in the current analysis run (for example where they lead to streams on branches of the model which are not run by the current task plan) have no effect on whether or not the error message is produced.
Data ExpectedThis field is available when the Pipe Type is Push or Pull. This flag allows the user to specify that the pipe is expecting to receive data. If ticked but no data is received this is treated as an error.
Only collect from same runEvery time the analysis engine runs all of the Streamsets that are created by all of the Streams affected by that analysis run are given the same Run ID. If this flag is ticked then the pipe will only collect Streamsets from the input Stream that have the same Run ID as the Streamset currently being created by the outpu Stream.
Max Stream SetsIn almost all cases this specifies the number of stream sets to be retrieved from the input stream. However, if this is a push pipe with positive offsets this value indicates the maximum number of streamsets that can be created i.e. the maximum number of cycles this pipe can initiate.
HistoriedIf ticked, the pipe will collect data from the input stream by period. So if the from and to date offsets are both 0.0, and the output stream requires stream generation for the period 17/10/07 - 18/10/07, data will be collected from the input stream for the period 17/10/07 - 18/10/07. If not ticked, all data will be collected from the input stream, regardless of period. In this case, the offsets are still used to determine whether the required data periods in the input stream exist before the stream calculation can be carried out.
From Date Offset

The offset applied to the start of the collection period, relative to the period in the output stream that requires populating.

The units are the period of the output stream, that is, if the output stream has a daily period, then setting from date offset = -1.0 means that the start of the collection period will be 1 day earlier than the start of the period in the output stream that is being calculated.

If this is a push pipe then a positive offset can be input. This will tell the stream to run again and generate another stream set.

To Date Offset

The offset applied to the end of the collection period, relative to the period in the output stream that requires populating.

The units are the period of the output stream, that is, if the output stream has a daily period, then setting to date offset = -1.0 means that the end of the collection period will be 1 day earlier than the end of the period in the output stream that is being calculated.

If this is a push pipe then a positive offset can be input. This will tell the stream to run again and generate another stream set.

Read Future Data

If you are running a Transactional Stream then it is possible that while your analysis run is taking place, other analysis runs which started after yours may have managed to complete before yours thereby generating additional Streamsets on the input Stream. These additional Streamsets will then have a future data relative to the date of the Streamset you are generating. By default PhixFlow will ignore input Streamsets that have a date in the future relative to the Streamset being generated.

However, for transactional streams it is possible to tell the pipe to include future Streamsets by ticking this box.

This field is only available if the input Stream is Transactional and the following fields are empty:

  • Only collect from the same run
  • Max Stream Sets (this may also be set to zero)
  • Historied

 

Pipe ViewThe pipe view is used to control what data is retrieved down the pipe and in some circumstances how that data is to be formatted. You can select from any of the views that have been configured on the source stream.

The pipe view is used in three contexts.

During Drill Down

When drilling down from an alarm or stream item the pipe view is only used to determine which attributes from the source stream should be shown in the drill down display and in what order.

If a view is not specified, then all attributes are shown.

During Look Ups

Pipe views can also be used on lookup pipes to limit the fields that are returned by the lookup request. This is most useful in the scenario where the you want to read and cache data on a lookup pipe from a stream that has lots of attributes but where only a small number of attributes are actually required. You can simply create a new view on the source stream listing only the attributes needed, then specify it as the pipe view on the lookup pipe. Only those attributes specified on the view will then be loaded.

Without a pipe view, the pipe will load and cache all of the attributes from the stream which may consume a significant amount of free memory if there are a large number of records.

During File Export

When sending data to a file exporter only those fields specified on the pipe view will be exported. If no pipe view is supplied then all fields will be exported.

If the file exporter is configured to export to Excel or to HTML, and no Excel template is specified on the exporter, then the pipe view will be checked to see if an Excel template has been specified there. This template will then be used as part of the export. See the description of file exporters for further details. If an Excel template has been specified on the exporter then this will override any template specified on the pipe view.

If the file exporter is configured to export to HTML and the pipe view is a chart view then the output will be a picture of the chart in PNG format.
Pipe ExporterAn exporter can be selected from the set of Database Exporters configured for the input stream. This exporter can then be used from the Drill Down View. This feature is useful when PhixFlow is used to 'recommend' a set of updates. By configuring an alarm to be generated when a set of recommendations is made, the user can drill down through the alarm to see the list of recommendations and then hit the exporter icon to apply them.

Any filters applied on the pipe will be applied when the data is pushed to the pipe exporter, so it is possible that not all of the data in the grid will be exported - some records may be rejected by the filter.
Max Records To ReadThe maximum number of records that should be read down this pipe. The pipe may read more than this number of records if it is configured to carry out multiple reads simultaneously e.g. if it is connected to a File Collector which has been configured to read multiple files simultaneously or if this pipe strategy is "Directed" with multiple workers.

The following fields are configured through separate tabs on the form:

FieldDescription
FilterAllows the user to set up a filter on the pipe.
Sort/GroupSee the Order By Attribute form.
Aggregate AttributesSee the Aggregate Attribute form.

Form Icons

The form provides the standard form icons.

The form also provides the following icons on the Filter tab:

Adds a clause to the filter.

Deletes the selected clause or condition from the filter.

Adds a condition to a clause of the filter.

The form also provides the following icons on both the Sort/Group and Aggregate Attributes tabs:

Shows the list of attributes that can be added as sort/group or aggregate attributes.

Deletes the selected object from the list.

Adds an object to the list.

See Also

  • No labels