Page Comparison

Overview

A pipe is a connector that links two elements in a PhixFlow model and sends data from the input to the output. Pipes allows you to control which attributes and which records from the input are delivered by to the output, although in most cases - with minimal configuration - you will get all columns and the records from the current run.

...

Country Is in ["England","France","Germany"] fx

Sort/Group

Use this section to group and sort data as it comes through the pipe. for lookup pipes, this section is called Order/Index. This section has:

a toolbar with standard buttons
a grid that lists the attributes that you want to sort or use to group
below the grid are the following options:

...

Note

You cannot aggregate data from attributes if the pipe's input is from:

If you need to aggregate data from a database collector, you can use an SQL query.

This section has:

a toolbar with standard buttons
a grid that lists the attributes that you want to aggregate.

...

If a single read brings back over 90% of the specified cache size, a warning message will be logged to the console.

If a single read brings back 100% or more of the cache size, a second warning message will be generated. If the Enforce Cache Size limit flag is ticked in System Configuration, instead of a warning an error will be generated, and the analysis run will stop completely

Check whether PhixFlow is looking up:

many records
The pipe does a single lookup onto a stream or database table to get a large number of records in one go, for example 10,000 records.
few records may times
The pipe does many lookups, getting a small number of records for each lookup, for example, 10 records at a time. In this case, PhixFlow is usually using a key value, such as an account number, to get the data. The key value is:
- for a stream - the attribute used to filter the pipe
- for a database collector - a condition in the database query. For example:

Field

Description

Data Expected

This checkbox is available when the pipe Type is Push or Pull.

Insert excerpt

	_check_box_untick
	_check_box_untick
nopanel	true

means the pipe may receive no data from its input during an analysis run.

Insert excerpt

	_check_box_ticked
	_check_box_ticked
nopanel	true

means PhixFlow reports an error if the pipe receives no data from the input datasource, collector or stream during an analysis run.

Allow Incomplete Stream Sets

Insert excerpt

	_check_box_untick
	_check_box_untick
nopanel	true

to complete a stream set before passing the data to the output stream. During the analysis run, PhixFlow pulls data into the input stream until the stream set is complete. If it cannot complete the stream set, PhixFlow reports an error message.

PhixFlow cannot complete a stream set if:

either the input stream is set to be
Insert excerpt
_static
_static
nopanel true
or all of the pipes reading from the stream are static.

Insert excerpt

	_check_box_ticked
	_check_box_ticked
nopanel	true

PhixFlow ignores incomplete stream sets in a static input stream and does not report an error.

Tip
You must tick this checkbox on all the pipes that will read from a static (or effectively static) input stream in the analysis run. PhixFlow will report an error if there is any pipe trying to complete the stream set during the analysis run.

Pipes that are not used in the analysis run do not try to complete a stream set, so will not report an error. (Unused pipes can occur if they lead to streams on branches of the model that are not being run.)

Cache Size

Anchor

	pipeCacheSize
	pipeCacheSize

PhixFlow This option is useful for lookup pipes. PhixFlow uses the pipe cache when it looks-up data from streams or database collectors. For efficiency, the records are cached (stored temporarily in memory) so that if the same set of records need to be looked up again they are readily available without going back to the database.

Enter a number to set a limit on the data cache size available for the pipe. You need to estimate the largest number of records that the lookup pipe will return on a single read.

If you do not set a limit for the cache, PhixFlow uses the system default set in System Configuration → System Tuning → Maximum Pipe Cache Size. You may want to set a cache size on individual pipes if: .......PhixFlow may need to:

either lookup many records: the pipe does a single lookup onto a stream or database table to get a large number of records in one go (e.g. 10,000 records)
or lookup few records may times: the pipe does many lookups, getting a small number of records for each lookup (e.g. 10 records at a time).

either lookup many records: the pipe does a single lookup onto a stream or database table to get a large number of records in one go (e.g. 10,000 records)
or lookup few records may times: the pipe does many lookups, getting a small number of records for each lookup (e.g. 10 records at a time).

Lookup for a few records may times

In this case, PhixFlow is usually using a key value, such as an account number, to get the data. The key value is

for a stream - the attribute used to filter the pipe
for a database collector - a condition in the database query. For example:

Code Block
WHERE AccountNumber = _out.AccountNum

This field allows you to set a limit on the size of the cache. Setting a limit is important because if you do not, the cache can become very large and consume a lot of memory, which can lead to a slow down in both your tasks and those of other users of PhixFlow.

To set the cache size, try to estimate the largest number of records that the lookup pipe will return on a single read.

If you do not set a limit, it will default to the system-wide default, specified in the Maximum Pipe Cache Size in the System Tuning tab of the System Configuration.

Panel

bgColor	#e6f0ff
titleBGColor	#99c2ff
title	Cache warnings and errors

Code Block
WHERE AccountNumber = _out.AccountNum

If you do not set a limit for the cache, PhixFlow uses the system default set in System Configuration → System Tuning → Maximum Pipe Cache Size.

Technical breakout

Expand

title	Warnings and Errors

In the log for an analysis run, which is available in the system console, PhixFlow reports warnings when a single read returns:

over 90% of the specified cache size
100% or more of the cache size.

PhixFlow reports an error and stops the analysis run when:

a single read returns 100% or more of the cache size
and the System Configuration → Enforce Cache Size limit flag is ticked.

Code Block

title	Error Message: Cache Size Limit Exceeded

The Pipe "stream_name.lookup_pipe_name" cache is 100% full (the cache size is 10).

Panel

bgColor

#e6f0ff

titleBGColor

#99c2ff

title

10).

Expand

title	More cache details

Every time the lookup pipe is referenced

, PhixFlow calculates the values of all of the variable elements of the query or pipe filter, and checks if it already has a set of data in the cache retrieved using this set of variable values. If so the data is immediately returned from the cache. Otherwise, a new set of data is read from the stream of collector. If adding the new records to the cache would cause it to exceed the maximum cache size, previously cached results are removed to make enough room for the new results.

If there are a large number of records to cache, it can use a lot of memory. This means PhixFlow will run more slowly for all users.

To set the cache size, try to estimate the largest number of records that the lookup pipe will return on a single read.

If you do not set a limit, it will default to the system-wide default, specified in the Maximum Pipe Cache Size in the System Tuning tab of the System Configuration.

PhixFlow may need to:

Buffer SizeThe

, PhixFlow calculates the values of all of the variable elements of the query or pipe filter, and checks if it already has a set of data in the cache retrieved using this set of variable values. If so the data is immediately returned from the cache. Otherwise, a new set of data is read from the stream of collector. If adding the new records to the cache would cause it to exceed the maximum cache size, previously cached results are removed to make enough room for the new results.

Buffer SizeEnter a number for the buffer size used to perform the stream calculation. If a large amount of data is being processed, then setting a large buffer size will give better performance.Pipe ViewThe pipe view is used to limit which fields are retrieved down the pipe and in what order, and in some circumstances how each field is to be formatted. You can select from any of the views that have been configured on the source stream. Please note that any sorting or filtering of records will have to be applied directly on the pipe, and will not be

Use this option to look up data from attributes that are present in a stream view on the input stream

Select a stream view from the list. If the input stream has no views, the list will be empty.

Note
Sorting or filtering of records must be set directly on the pipe. It is not inherited from the pipe view.

The pipe view is used in two contexts.

During lookups

Pipe views can also be used on lookup pipes to limit the fields that are returned by the lookup request. This is most useful in the scenario where the you want to read and cache data on a lookup pipe from a stream that has lots of attributes but where only a small number of attributes are actually required. You can simply create a new view on the source stream listing only the attributes needed, then specify it as the pipe view on the lookup pipe. Only those attributes specified on the view will then be loaded.
Without a pipe view, the pipe will load and cache

Use the pipe view to limit the attributes that the pipe reads when:

a stream has lots of attributes containing many data records
but you only need data from a few attributes.

Only the data for the attributes in the stream view are sent to the output stream.

Pipe views are very useful:

during lookups a pipe loads and caches all of the attributes from the stream

which may consume a significant amount of free memory if there are a large number of records.

During File Export

When sending data to a file exporter only those fields specified on the pipe view will be exported. If no pipe view is supplied then all fields will be exported.
If the file exporter is

. This can use a lot of memory, especially when there are many records.

during file export all data records from all attributes are exported.

To set up a pipe view:

Create a new view on the source stream. In the view, only add the attributes you need.
In the pipe Pipe View option, select the pipe view.
Run analysis. PhixFlow only looks-up or exports data from the attributes specified on the view.

Expand

title	Using pipe views with file exporters

If a File Exporter is configured to export to Excel or to HTML:

and it has an Excel template,

and no Excel template is specified on the exporter, then the pipe view will be checked to see if an Excel template has been specified there. This template will then be used as part of the export. See the description of file exporters for further details. If an Excel template has been specified on the exporter then this will override any template specified on the pipe view.
If the file exporter is configured to export to HTML

PhixFlow uses this template and does not use any template specified in the pipe view.
but it does not have an Excel template, PhixFlow will use the Excel template specified in the pipe view (if there is one)
and the pipe view is a chart view

then the output will be a

, PhixFlow will export a PNG picture of the chart

in PNG format

.

Max Records To Read

The

Enter a number for the maximum number of records that should be read down this pipe. The pipe may read more than this number of records if it is configured to carry out multiple reads simultaneously

e.g. if it is

. For example:

a pipe can be connected to a File Collector

which has been configured to read

that reads multiple files simultaneously

or if this

the pipe strategy is

"

Directed

" with

with multiple workers.

Strategy

The Execution Strategy determines Select an option to specify how this pipe should be implemented. See the section on Directed Merge Strategy

Default
Directed

Max Workers

This field is available when Strategy is Directed

The maximum number of concurrent worker tasks.

If blank, this defaults to 1.

Worker Size

This field is available when Strategy is Directed

The number of key values to read for a single worker task (which runs a single select statement).

If blank, this defaults to 1000. This is the maximum value that can be used when reading from an Oracle database.

Versions Compared

Old Version 70

New Version 71

Key

Overview

Sort/Group

During lookups

During File Export