This page is for data modellers. It provides an introduction to streams and pipes.

Overview

In an analysis model, a data set is represented by a stream. A stream is a bit like an Excel spreadsheet, in that it contains a set of data with:

columns - these are the stream attributes
rows - these are the data records.

In PhixFlow you connect two streams with a pipe. A pipe sends data from the input stream to the output stream. Pipes also connect other types of modelling object, such as a datasource or file exporter. Usually a pipe with the default settings passes all attributes and records onto the next object. However, you can use the pipe properties to control which attributes and records from the input object you want to pass through.

When you run an analysis model, PhixFlow uses information in each object's properties to process the data. This means each stream and pipe can transform the data in the analysis model. With each analysis run, the data set in a stream can change, so PhixFlow keeps a snapshot for each run. The snapshot is called a stream set. If there is a problem in the analysis run, you can "undo" it by rolling back the run. PhixFlow reverts the data to the selected, previous stream set. You can also copy or move data from a stream set.

To look at the data in a stream you use a stream view. The default view shows data in a grid. You can also create different views such as graphs and charts. Stream view properties have lots of options to control which attributes are included in the view, and how to sort the records.

Sections on this page

Pages in this topic:

Which Stream to Use
Stream Set List
Copy or Move Stream Data
Using Directed Pipes to Read from Large Data Sets
Properties:
Rolling back stream set:
- Rollback
- Rollback or Repeat Cycles

Types of Stream

There are several types of stream:

Calculate Stream

Calculate streams are the most basic stream type in PhixFlow. An output record will be produced for each input record.

Merge Stream

Merge streams combine sets of input data. In each input pipe a grouping is defined, and an output record is produced for each key value combination that is produced by this grouping applied across all inputs.

Aggregate Stream

Aggregate streams aggregate input data. In the input pipe a grouping is defined, and an output record is produced for each key value combination that is produced by the grouping.

Simple aggregations are better performed using aggregate pipes.

Aggregate streams are functionally identical to merge streams, but by convention, when there is only one input, an Aggregate Stream is used - this displays as a on the model view. Often this helps to clarify the purpose of the stream in the model.

Cartesian Stream

Cartesian streams perform a cartesian join across all inputs. Although this can be useful in some cases, mostly it is easier and simpler to multiply output records with either an output multiplier - which can be configured for any stream type - or to use a multiplier pipe.

CalculateBySet Stream

Calculate by Set streams are like calculate streams in that an output record is produced for each input record. But in addition a grouping can be configured on the input pipe which allows, for each record processed, related rows to be included in calculations.

Publishing Streams

When you make changes to a stream's properties or its attributes, PhixFlow publishes the changes to the stream data tables in the PhixFlow database. This happens automatically in the background. Publishing many streams or streams with many attributes can take some time, and may slow performance.

If the stream properties are set incorrectly, PhixFlow will not be able to publish the stream data to the database. If this happens, the Console will report the publishing error. PhixFlow will also display an error message if you try to interact with the stream, for example to view its data or to run analysis. You must correct the stream properties, so that PhixFlow can retry publishing the stream.

During the publishing process, PhixFlow may create temporary tables in its database. These are kept for a period, then automatically removed when a system task runs. For information about:

the system task, see Using Tasks and Task Plans
configuring the period that temporary tables are kept; see System Configuration → Delete Temp Tables after Days.

To ensure that PhixFlow can publish data changes, its database must have enough space to hold a copy of the largest stream. For the different databases, the space needs to be in:

Oracle: temporary table space
SQL Server: temporary file group
Maria DB: the file system.

Understanding Streams and Pipes