PhixFlow Help

Which Stream to Use

The following table outlines scenarios and the stream type to use. More complex scenarios include types of pipe, filter and ordering or indexing.

ScenarioStreamExample
You only have 1source and want 1 record per input record.
See When to Use a Calculate Stream
CalculateYou have a comma separated file that you want to load into PhixFlow.

You only have 1source, but you want to group the data and only pull back aggregated information for each group.
See When to Use an Aggregate Stream

AggregateYou want to find the earliest entry in a task list.
Combine data from 2 sources into 1 set of data. For each record in each data set, you get one record.
See Merging Two Data Sets
CalculateYou have a set of customers stored in one system. You have a set of customers in another system. There are no overlaps. You want all your customers in one list.
Combining 2 sets of data that are a similar size and have a common key. For each pair of matching records from the data sets, a single record is produced in the output.
See Merging Similar Data Sets
MergeComparing a stream of thousands of invoice totals with a stream of thousands of payments for each customer.
Finding records with the same key in a large stream for a large stream of data. For each pair of matching records from the data sets, a single record is produced in the output.
See Deduplicating Similar Data Sets
Merge
with directed pipe
Finding account details for 1 million records in a reference list of all (~20m) accounts.
Combining a large stream with data from a small stream, where the values of the small stream will be repeated throughout the result. For each pair of matching records from the data sets, a single record is produced in the output.
See Enriching Data with Data From Another Set
Calculate stream
with lookup pipe
with order/index set
Find the description for each code in a stream of thousands from a stream containing mapping data. There are only ~100 possible codes.
Combining a large stream with data from a small stream, where values in the small stream will only used once in the result. For each pair of matching records from the data sets, a single record is produced in the output.
See Combining Data Using a Lookup Pipe
Calculate stream
with lookup pipe with filter
You have a stream containing all attendees of an upcoming football match and a small stream of people who are banned from attending matches. 
Combining a large stream with data from a small stream, where the small stream the same rows are repeated throughout the result, but the filter values change slightly. For each pair of matching records from the data sets, a single record is produced in the output.
Combining Data Using a Cache Extraction Filter Lookup Pipe
Calculate stream
with lookup pipe
with cache extraction filter
You have a price list for 4 different products with different prices between different dates.
You want to look back at a previous record within a group in a stream, or create a cumulative total per group. You get the same number of records as you put in.
See Grouping and Referencing Data Using Calculate By Set Stream
Calculate by setFor a given account, you want to find the difference between each consecutive debit/credit to the account.

Please let us know if we could improve this page feedback@phixflow.com