PhixFlow Help
Which Stream to Use
The following table outlines scenarios and the stream type to use. More complex scenarios include types of pipe, filter and ordering or indexing.
Scenario | Stream | Example |
---|---|---|
You only have 1source and want 1 record per input record. See When to Use a Calculate Stream | Calculate | You have a comma separated file that you want to load into PhixFlow. |
You only have 1source, but you want to group the data and only pull back aggregated information for each group. | Aggregate | You want to find the earliest entry in a task list. |
Combine data from 2 sources into 1 set of data. For each record in each data set, you get one record. See Merging Two Data Sets | Calculate | You have a set of customers stored in one system. You have a set of customers in another system. There are no overlaps. You want all your customers in one list. |
Combining 2 sets of data that are a similar size and have a common key. For each pair of matching records from the data sets, a single record is produced in the output. See Merging Similar Data Sets | Merge | Comparing a stream of thousands of invoice totals with a stream of thousands of payments for each customer. |
Finding records with the same key in a large stream for a large stream of data. For each pair of matching records from the data sets, a single record is produced in the output. See Deduplicating Similar Data Sets | Merge with directed pipe | Finding account details for 1 million records in a reference list of all (~20m) accounts. |
Combining a large stream with data from a small stream, where the values of the small stream will be repeated throughout the result. For each pair of matching records from the data sets, a single record is produced in the output. See Enriching Data with Data From Another Set | Calculate stream with lookup pipe with order/index set | Find the description for each code in a stream of thousands from a stream containing mapping data. There are only ~100 possible codes. |
Combining a large stream with data from a small stream, where values in the small stream will only used once in the result. For each pair of matching records from the data sets, a single record is produced in the output. See Combining Data Using a Lookup Pipe | Calculate stream with lookup pipe with filter | You have a stream containing all attendees of an upcoming football match and a small stream of people who are banned from attending matches. |
Combining a large stream with data from a small stream, where the small stream the same rows are repeated throughout the result, but the filter values change slightly. For each pair of matching records from the data sets, a single record is produced in the output. Combining Data Using a Cache Extraction Filter Lookup Pipe | Calculate stream with lookup pipe with cache extraction filter | You have a price list for 4 different products with different prices between different dates. |
You want to look back at a previous record within a group in a stream, or create a cumulative total per group. You get the same number of records as you put in. See Grouping and Referencing Data Using Calculate By Set Stream | Calculate by set | For a given account, you want to find the difference between each consecutive debit/credit to the account. |
Please let us know if we could improve this page feedback@phixflow.com