About Master Lists
Lots of organisations do a lot of process work using some variation of what we call a master list. This is a bit like using an Excel spreadsheet that has a large number of rows and a large number of columns. The spreadsheet is passed around various people in the different departments so that people can:
- either update data
- or extract data to act on.
You can create a master list in PhixFlow
Manage a Small Master List
Scenario
Implement an approval process for credits on invoices in PhixFlow, using a master list.
Solution:
- Create a merge stream to act as a master list, keyed off a unique identifier.
- Create streams and/or file collectors to update the master list.
- Create a pull pipe linking the master list Stream to itself, with its date offsets set to -1, -1.
- Every time the stream is run, the master list stream is recreated from the previous version of itself and the updates that happen via other streams.
Manage a Large Master List
Scenario
A very large master list is periodically updated with a small create/update/delete feed. It is computationally expensive and time consuming to create a completely new master list stream, when only (for example) 220 out of 150,000,000 records are actually updated.
Implement a master list in PhixFlow where the master list is very large, and the number of updates is tiny by comparison.
Solution:
- Create the Stream to act as a master list keyed on a unique identifier.
- Create streams and/or file collectors to update the master list.
- Create Secondary Streams to pull data from the master listStream, along Non-Historied pipes.
- Group non-historied pipes by the unique identifier.
- Construct the attribute logic in those secondary streams can distinguish the correct data from all the records that share the unique identifier.