Scenario
A very large MasterList is periodically updated with a small create/update/delete feed. It is computationally expensive and time consuming to create a completely new master list stream, when only (for example) 220 out of 150,000,000 records are actually updated.
Implement a MasterList in PhixFlow where the MasterList is very large, and the number of updates is tiny by comparison.
Solution:
- Create the Stream to act as a MasterList keyed on a unique identifier.
- Create streams and/or file collectors to update the MasterList.
- Create Secondary Streams to pull data from the MasterList Stream, along Non-Historied pipes.
- Group non-historied pipes by the unique identifier
- Construct the attribute logic in those secondary streams can distinguish the correct data from all the records that share the unique identifier.