Managing a Large MasterList

Scenario

A very large MasterList is periodically updated with a small create/update/delete feed. It is computationally expensive and time consuming to create a completely new master list stream, when only (for example) 220 out of 150,000,000 records are actually updated.

Implement a MasterList in PhixFlow where the MasterList is very large, and the number of updates is tiny by comparison.

Solution:

Create the Stream to act as a MasterList keyed on a unique identifier.
Create streams and/or file collectors to update the MasterList.
Create Secondary Streams to pull data from the MasterList Stream, along Non-Historied pipes.
Group non-historied pipes by the unique identifier
Construct the attribute logic in those secondary streams can distinguish the correct data from all the records that share the unique identifier.

masterListLarge

Managing a Large MasterList

Scenario

See Also