PhixFlow Help

Managing a Large Master List

Scenario

A very large MasterList is periodically updated with a small create/update/delete feed. It is computationally expensive and time consuming to create a completely new master list stream, when only (for example) 220 out of 150,000,000 records are actually updated.

Implement a MasterList in PhixFlow where the MasterList is very large, and the number of updates is tiny by comparison.

Solution:

  • Create the Stream to act as a MasterList keyed on a unique identifier.
  • Create streams and/or file collectors to update the MasterList.
  • Create Secondary Streams to pull data from the MasterList Stream, along Non-Historied pipes.
  • Group non-historied pipes by the unique identifier
  • Construct the attribute logic in those secondary streams can distinguish the correct data from all the records that share the unique identifier.

 

Please let us know if we could improve this page feedback@phixflow.com