Skip to end of metadata
Go to start of metadata

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

Version 1 Next »

Managing a Large MasterList

Scenario

A very large MasterList is periodically updated with a small create/update/delete feed. It is computationally expensive and time consuming to create a completely new master list stream, when only (for example) 220 out of 150,000,000 records are actually updated.

Implement a MasterList in PhixFlow where the MasterList is very large, and the number of updates is tiny by comparison.

Solution:

  • Create the Stream to act as a MasterList keyed on a unique identifier.
  • Create streams and/or file collectors to update the MasterList.
  • Create Secondary Streams to pull data from the MasterList Stream, along Non-Historied pipes.
  • Group non-historied pipes by the unique identifier
  • Construct the attribute logic in those secondary streams can distinguish the correct data from all the records that share the unique identifier.

See Also

  • href="../../../../../../doc/training/Exercises.pdf">Patterns and Exercise Training Document
  • href="http://www.phixflow-online.com:8081/patterns">PhixFlow Online Patterns and Training
  • No labels