PhixFlow Help

In Memory Streams

Owned by Former user (Deleted)
Last updated: Oct 07, 2015 by Chris Welford
1 min read

Scenario

Very large quantities of unsorted data from a file are required to be processed. PhixFlow can process this data into a stream, but that can prove inefficient at best (and at worst, can cause the system to fail if the volue of data is too large).

Read a file, but only process an aggregate of the records in the file.

Solution:

  • Create a stream to read a file.
  • Mark the stream as an InMemory Stream.
  • Create another stream with a pull pipe from the first stream to collect the aggregated data.

 

Please let us know if we could improve this page feedback@phixflow.com