In Memory Streams

Scenario

Very large quantities of unsorted data from a file are required to be processed. PhixFlow can process this data into a stream, but that can prove inefficient at best (and at worst, can cause the system to fail if the volue of data is too large).

Read a file, but only process an aggregate of the records in the file.

Solution:

• Create a stream to read a file.
• Mark the stream as an InMemory Stream.
• Create another stream with a pull pipe from the first stream to collect the aggregated data.