In Memory Streams
Scenario
Very large quantities of unsorted data from a file are required to be processed. PhixFlow can process this data into a stream, but that can prove inefficient at best (and at worst, can cause the system to fail if the volue of data is too large).
Read a file, but only process an aggregate of the records in the file.
Solution:
- Create a stream to read a file.
- Mark the stream as an InMemory Stream.
- Create another stream with a pull pipe from the first stream to collect the aggregated data.