PhixFlow Help

Skip to end of metadata
Go to start of metadata

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 2 Current »

In Memory Streams

Scenario

Very large quantities of unsorted data from a file are required to be processed. PhixFlow can process this data into a stream, but that can prove inefficient at best (and at worst, can cause the system to fail if the volue of data is too large).

Read a file, but only process an aggregate of the records in the file.

Solution:

  • Create a stream to read a file.
  • Mark the stream as an InMemory Stream.
  • Create another stream with a pull pipe from the first stream to collect the aggregated data.

 

  • No labels