How To: Configure an In Memory Stream

An "In Memory" stream can efficiently read a large amount of unsorted data to generate summary results. An In Memory stream itself stores no data, but the results can be read into another stream.

Configuration of In Memory streams is illustrated in this article with an example:

You have a file of record with, on each line, two key values: KeyVal (e.g. "Key5"), Node (e.g. "AA112"); and a count of events for that key combination, EventCount. A snippet of the files appears as below:

103,"AA112","Key5"
115,"AA174","Key3"
126,"AA100","Key1"
145,"AA169","Key5"
126,"AA164","Key3"
158,"AA196","Key0"
117,"AA150","Key2"

This data is unsorted, and you want to generate a total count for each key combination. An In Memory stream is a good way to achieve this efficiently.

The following configuration will result in the model:

To do this:

• Create a file collector on the file as normal, with attributes:

  • EventCount

  • Node

  • EventCount

• Create an In Memory stream:
  • Stream Function = Calculate
  • Storage Type = In Memory
  • Linked to the file collector with a pipe "in"
  • Attributes as below:

1. • The stream attributes above are configured as:
     • KeyVal - a key value:
       • Populated with the expression in.KeyVal to take the KeyVal from the file collector
       • To make the this attribute a key value the Key flag has been ticked on the stream attribute configuration form:

• • • • This means that a record will be created in the cache for each new value of KeyVal
  • Node - a key value, configured similarly to KeyVal, with the flag Key ticked
    • This means that a record will be created in the cache of each new combination of KeyVal and Node
  • EventCountTotal
    • Populated with the expression: _cacheRecord.EventCountTotal+in.EventCount
      • The variable _cacheRecord is an Internal Variable: a record is created in the cache for each combination of key values, and _cacheRecord is the record for the current key. The record has a value for each attribute defined in the stream, e.g. EventCountTotal.
      • So the expression _cacheRecord.EventCountTotal+in.EventCount takes the current total count (_cacheRecord.EventCountTotal) then adds the new value read in from the file to it (+in.EventCount)
  • RecordCount - similarly to EventCountTotal, this attribute is building a total across records which share a key value combination
    • In this case we are generating a simple count of records from the file with the expression _cacheRecord.RecordCount+1
      
      
• Create a stream to record the results of the In Memory stream:
  • This is a simple calculate stream which reads in the attributes of the In Memory stream, with attributes as below

After running the final stream we get results like:

Mode Ref: 009