How To: Configure an In Memory Stream

An "In Memory" stream can efficiently read a large amount of unsorted data to generate summary results. An In Memory stream itself stores no data, but the results can be read into another stream.

Configuration of In Memory streams is illustrated in this article with an example:

You have a file of records with, on each line, two key values: KeyVal (e.g. "Key5") and Node (e.g. "AA112"); and a count of events for that key combination, EventCount. A snippet of the file is shown below:

103,"AA112","Key5"
115,"AA174","Key3"
126,"AA100","Key1"
145,"AA169","Key5"
126,"AA164","Key3"
158,"AA196","Key0"
117,"AA150","Key2"

This data is unsorted, and you want to generate a total count for each key combination. An In Memory stream is a good way to achieve this efficiently.

The following configuration will result in the model:

To create this model:

Create a file collector

• Create a file collector on the file as normal, with attributes:

  • EventCount

  • Node

  • EventCount

Create an In Memory Stream

Create an In Memory stream:

• Stream Function = Calculate
• Storage Type = In Memory
• Linked to the file collector with a pipe "in"

Create the stream attributes

Create the stream attributes as below:

The stream attributes above are configured as:

KeyVal - a key value

Populated with the expression in.KeyVal to take KeyVal from the file collector

To make this attribute a key value the Key flag has been ticked on the stream attribute configuration form:

This means that a record will be created in the cache for each new value of KeyVal

Note - a key value

Configured similarly to KeyVal, with the flag Key ticked

This means that a record will be created in the cache of each new combination of KeyVal and Node

EventCountTotal - an accumulated total count

Populated with the expression: _cacheRecord.EventCountTotal+in.EventCount

The variable _cacheRecord is an Internal Variable: a record is created in the cache for each combination of key values, and _cacheRecord is the record for the current key. The record has a value for each attribute defined in the stream, e.g. EventCountTotal.

So the expression _cacheRecord.EventCountTotal+in.EventCount takes the current total count (_cacheRecord.EventCountTotal) then adds the new value read in from the file to it (+in.EventCount)

RecordCount - a record count

Similarly to EventCountTotal, this attribute is building a total across records which share a key value combination

In this case we are generating a simple count of records from the file with the expression _cacheRecord.RecordCount+1

Store the results of the In Memory Stream

Create a stream to record the results of the In Memory stream: this is a simple calculate stream which reads in the attributes of the In Memory stream, with attributes as below

Results

After running the final stream we get results like:

Mode Ref: 009