Create Unique Indexes / Record Identifiers

Scenario

When processing large volumes of data, where the data is uniform, but coming from many different sources (e.g. a mediation system will process data from multiple switches) and doing a great many transformations, tagging each record at the point it enters the system with a unique number is useful for auditing and tracking.

Generate a counter – a unique number – for all the records in a stream set.
Combine 2 or more stream sets and for each stream set have the counter continue to increment over the value of the last record of the previous stream set.

Solution:

• Create a Stream to read in data from a DB table or file
• Create an attribute of that stream to contain a unique number
• Use the built in cache() function to set the attribute. Increment the cache for the attribute to ensure that the next record contains a unique number.