By the end of this chapter you will be able to:
- Use a loop pipe to read a Stream's own historic data when generating a new Stream Set
- Use Data to Read = Previous
Files containing details of metered usage can be found in the input files, in the directory:
…\inputData\MeteredUsage\input
Read in a copy of these files on the CenterView server:
- Add a sequence:
- Find the button
- Show the list of Sequences in the left-hand menu bar
- Press
- Enter the following details into the sequence configuration form:
- Name: MeteredSeq
- Start Value: 1
- Press OK to save your changes
- Find the button
- Create a new model view
- Create a file collector to read files of metered usage data:
- Add a File Collector to your model, with Name Metered Usage
- Enter the following settings:
- Input Directory Expr.: "MeteredUsage/input"
- File Pattern Expression: ".*"
- Enabled: ticked
- Number of Header Lines: 1
- Allow Non-Scheduled Collection: ticked
- Create the columns in the file collector and create a stream
- Update the File Pattern Expression in the file collector to: "meteredUsage" + nextValue("MeteredSeq") + ".txt" (the file names have a sequence number)
- Add a merge called Total Metered Usage
- Complete configuration of Total Metered Usage:
- Add a pipe from Metered Usage into Total Metered Usage, with Name in
- Add the grouping key CustomerRef to the pipe in
- Drag all Attributes from Metered Usage into Total Metered Usage
- Add a pipe from the Stream into itself
- Edit details for the loop pipe:
- Set the Name to prev
- Set Data to Read to Previous
- Add the grouping key CustomerRef
- In the stream Total Metered Usage:
- Set the expression on the Attribute CustomerRef to _key[1]
- Set the expression on the Attribute MeteredUsage to: prev.MeteredUsage + in.MeteredUsage
This expression will add the value from the previous Stream Set – that is, the data from the previous run - for the current key value to the new value being read in for this key value
In this way you will build a running total for metered usage for each customer
- Save your changes
You cannot, in most cases, do a Merge or Aggregate directly from a File Collector. This is because when bringing data in for a Merge or Aggregate CenterView needs to sort the data (by the grouping keys being used). When reading data in from a file, there is no way for CenterView to guarantee the order of the records. When you want to Merge or Aggregate data from files, load this first into a Calculate Stream and then add a Merge/Aggregate Stream.
- Run Analysis on Total Metered Usage three times
- You will see in Metered Usage that in each Stream Set a file has been read in
- Look at the Stream Sets generated in Total Metered Usage - you will see that a running total is maintained for each customer as each new data set is read in