Building a history

By the end of this chapter you will be able to:

• Use a loop pipe to read a Stream's own historic data when generating a new Stream Set
• Use Data to Read = Previous

Files containing details of metered usage can be found in the input files, in the directory:
…\inputData\MeteredUsage\input
Read in a copy of these files on the PhixFlow server:

• Add a sequence:
  • Find the button  - Show the list of Sequences in the left-hand menu bar
  • Press  to create a new sequence
  • Enter the following details into the sequence configuration form:
  • Name: MeteredSeq
  • Start Value: 1
  • Press OK to save your changes
• Create a new model view
• Create a file collector to read files of metered usage data:
• Add a File Collector to your model, with Name Metered Usage
• Enter the following settings:
  • Input Directory Expr.: "MeteredUsage/input"
  • File Pattern Expression: ".*"
  • Enabled: ticked
  • Number of Header Lines: 1
  • Allow Non-Scheduled Collection: ticked
• Create the columns in the file collector and create a stream
• Update the File Pattern Expression in the file collector to: "meteredUsage" + nextValue("MeteredSeq") + ".txt" (the file names have a sequence number)
• Add a merge called Total Metered Usage
• Complete configuration of Total Metered Usage:
• Add a pipe from Metered Usage into Total Metered Usage, with Name in
• Add the grouping key CustomerRef to the pipe in
• Drag all Attributes from Metered Usage into Total Metered Usage
• Add a pipe from the Stream into itself
• Edit details for the loop pipe:
  • Set the Name to prev
  • Set Data to Read to Previous
  • Add the grouping key CustomerRef
• In the stream Total Metered Usage:
  • Set the expression on the Attribute CustomerRef to _key[1]
  • Set the expression on the Attribute MeteredUsage to: prev.MeteredUsage + in.MeteredUsage

This expression will add the value from the previous Stream Set – that is, the data from the previous run - for the current key value to the new value being read in for this key value
In this way you will build a running total for metered usage for each customer

• Save your changes

  You cannot, in most cases, do a Merge or Aggregate directly from a File Collector. This is because when bringing data in for a Merge or Aggregate PhixFlow needs to sort the data (by the grouping keys being used). When reading data in from a file, there is no way for PhixFlow to guarantee the order of the records. When you want to Merge or Aggregate data from files, load this first into a Calculate Stream and then add a Merge/Aggregate Stream.

• Run Analysis on Total Metered Usage three times
• You will see in Metered Usage that in each Stream Set a file has been read in
• Look at the Stream Sets generated in Total Metered Usage - you will see that a running total is maintained for each customer as each new data set is read in