Page Comparison

...

1. Click to select the stream that may contain duplicates.
2. Right-click on the model view pane, and select 'Merge selected streams'. 
   
   
3. in the pipe configuration dialog that pops up, group on the field with duplicated data and click the green tick to save your input.
   
   
4. in the Automatic Stream Configuration dialog that appears, select 'just key attributes' from the drop down. 
   
5. Run analysis on the stream that results. Viewing the data, it can be seen that for each value of the grouping key, 
   PhixFlow reports the number of records in that group, and also highlights lines where it is greater than one.
6. If after inspecting the data you think the groups of duplicated records are defined by some key other than what you have tried, you can repeat the above procedure, generating a new merge stream but with different grouping keys.
   
   

Step-by-step guide: Removing Duplicates

...

1. Load all data (including duplicates) into a stream
2. Create a new stream from this stream - make it an aggregate stream.
   

3. On the pipe linking the two streams, set the maximum number of records to be one and group it on the field with duplicated data. 

   Panel
   bgColor #DCDCDC titleBGColor #808080 title Optional
   If this is case 3, apply sorting on another attribute, depending on which record you want. E.g. to get the latest record, sort by the last updated date. See 'outMult' in the screenshot below for illustration.

   

4. Run the analysis and inspect the resultant stream data. There should now be only unique records.
   

...

Related articles

...