Page Comparison

Scenario

Files (or database records) can often show up with duplicate data. Often it is OK, and sometimes it is required to ignore duplicate records.

...

Panel

There are three cases of duplicate records:

Two or more records have identical values in each and every field (true duplicates).
Two or more records have identical values in some fields, and the fields that do not have matching values are of no consequence (it does not matter which record we take).
Two or more records have identical values in some fields, and one of the variable fields gives us a vale we can select on (in practice, usually a datetime field like 'last updated time').

Click to select the stream that may contain duplicates.
Right-click on the model view pane, and select 'Merge selected streams'.

Image Added
in the pipe configuration dialog that pops up, group on the field with duplicated data and click the green tick to save your input.

Image Added
in the Automatic Stream Configuration dialog that appears, select 'just key attributes' from the drop down.
Image Added
Run analysis on the stream that results. Viewing the data, it can be seen that for each value of the grouping key,
PhixFlow reports the number of records in that group, and also highlights lines where it is greater than one.
If after inspecting the data you think the groups of duplicated records are defined by some key other than what you have tried, you can repeat the above procedure, generating a new merge stream but with different grouping keys.

Load all data (including duplicates) into a stream
Create a new stream from this stream - make it an aggregate stream.
Make Image Added
On the pipe linking the
2
two streams
an aggregate pipe, grouped
, set the maximum number of records to be one and group it on the field with duplicated data
, and sorted by another field
.
Panel
bgColor #e6f0ff
titleBGColor #99c2ff
title Optional
If this is case 3, apply sorting on another attribute, depending on which record you want.
e
E.g.
if you want
to get the latest record,
you could
sort by the last updated date.
In the second stream, reference data coming from the input pipe using an array index . i.e. in[1].value to just retrieve the first of the grouped records.

...

See 'outMult' in the screenshot below for illustration.
Image Added
Run the analysis and inspect the resultant stream data. There should now be only unique records.