Versions Compared

Old Version 8

changes.mady.by.user Dominic Sandhu (Unlicensed)

Saved on

compared with

New Version 9

changes.mady.by.user Dominic Sandhu (Unlicensed)

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

Scenario

Panel
bgColor#DCDCDC
titleBGColor#808080
titleWhat counts as a duplicate?

There are three cases of duplicate records:

  1. Two or more records have identical values in each and every field (true duplicates).
  2. Two or more records have identical values in some fields, and the fields that do not have matching values are of no consequence (it does not matter which value we take).
  3. Two or more records have identical values in some fields, and one of the variable fields gives us a vale we can select on (in practice, usually a datetime field like 'last updated time').

...

  1. Load all data (including duplicates) into a stream
  2. Create a new stream from this stream - make it an aggregate stream.

  3. On the pipe linking the two streams, set the maximum number of records to be one and group it on the field with duplicated data. 

    Panel
    bgColor#DCDCDC
    titleBGColor#808080
    titleOptional

    If this is case 3, apply sorting on another attribute, depending on which record you want. E.g. to get the latest record, sort by the last updated date. See 'outMult' in the screenshot below for illustration.

  4. Run the analysis and inspect the resultant stream data. There should now be only unique records.

...