Skip to end of metadata
Go to start of metadata

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 2 Next »

Inexact Matching DateTimes

Scenario

Record written to different systems are timestamped when they are written. Timestamps may not match for a variety of reasons (system clocks out of sync, delays in writing, batch processing etc), but there may still be a need to reconcile an match records.

Create 2 streams, with at least 1 DateTime field, and one other field to act as a key. Merge the two streams such that you pick up the records where the keys match and the datetime values are within a set tolerence range.

Solution:

  • Create a stream for each input.
  • Create an calculate stream to perform the join.
  • Make the pipe from one of the streams a pull pipe and the pipe from the other is a lookup pipe.
  • In the join stream, for each record on the pull pipe, cycle thru all the records from the lookup pipe for that key and flag those records where there is a match based on a date range.
  • Date ranges should be calculated based on converting dates to integers. Note : there is a dateDiff function in PhixFlow that will give the difference in milliseconds between 2 dates

 

 

  • No labels