Scenario
Sometimes the data that you are trying to match is not exactly the same, but you still want to be able to align and merge data, or create a report on likely matches.
Create 2 streams for data. The goal is to match up those records from stream 1 where at least 3 of 5 fields match stream 2.
Solution:
- Create a stream for each input
- Create a (calculate) stream for the join.
- Make the pipe from one of the streams a pull pipe and the pipe from the other is a lookup pipe.
- In the join stream, for each record on the pull pipe, cycle thru all the records from the lookup pipe and flag those records where there is a match.