| Maximum Number of Records Per Group | If a value is entered into this field, then when PhixFlow collates the input records into groups according to the specified Grouping Attribute Details (see below) once the maximum number of records have been added to the group any additional records for this group will be discarded. Records will be added to the group according to the specified sort order. This can be useful, for example, where you know that for a given set of grouping attributes you may get multiple records but you are only interested in the most recent record for that set. In this case you can configure the pipe to group by the grouping attributes, sort by an appropriate date attribute in descending order and set the Maximum Number of Records to 1. This will ensure that you only get the most recent record for the specified group when you read from this pipe. |
| Index Type | Look-up pipes can be configured for fast "indexed" access to cached data collected from external tables, files or from other streams. Indexed access is controlled through configuring a pipe with an index and setting index expressions on grouping attributes. If the Type field on the Pipe is set to 'Look-up' then the field "Index Type" becomes available. This can have the value "None" meaning that there are no index keys or "Exact Match", "Best Match" or "Near Match" as described below:- Exact Match: The pipe will retrieve data from its cache based on an exact match look-up with the values provided after evaluating the index expressions on the "Group By" attributes.
- Best Match: The pipe will retrieve data from its cache based on a "Best Match" look-up after evaluating the index expressions on the "Group By" attributes. Note that the last Group By Attribute with a key expression is used for the best match lookup. The index keys on any Group By attributes with a lower sequence number are used as an initial "Exact Match" to find the set of data on which to do the "Best Match". The "Best Match" is defined as the longest key value which matches the evaluated index expression.
- Near Match: The pipe will retrieve data from its cache based on a "Near Match" look-up after evaluating the index expressions on the "Group By" attributes. Note that the last Group By Attribute with a key expression is used for the near match lookup. The index keys on any Group By attributes with a lower sequence number are used as an initial "Exact Match" to find the set of data on which to do the "Best Match". When "Near Match" is selected, an additional field appears where you can enter an expression which should evaluate to a number representing the allowed number of edits (e.g. deletions, insertions, substitutions and transpositions) which can be made when comparing the result of the index expression to the index key in order to achieve a match.
For example if the index key is "Smyhte" and the result of the index expression is "Smith" this would still be a match providing that the allowed number of edits is 3 or more (i.e. substitute the 'i' for a 'y', transpose the 't' and the 'h' and then insert an 'e' at the end).
|