...
Add a clause or condition. | |
 | Delete a clause or condition. |
 | Specifies that the value entered is a literal value. Click this icon to change this - to specify that the value entered be evaluated as an expression. |
 | Specifies that the value entered is a evaluated as an PhixFlow expression. Click this icon to change this - to specify that the value entered be treated as a literal. Note : ["123", "234", "345"] looks like a literal value but it can be evaluated as an expression. |
| Open the expression in a larger editor. |
Cache Extraction Filter
On lookup pipes, you can define a A cache extraction filter .
Lookup pipes hold retrieved data in a cache.
For each record processed in the Stream, records are retrieved from the cache using the index expressions. But the data required for each record can be further filtered using a Cache Extraction Filter.
Cache Extraction Filters are entered free hand, although they have a similar format to the text version of a filter that is shown as a filter is created in the main Conditions box of the Filter tab for Pipes (see above)allows you to further filter the data retrieved by a pipe. These are not commonly used, but are sometimes helpful when either:
- Optimising performance on a lookup pipe when for a set of records, the record you require from the lookup depends on non-key data, e.g. the date
- When getting data from a pull pipe when the filter requires that you compare one value in each record with another; this is not possible within a standard filter.
For case 1, when using a lookup pipe, data retrieved is stored in a cache. See cache size for details. The cache extraction filter allows you, as you are processing a set of output records, to use different cached entries from the lookup for each of the records are you are processing. This is very fast compared to looking up from the source (i.e. going back to an external DB table or even another PhixFlow stream) for each output record.
E.g. if you wanted want to look up the Credit Rating credit rating for a customer , you might created an Index Look-up Pipe for a set of transactions - in the output, each transaction is represented by a single output record. You create an indexed lookup pipe using CustNo as the key for the index. This means that for each record processed new CustNo you encounter in the Streamdata, all the records credit rating entries for that CustNo would be retrieved from by the pipe and placed into the cache for the CustNo in the record. But if this data was . The credit rating for each customer is fully historied, so you would get a number of records or each CustNoentries for each CustNo. To get the relevant lookup entry for each output report (each transaction), you need to compare the transaction date of the output record to the dates of credit rating entries in the cache. So to extract the current relevant record, you could include a Cache Extraction Filter cache extraction filter in the form:
| Code Block |
|---|
StartDate >= |
...
_out.TransDate && (EndDate <= |
...
_out.TransDate || EndDate == _NULL) |
Cache extraction filters are entered free hand.
The attribute names referenced must exist in a stream. This means that the source Stream - that is, the Stream you are reading from - oreach attribute must be one of:
- an attribute in a source stream, if you are reading
...
- from
...
- a stream
- if you are reading from an external database table, one of the fields returned by the database collector AND an attribute in the output stream - i.e. to use an attribute with the source as a database collector, there must be an attribute of matching name in the output stream
- an attribute in the destination stream, in which case you will refer to it using the format
_out.AttributeName
Filter on Current User
Views of entities that show a User or Owner attribute can be filtered to show only those records for which the Owner / User matches the current logged in user. This particularly applies to Alarms and user Tasks. To set a filter where the Owner equals the current logged in user, add a condition to the filter like this:
...
| Field | Description | |||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
Mandatory | If ticked, when multiple Streams are being merged then there must be an input record from this Pipe for an output record to be generated by the output Stream. If this is a push pipe with positive offsets and this flag is ticked then the notification to create another stream set will only be pushed along the pipe if the last stream set created contains at least one record. | |||||||||||||||||||||||||||||||
| Multiplier | When processing data, a Stream first constructs CandidateSets ready for processing from its (non-Multiplier) input Pipes. For each Multiplier Pipe, the Stream then multiplies each CandidateSet by creating a copy of the original CandidateSet for each row returned from the Multiplier Pipe. Note that each resulting CandidateSet contains the original CandidateSet plus one row from the Multiplier Pipe. | |||||||||||||||||||||||||||||||
| Execution Strategy | The Execution Strategy determines how this pipe should be implemented. See the section on Directed Merge Strategy | |||||||||||||||||||||||||||||||
| Max Workers | The maximum number of concurrent worker tasks. If blank, this defaults to 1. | |||||||||||||||||||||||||||||||
| Worker Size | The number of key values to read for a single worker task (which runs a single select statement). If blank, this defaults to 1000. This is the maximum value that can be used when reading from an Oracle database. | |||||||||||||||||||||||||||||||
| The cache is used when carrying out lookups from streams or database collectors. When doing a lookup, there are two common scenarios:
In case 2, the results returned are typically based on a key value, e.g. an account number. This will be used in the filter of the pipe, if you are reading from a stream, or in the query, if you are reading from a database collector. For example, the query in a database collector will include the condition:
For efficiency, the records are cached (stored temporarily in memory) so that if the same set of records need to be looked up again they are readily available without going back to the database. This field allows you to set a limit on the size of the cache. Setting a limit is important because if you do not, the cache can become very large and consume a lot of memory, which can lead to a slow down in both your tasks and those of other users of PhixFlow. To set the cache size, try to estimate the largest number of records that the lookup pipe will return on a single read. If you do not set a limit, it will default to the system-wide default, specified in the Maximum Pipe Cache Size in the System Tuning tab of the System Configuration.
| |||||||||||||||||||||||||||||||
| Buffer Size | The buffer size used to perform the stream calculation. If a large amount of data is being processed, then setting a large buffer size will give better performance. | |||||||||||||||||||||||||||||||
| Allow Incomplete Stream Sets | Normally, when a pipe tries to read from an input stream that contains an incomplete stream set, PhixFlow will attempt to complete the stream set before passing data down the pipe. However if the stream is static (i.e. the stream has its 'static' flag ticked) or is effectively static (i.e. all of the pipes reading from it in this analysis run are static) then, instead of completing the stream set, an error message is produced indicating that you cannot read from this stream because it contains an incomplete stream set. If you do not want this error message to be produced when reading from static (or effectively static) streams, but would instead prefer PhixFlow to ignore the incomplete stream sets, then you must tick this box on all pipes that will read from the input stream in this analysis run. If there are multiple pipes that read from the input stream during this analysis run and even one of the pipes does not have this box ticked then you will not be allowed to read from the stream and the error message will be produced. Pipes which are not used in the current analysis run (for example where they lead to streams on branches of the model which are not run by the current task plan) have no effect on whether or not the error message is produced. | |||||||||||||||||||||||||||||||
| Data Expected | This field is available when the Pipe Type is Push or Pull. This flag allows the user to specify that the pipe is expecting to receive data. If ticked but no data is received this is treated as an error. | |||||||||||||||||||||||||||||||
| Pipe View | The pipe view is used to limit which fields are retrieved down the pipe and in what order, and in some circumstances how each field is to be formatted. You can select from any of the views that have been configured on the source stream. Please note that any sorting or filtering of records will have to be applied directly on the pipe, and will not be inherited from the pipe view. The pipe view is used in three contexts. During Look UpsPipe views can also be used on lookup pipes to limit the fields that are returned by the lookup request. This is most useful in the scenario where the you want to read and cache data on a lookup pipe from a stream that has lots of attributes but where only a small number of attributes are actually required. You can simply create a new view on the source stream listing only the attributes needed, then specify it as the pipe view on the lookup pipe. Only those attributes specified on the view will then be loaded.Without a pipe view, the pipe will load and cache all of the attributes from the stream which may consume a significant amount of free memory if there are a large number of records. During File ExportWhen sending data to a file exporter only those fields specified on the pipe view will be exported. If no pipe view is supplied then all fields will be exported. During Drill DownWhen drilling down from an alarm or stream item the pipe view is only used to determine which attributes from the source stream should be shown in the drill down display and in what order. | |||||||||||||||||||||||||||||||
| Pipe Exporter | An exporter can be selected from the set of Database Exporters configured for the input stream. This exporter can then be used from the Drill Down View. This feature is useful when PhixFlow is used to 'recommend' a set of updates. By configuring an alarm to be generated when a set of recommendations is made, the user can drill down through the alarm to see the list of recommendations and then hit the exporter icon to apply them. Any filters applied on the pipe will be applied when the data is pushed to the pipe exporter, so it is possible that not all of the data in the grid will be exported - some records may be rejected by the filter. | |||||||||||||||||||||||||||||||
| Max Records To Read | The maximum number of records that should be read down this pipe. The pipe may read more than this number of records if it is configured to carry out multiple reads simultaneously e.g. if it is connected to a File Collector which has been configured to read multiple files simultaneously or if this pipe strategy is "Directed" with multiple workers. |
...