Parallel Collectors

Scenario

Some systems deal with massive quantities of data. Where the processing bottleneck is found to be the database data extraction the data can, under certain circumstances, be extracted in parallel to speed up processing.

Use Multiple Pull pipes to extract data from a single DB collector into a single stream.

Solution:

• Create the DB collector and the Stream to contain the data
• Link the collector and stream with multiple pull pipes.
• Each pipe has a unique name.
• In the 'where' clause of the SQL statement in the collector, use the pipe name to limit the set of data being to be retrieved. E.g if the pipe is called db2, then have the SQL query only return those records where the id ends in 2. this can be does using the _pipeName internal variable.