Speed Up Data Loading Using Parallel Collectors

Scenario

Some systems deal with massive quantities of data. Where the processing bottleneck is found to be the database data extraction the data can, under certain circumstances, be extracted in parallel to speed up processing.

Use multiple pull pipes to extract data from a single database collector into a single table.

Solution:

• Create the database collector and the table to contain the data
• Link the collector and table with multiple pull pipes.
• Each pipe has a unique name.
• In the 'where' clause of the SQL statement in the collector, use the pipe name to limit the set of data being to be retrieved. E.g if the pipe is called db2, then have the SQL query only return those records where the id ends in 2. this can be does using the _pipeName internal variable.