Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

Insert excerpt
_Banners
_Banners
nameanalysis
nopaneltrue

Scenario

Some systems deal with massive quantities of data. Where the processing bottleneck is found to be the database data extraction the data can, under certain circumstances, be extracted in parallel to speed up processing.

Use

...

multiple pull pipes to extract data from a single

...

database collector into a single

...

table.

Solution:

  • Create the

...

  • database collector and the

...

  • table to contain the data
  • Link the collector and

...

  • table with multiple pull pipes.
  • Each pipe has a unique name.
  • In the 'where' clause of the SQL statement in the collector, use the pipe name to limit the set of data being to be retrieved. E.g if the pipe is called db2, then have the SQL query only return those records where the id ends in 2. this can be does using the _pipeName internal variable.

...