Very large quantities of data are required to be processed from a set of files. It is useful to be able to process each file into a separate
...
table, such that if we are processing 10 files and there is a failure in in the 9th file, we will not have to re-process the first 8 files.
Load the data from 10 files into 10 different
...
tables in the same
...
recordset.
Solution:
Create a
...
table connected to a file collector
Ensure the file collector will collect multiple files when it is activated (i.e. ensure there are) multiple files that match the file collector criteria.
Create pipe connecting the
...
table to itself.
Set the pipe attribute : type to 'push'.
Set the pipe attributes : From and To Date Offsets to 1