Name

Enter the name of the column, which can contain any combination of letters, numbers and _ the underscore character.

If you use  

PhixFlow always uses this name to refer to this attribute.

Order

Enter a number that matches the column number in the input file. For example, if you want to extract the third, first and fifth column of data from a file, the three rows in this grid will have the order:

• 3

• 1

• 5

Type

Enter one of the data types:

• String

• Integer

• Float

• Date - see Supported Date/Datetime Format Patterns, below

• Datetime - see  Supported Date/Datetime Format Patterns, below

• TrueFalse -  see Supported TrueFalse Values, below

If you use  

Length

Enter the maximum length of the field in the input file

• For String, the maximum length of the string

• For Integer, the maximum number of digits.

This is:

• required when File Type is Fixed Length Records

• optional when File Type is Comma Separated Values or Excel Spreadsheet, as only the field sizes of the fields in the table that this collector will write to are important.

If you use  

Maximum Files

Enter the maximum number of files that PhixFlow will process when the file collector runs.

Minimum Files

Enter the minimum number of files that PhixFlow expects to find when the file collector runs. If fewer files are found then this is treated as an error.

Max Records Per File

Enter the maximum number of records that PhixFlow will read from each file.

Errors Before Rollback

Parallel Readers

Enter the number of files to process in parallel. If blank, this defaults to 1. If the file collector is configured to read files in sequence, this field is ignored and a single file reader is used.

Unreadable Directories

Select what PhixFlow will do if there are unreadable directories when it is searching a directory hierarchy for files to import.

• Error: the search fails and the unreadable directories are reported in the log.

• Warning: the search continues and unreadable directories are reported in the log.

• Ignore: the search continues and unreadable directories are ignored.

XPath Expression

Available when File Type is XML File.

Enter valid XPath syntax. For information about how to use XPath expressions and how to use the returned data in the corresponding stream attribute expressions, see XPath Examples.

Character Set

Column Separator

Separator Character

Quote Style

Quote Character

Ignore Extra Columns

Ignore Missing Columns

Import Rows Matching

Enter an expression that PhixFlow compares to each line in the file against the Expressions and PhixScripts. Only lines that match are imported.

Replace Text Matching

In each imported line, find all occurrences of the expression that you enter in Replace Text Matching and replace it with the expression that you enter in With; see Expressions and PhixScripts.

With

Excel Data Range Expression

Leave this field blank or enter an expression for the spreadsheet data range that PhixFlow will look in. The data that PhixFlow extracts from the range is defined in the File Columns section, below. The expression can specify:

• blank: PhixFlow looks at the whole first worksheet in the Excel file, even if this is a hidden sheet. 

• columns on the first worksheet, for example, "A:C"

• columns and rows on the first worksheet, for example, "A1:G10"

• the worksheet and data range, with the format "WorksheetName!TopLeftCell:BottomRightCell", for example, "DailyCalls!A1:G100"

You cannot specify:

• only a worksheet.

• columns for a specified worksheet.

If the worksheet name contains:

• spaces, you must enclose the name in single quotes, for example My Sheet is expressed as "'My Sheet'!A1:G10"

• single quotes, you must
  

  • enclose the whole name in single quotes

  • and use two single quotes (apostrophes).
    For example, Gary'sSheet is expressed as "'Gary''s Sheet'!A1:G100"

do (
    $rangeList = [],
    forEach($sheet, _worksheets,
        if ( listContains(["sheet A", "sheet'B"], $sheet),
            do(
                $safeSheet = "'" + replaceAll($sheet, "'", "''") + "'",
                $range = $safeSheet + "!A1:B10",
                addElement($rangeList, $range)
            )
        )
    ),
    $rangeList
)

Ignore Undefined Values

Available when File Type is Excel Spreadsheet. When importing the file:

Maximum Excel File Size (MB)

Specify the maximum file size, in megabytes, to process. This prevents upload attempts on excessively large files from slowing down the server.

The default is 0, which means no restriction. Specify a number greater than 0, for example 500, to prevent the file collector from processing files larger than this size.

File Password

Available if File Type is Excel Spreadsheet.

If you are reading a spreadsheet which is password protected, enter the password here so that the file can be unlocked.

Confirm Password

Available when File Type is Excel Spreadsheet.

If you are reading a spreadsheet which is password protected, confirm the password here so that the file can be unlocked.

Log Traffic

• Log File Collector Traffic : when ticked, PhixFlow always logs the number of records read from files by file collectors, whatever is set here.

Validate File Format

Tick to indicate you have completed the properties and the XML description matches the file format.

Stream Item Node

(Stream item is the old term for a record in a table.)

Enter an expression for a string or list of strings. Each string is a node name in the binary file that will generate an output record.

File Format Description

Enter an expression, using XML Binary File Grammar, describing the format of data in the file.

dd/MM/yy dd/MMM/yy dd/MM/yyyy dd/MMM/yyyy dd-MM-yy dd-MMM-yy dd-MM-yyyy dd-MMM-yyyy MM/dd/yy MMM/dd/yy MM/dd/yyyy MMM/dd/yyyy MM-dd-yy MMM-dd-yy MM-dd-yyyy MMM-dd-yyyy yyyyMMdd

dd/MM/yy HH:mm:ss dd/MMM/yy HH:mm:ss dd/MM/yyyy HH:mm:ss dd/MMM/yyyy HH:mm:ss dd-MM-yy HH:mm:ss dd-MMM-yy HH:mm:ss dd-MM-yyyy HH:mm:ss dd-MMM-yyyy HH:mm:ss MM/dd/yy HH:mm:ss MMM/dd/yy HH:mm:ss MM/dd/yyyy HH:mm:ss MMM/dd/yyyy HH:mm:ss MM-dd-yy HH:mm:ss MMM-dd-yy HH:mm:ss MM-dd-yyyy HH:mm:ss MMM-dd-yyyy HH:mm:ss yyyyMMdd.HHmmss

y

year

year

1996

M

month of year

month

Jul; 07

d

day of month

number

10

H

hour of day (0~23)

number

0

m

minute of hour

number

30

s

second of minute

number

55

true,yes,T,Y,1

false,no,F,N,0

_fileName

The name of the file.

_lineNumber

The line number of the record within the file it was read from.

The _lineNumber attribute is not available for file collectors of File Type is File Details Only.

_modifiedDate

The datetime of when the file was last modified.

The last modified time of a single file residing within a .gz or a .tgz container can not be determined by phixflow, instead the datetime of when the corresponding gz/tgz container was created will be returned.

_path

The full path to the file which is the result of concatenating the _rootDirectory and the _subDirectory values.

_rootDirectory

The root base directory (if specified) concatenated with the value evaluated in the Collectors 'Input Directory Expr' field.

_size

The size of the file in bytes.

The size of a single file residing within a .gz or a .tgz container can not be determined by phixflow, instead a size of -1 will be returned.

_subDirectory

The sub-directory relative to the _rootDirectory in which the corresponding file resides.

_worksheet

The name of the current worksheet of the Excel file. The _worksheet is only available if the File Type is Excel Spreadsheet.

_range

The Excel range expression that was used. The _range attribute is only available if the File Type is Excel Spreadsheet.

zip

".zip"

A zip archive created by either windows programs such as winzip etc or unix commands such as zip. For example, to create a compressed zip file called dailyCalls.zip that includes a single file called dailyCalls_20120918.csv:

zip dailyCalls.zip dailyCalls_20120918.csv

tar

".tar"

A tar archive created by the unix tar command. For example to create a tar file called dailyCalls.tar that includes a single file called dailyCalls_20120918.csv:

tar -cvf dailyCalls.tar dailyCalls_20120918.csv 

gz

".gz"

A gz archive created by the unix gzip command. For example, to create a compressed gz file called dailyCalls_20120918.csv.gz that includes a single file called dailyCalls_20120918.csv. 

 gzip dailyCalls_20120918.csv

The unix gzip command always assumes the named container has a single file of the same name contained within.

tgz

".tgz"

A joint tarred gz archive created by combining both the tar and gzip commands into a single command. For example, to create a compressed tgz file called dailyCalls.tgz that includes a single file called dailyCalls_20120918.csv:

tar -cvzf dailyCalls.tgz dailyCalls_20120918.csv

DailyCalls10.zip

/DailyCalls10.csv

".*Calls10.*"

".*

DailyCalls10.zip/DailyCalls10.csv

DailyCalls.tar

/subdir1/calls10.csv
/subdir1/calls20.csv
/subdir2/calls100.csv
/subdir2/calls200.csv
/subdir3/calls1000.csv
/subdir3/calls2000.csv

".*calls10.*"

".*

".*subdir2.*

DailyCalls.tar/subdir1/calls10.csv
DailyCalls.tar/subdir3/calls1000.csv

Outer.zip

/subdir1/calls10.csv
/subdir1/calls20.csv
/subdir1/Inner.zip/innerdir/calls100.csv
/subdir1/Inner.zip/subdir2/calls1000.csv

Note that Outer.zip contains a compressed zip file called Inner.zip

".*calls10.*"

".*subdir1.*

".*subdir2.*

Outer.zip/subdir1/calls10.csv
Outer.zip/subdir1/Inner.zip/innerdir/calls100.csv

Outer.tar.gz

/Outer.tar/subdir1/calls10.csv
/Outer.tar/innerdir/calls100.csv
/Outer.tar/subdir1/Inner.zip/innerdir/calls1000.csv

Note that Outer.tar.gz contains a tar container which in turn contains a compressed zip file called Inner.zip

".*calls10.*"

".*subdir1.*innerdir.*

Outer.tar.gz/Outer.tar/subdir1/Inner.zip/innerdir/calls1000.csv