PhixFlow generates processing statistics which are saved to the database. These statics are for use by administrators and designers who are tuning for performance.

See System Configuration for how to enable and manage stats generation.

Processing Statistics

The following statistics are currently recorded and available as data records in the STATS table. (See STATS record definition)

Activity Statistics

Activity Statistics are recorded from TaskPlans executed, Actions triggered.

STATS_TYPE	INITIATOR_TYPE	INITIATOR_ID	INITIATOR_NAME	FULL_CONTEXT	DATA_TYPE	DATA_VALUE
Activity	TaskPlan, Action	TaskPlan/Action ID	TaskPlan/Action name	TaskPlan/Action Name	numStarted	{Number of tasks started in this timeslot. The timeslot is set in the System Configuration > System Tuning Tab Stats DB Logging Seconds field}
Activity	TaskPlan, Action	TaskPlan/Action ID	TaskPlan/Action name	TaskPlan/Action Name	numActive	{Number of tasks still running at the end of this timeslot. The timeslot is set in the System Configuration > System Tuning Tab - Stats DB Logging Seconds field}
Activity	TaskPlan, Action	TaskPlan/Action ID	TaskPlan/Action name	TaskPlan/Action Name	numEnded	{Number of tasks ended in this timeslot. The timeslot is set in the System Configuration > System Tuning Tab - Stats DB Logging Seconds field}
Activity	TaskPlan, Action	TaskPlan/Action ID	TaskPlan/Action name	TaskPlan/Action Name	sumDuration (seconds)	{Total duration of task ending in this timeslot. The timeslot is set in the System Configuration > System Tuning Tab - Stats DB Logging Seconds field}

Stream Statistics

This is a record of data volumes written to Streams and Exporters

STATS_TYPE	INITIATOR_TYPE	INITIATOR_ID	INITIATOR_NAME	CONTEXT_TYPE	CONTEXT_ID	CONTEXT_NAME	FULL_CONTEXT	DATA_TYPE	DATA_VALUE
Stream	TaskPlan, Action	TaskPlan/Action ID	TaskPlan/Action name	Stream	Stream ID	Stream Name	TaskPlan/Action Name.Stream Name	insertCount	{Count of database insert batches}
Stream	TaskPlan, Action	TaskPlan/Action ID	TaskPlan/Action name	Stream	Stream ID	Stream Name	TaskPlan/Action Name.Stream Name	insertVolume	{Volume of individual records inserted}
Stream	TaskPlan, Action	TaskPlan/Action ID	TaskPlan/Action name	Stream	Stream ID	Stream Name	TaskPlan/Action Name.Stream Name	insertVolumeBytes (bytes)	{Volume of records bytes inserted}
Stream	TaskPlan, Action	TaskPlan/Action ID	TaskPlan/Action name	Stream	Stream ID	Stream Name	TaskPlan/Action Name.Stream Name	insertTimeTotal (seconds)	{Time taken to insert records}
Stream	TaskPlan, Action	TaskPlan/Action ID	TaskPlan/Action name	Stream	Stream ID	Stream Name	TaskPlan/Action Name.Stream Name	updateCount	{Count of database updated batches}
Stream	TaskPlan, Action	TaskPlan/Action ID	TaskPlan/Action name	Stream	Stream ID	Stream Name	TaskPlan/Action Name.Stream Name	updateVolume	{Volume of individual records updated}
Stream	TaskPlan, Action	TaskPlan/Action ID	TaskPlan/Action name	Stream	Stream ID	Stream Name	TaskPlan/Action Name.Stream Name	updateVolumeBytes (bytes)	{Volume of records bytes updated}
Stream	TaskPlan, Action	TaskPlan/Action ID	TaskPlan/Action name	Stream	Stream ID	Stream Name	TaskPlan/Action Name.Stream Name	updateTimeTotal (seconds)	{Time taken to update records}

The timeslot is set in the System Configuration > System Tuning Tab

Stats DB Logging Seconds field

Pipe Statistics

This is a record of Pipe Read Statistics

STATS_TYPE	INITIATOR_TYPE	INITIATOR_ID	INITIATOR_NAME	CONTEXT_TYPE	CONTEXT_ID	CONTEXT_NAME	FULL_CONTEXT	DATA_TYPE	DATA_VALUE
Pipe	TaskPlan, Action	TaskPlan/Action ID	TaskPlan/Action name	Pipe	Pipe ID	Pipe Name	TaskPlan/Action Name.Stream Name.pipe Name	readCount	{Numer of reads}
Pipe	TaskPlan, Action	TaskPlan/Action ID	TaskPlan/Action name	Pipe	Pipe ID	Pipe Name	TaskPlan/Action Name.Stream Name.pipe Name	readTime (seconds)	{Time taken to read}
Pipe	TaskPlan, Action	TaskPlan/Action ID	TaskPlan/Action name	Pipe	Pipe ID	Pipe Name	TaskPlan/Action Name.Stream Name.pipe Name	readTimeMax (seconds)	{Maximum read time}
Pipe	TaskPlan, Action	TaskPlan/Action ID	TaskPlan/Action name	Pipe	Pipe ID	Pipe Name	TaskPlan/Action Name.Stream Name.pipe Name	readRecCount	{Number of records read}

This is a record of Pipe Queue Statistics

STATS_TYPE	INITIATOR_TYPE	INITIATOR_ID	INITIATOR_NAME	CONTEXT_TYPE	CONTEXT_ID	CONTEXT_NAME	FULL_CONTEXT	DATA_TYPE	DATA_VALUE
Pipe	TaskPlan, Action	TaskPlan/Action ID	TaskPlan/Action name	Pipe	Pipe ID	Pipe Name	TaskPlan/Action Name.Stream Name.pipe Name	sumEnqueueTime (seconds)	{Time take to add to queue}
Pipe	TaskPlan, Action	TaskPlan/Action ID	TaskPlan/Action name	Pipe	Pipe ID	Pipe Name	TaskPlan/Action Name.Stream Name.pipe Name	enqueueCount	{Pipe added to queue}
Pipe	TaskPlan, Action	TaskPlan/Action ID	TaskPlan/Action name	Pipe	Pipe ID	Pipe Name	TaskPlan/Action Name.Stream Name.pipe Name	sumDequeueTime (seconds)	{Time take to remove from queue}
Pipe	TaskPlan, Action	TaskPlan/Action ID	TaskPlan/Action name	Pipe	Pipe ID	Pipe Name	TaskPlan/Action Name.Stream Name.pipe Name	dequeueCount	{Pipe removed from queue}

This is a record of Pipe Lookup Statistics

STATS_TYPE	INITIATOR_TYPE	INITIATOR_ID	INITIATOR_NAME	CONTEXT_TYPE	CONTEXT_ID	CONTEXT_NAME	FULL_CONTEXT	DATA_TYPE	DATA_VALUE
Pipe	TaskPlan, Action	TaskPlan/Action ID	TaskPlan/Action name	Pipe	Pipe ID	Pipe Name	TaskPlan/Action Name.Stream Name.pipe Name	cacheSize	{Size of lookup pipe cache}
Pipe	TaskPlan, Action	TaskPlan/Action ID	TaskPlan/Action name	Pipe	Pipe ID	Pipe Name	TaskPlan/Action Name.Stream Name.pipe Name	addedToCache	{Lookup pipe added to cache}
Pipe	TaskPlan, Action	TaskPlan/Action ID	TaskPlan/Action name	Pipe	Pipe ID	Pipe Name	TaskPlan/Action Name.Stream Name.pipe Name	removedFromCache	{Lookup pipe removed from cache}
Pipe	TaskPlan, Action	TaskPlan/Action ID	TaskPlan/Action name	Pipe	Pipe ID	Pipe Name	TaskPlan/Action Name.Stream Name.pipe Name	cacheHits	{Count a read done from cache}
Pipe	TaskPlan, Action	TaskPlan/Action ID	TaskPlan/Action name	Pipe	Pipe ID	Pipe Name	TaskPlan/Action Name.Stream Name.pipe Name	cacheMisses	{Count a read not yet from the cache}
Pipe	TaskPlan, Action	TaskPlan/Action ID	TaskPlan/Action name	Pipe	Pipe ID	Pipe Name	TaskPlan/Action Name.Stream Name.pipe Name	cacheParallelMisses	{Total reads from cache by two simulatneous processes}

Database Statistics Record Structure

The table structure for STATS is as follows:

Member	Class	Description
stats_type	enum	Statistics Type: E.g. 'Activity, Pipe, PipeLookup, PipeQueue, Stream'
fromDtm	Datetime	The start time of the period that this measure refers to
toDtm	Datetime	The end time of the period that this measure refers to
initiator_type	String	The object initiating the activity - eg. "TaskPlan", or "Action"
initiator_id	Uid	The ID of the initiating TaskPlan or Action / other object.
initiator_name	String	The name of the initiating TaskPlan or Action / other object.
context_type	String	The activity triggered by the initiator. eg "Stream"
context_id	Uid	e.g. The "Streams" ID that Analysis was run on.
context_name	String	eg. The name of the stream that Analysis was run on.`
full_context	String	This a dotted notation indicating the full context. eg. TaskPlan1.Stream X.in
data_type	String	The type of statistics data being recorded for this stats type. Eg. totalRecordsWritten, totalWriteTime.
data_value	Double	The value of the statistic being recorded for the given data_type.

fromDtm and toDtm dates can be identical on a record

They are not the same as the fromDTM and toDTM datetimes on streamsets - they give the start and end times for a stats logging window. The duration of the stats logging window is set in the system configuration.

fromDtm is when the monitor is opened.
toDtm is when the statistic is recorded/written to the stats table.

If a task finishes inside of a stats logging window, then the datetimes will be the same.