PhixFlow generates processing statistics which are saved to the database. These statics are for use by administrators and designers who are tuning for performance.
See System Configuration for how to enable and manage stats generation.
Processing Statistics
The following statistics are currently recorded and available as data records in the STATS table. (See STATS record definition)
The performance statistics are classified by 3 major dimensions:
Initiator
The initiator is the object representing the user activity that caused the system activity.
Initiators include
- (Running an) Action
- (Running a) Task Plan
- (Running a) Stream
- (Viewing data for a) Stream View
- System, for internal activities and things that can't be allocated to any specific cause e.g. memory and CPU usage.
Where one initiator could include another, the activity is recorded against the initiator closest to the user
e.g. the user runs an Action which runs a Task Plan - the initiator is the Action.
e.g. the user runs the same Task Plan directly - the initiator is the Task Plan
With the exception of System, the initiator is identified by name, type and id.
Context
The context is the most detailed object to which the activity can be attributed.
Contexts include
- Stream Action
- Task Plan
- Stream
- Pipe
- File / Database / Http Collector
- File / Database / Http Exporter
- System
With the exception of System, the context is identified by name, full name, type and id.
The full name is a dot-separated list of objects from the highest level to the lowest, ending in the context object.
Stats Type / Data Type
Stats Type is the specific aspect of system behaviour being measured.
Data Type
Data Type contains the units of the measurement.
Data types that represent a snapshot value are shown as the plural unit e.g. the Data Type for the amount of Java memory used is 'bytes'.
Data Types that represent a rate, or amount of throughput are shown as the average rate per second e.g. the data type for the number of items generated in a stream is 'items/s' i.e. items per second. The value is calculated by dividing the total number or amount recorded in the sampling period and dividing it by the duration of the sampling period in seconds. The rate per second is shown rather than the absolute number so that if the sampling period is changed, the numbers (the rate per second) stay the same
Data Types include
| Data Type | Description |
|---|---|
| activities/s | Activities per second |
| seconds | A simple time value, e.g. a maximum wait time for a database statement to execute |
| seconds/s | Seconds per second. Stats that record times cumulative times (e.g. the total of internal wait times) are normalised per second. Where this applies to a single-threaded stats type, the value will always be between 0.0 and 1.0, where a value near 1.0 means that that part of the system is waiting nearly 100% of the time. |
| items | A number of items (records), e.g. the number of items in a pipe cache. |
| items/s | The number of items processed per second e.g. the number of items generated per second. |
| ops/s | A number of operations per second (e.g. databaes reads) |
| bytes | A total number of bytes, e.g. the number of bytes of Java memory used. |
| busy | The fraction of the time that a resource is busy e.g. the CPU utilisation. |
Stats Type + Data Type
Most Stats Types record a single Data Type
| Stats Type | Data Type | Description |
|---|---|---|
| activity.start | activities/s | The number of activities that started |
| activity.end | activities/s | The number of activities that finished |
| activity.time | seconds/s | The time spent running the activities |
| csf.create | seconds/s | Time spent creating candidate sets |
| csf.find | seconds/s | Time spent finding data for candidate sets |
| csf.process | seconds/s | Time spent processing candidate sets to create stream items. |
| data.exec | seconds/s | Time spent executing database statements. |
| data.read | seconds/s | Time spent reading from a database. |
| data.read | ops/s | Number of database read operations / second |
| data.read | items/s | Number of items (records) read from database / second |
| data.write | seconds/s | Time spent writing to a database. |
| data.write | ops/s | Number of database write operations / second |
| data.write | items/s | Number of items (records) written to database / second |
| generate | items/s | Number of items generated / second |
| generate | seconds/s | Time spend per second generating items |
| output.enqueue | seconds/s | Time spent adding generated items to the output queue |
| output.dequeue | seconds/s | Time spent taking generated items from the output queue |
| output.dequeue | items/s | The number of items/s taken from the output queue |
| output.wait | seconds/s | Time spent waiting for the output writer |
| output.write | seconds/s | Time spent writing out items |
| output.write | items/s | The number of items written out / second |
| output.reject | items/s | The number of items rejected / second |
| java.memory.free | bytes | The amount of java memory that is free |
| java.memory.used | bytes | The amount of java memory that is used |
| java.memory.total | bytes | The total amount of java memory |
| lookup.added | items/s | The number of items added to a pipe cache per second |
| lookup.clash | ops | The number of lookups where another process was reading data for the same lookup |
| lookup.miss | ops/s | The number of lookups per second not satisfied by data already in the pipe cache |
| lookup.removed | items/s | The number of items removed from a pipe cache per second |
| lookup.size | items | The number of items in a pipe cache |
| pipe.lookup | ops/s | The number of lookups / second |
| pipe.lookup | seconds/s | The time spent doing lookups / second |
| pipe.pull.idle | ||
| pipe.pull.prepare | ||
| pipe.pull.prepared | ||
| pipe.pull.process | ||
| pipe.pull.processed | ||
| pipe.pull.read | ||
| pipe.pull.readresponse | ||
| pipe.pull.submitted | ||
| system.cpu | busy | The fraction of the time that the cpu is busy |
| workqueue.cspt.size | ||
Activity Statistics
Activity Statistics are recorded from TaskPlans executed, Actions triggered.
STATS_TYPE | INITIATOR_TYPE | INITIATOR_ID | INITIATOR_NAME | CONTEXT_TYPE | CONTEXT_ID | CONTEXT_NAME | FULL_CONTEXT | DATA_TYPE | DATA_VALUE |
|---|---|---|---|---|---|---|---|---|---|
| Activity | TaskPlan, Action | TaskPlan/Action ID | TaskPlan/Action name | TaskPlan/Action Name | numStarted | {Number of tasks started in this timeslot. The timeslot is set in the System Configuration > System Tuning Tab Stats DB Logging Seconds field} | |||
| Activity | TaskPlan, Action | TaskPlan/Action ID | TaskPlan/Action name | TaskPlan/Action Name | numActive | {Number of tasks still running at the end of this timeslot. The timeslot is set in the System Configuration > System Tuning Tab - Stats DB Logging Seconds field} | |||
| Activity | TaskPlan, Action | TaskPlan/Action ID | TaskPlan/Action name | TaskPlan/Action Name | numEnded | {Number of tasks ended in this timeslot. The timeslot is set in the System Configuration > System Tuning Tab - Stats DB Logging Seconds field} | |||
| Activity | TaskPlan, Action | TaskPlan/Action ID | TaskPlan/Action name | TaskPlan/Action Name | sumDuration (seconds) | {Total duration of task ending in this timeslot. The timeslot is set in the System Configuration > System Tuning Tab - Stats DB Logging Seconds field} |
Stream Statistics
This is a record of data volumes written to Streams and Exporters
STATS_TYPE | INITIATOR_TYPE | INITIATOR_ID | INITIATOR_NAME | CONTEXT_TYPE | CONTEXT_ID | CONTEXT_NAME | FULL_CONTEXT | DATA_TYPE | DATA_VALUE |
|---|---|---|---|---|---|---|---|---|---|
| Stream | TaskPlan, Action | TaskPlan/Action ID | TaskPlan/Action name | Stream | Stream ID | Stream Name | TaskPlan/Action Name.Stream Name | insertCount | {Count of database insert batches} |
| Stream | TaskPlan, Action | TaskPlan/Action ID | TaskPlan/Action name | Stream | Stream ID | Stream Name | TaskPlan/Action Name.Stream Name | insertVolume | {Volume of individual records inserted} |
| Stream | TaskPlan, Action | TaskPlan/Action ID | TaskPlan/Action name | Stream | Stream ID | Stream Name | TaskPlan/Action Name.Stream Name | insertVolumeBytes (bytes) | {Volume of records bytes inserted} |
| Stream | TaskPlan, Action | TaskPlan/Action ID | TaskPlan/Action name | Stream | Stream ID | Stream Name | TaskPlan/Action Name.Stream Name | insertTimeTotal (seconds) | {Time taken to insert records} |
| Stream | TaskPlan, Action | TaskPlan/Action ID | TaskPlan/Action name | Stream | Stream ID | Stream Name | TaskPlan/Action Name.Stream Name | updateCount | {Count of database updated batches} |
| Stream | TaskPlan, Action | TaskPlan/Action ID | TaskPlan/Action name | Stream | Stream ID | Stream Name | TaskPlan/Action Name.Stream Name | updateVolume | {Volume of individual records updated} |
| Stream | TaskPlan, Action | TaskPlan/Action ID | TaskPlan/Action name | Stream | Stream ID | Stream Name | TaskPlan/Action Name.Stream Name | updateVolumeBytes (bytes) | {Volume of records bytes updated} |
| Stream | TaskPlan, Action | TaskPlan/Action ID | TaskPlan/Action name | Stream | Stream ID | Stream Name | TaskPlan/Action Name.Stream Name | updateTimeTotal (seconds) | {Time taken to update records} |
The timeslot is set in the System Configuration > System Tuning Tab
Stats DB Logging Seconds field
Pipe Statistics
This is a record of Pipe Read Statistics
STATS_TYPE | INITIATOR_TYPE | INITIATOR_ID | INITIATOR_NAME | CONTEXT_TYPE | CONTEXT_ID | CONTEXT_NAME | FULL_CONTEXT | DATA_TYPE | DATA_VALUE |
|---|---|---|---|---|---|---|---|---|---|
| Pipe | TaskPlan, Action | TaskPlan/Action ID | TaskPlan/Action name | Pipe | Pipe ID | Pipe Name | TaskPlan/Action Name.Stream Name.pipe Name | readCount | {Numer of reads} |
| Pipe | TaskPlan, Action | TaskPlan/Action ID | TaskPlan/Action name | Pipe | Pipe ID | Pipe Name | TaskPlan/Action Name.Stream Name.pipe Name | readTime (seconds) | {Time taken to read} |
| Pipe | TaskPlan, Action | TaskPlan/Action ID | TaskPlan/Action name | Pipe | Pipe ID | Pipe Name | TaskPlan/Action Name.Stream Name.pipe Name | readTimeMax (seconds) | {Maximum read time} |
| Pipe | TaskPlan, Action | TaskPlan/Action ID | TaskPlan/Action name | Pipe | Pipe ID | Pipe Name | TaskPlan/Action Name.Stream Name.pipe Name | readRecCount | {Number of records read} |
This is a record of Pipe Queue Statistics
STATS_TYPE | INITIATOR_TYPE | INITIATOR_ID | INITIATOR_NAME | CONTEXT_TYPE | CONTEXT_ID | CONTEXT_NAME | FULL_CONTEXT | DATA_TYPE | DATA_VALUE |
|---|---|---|---|---|---|---|---|---|---|
| Pipe | TaskPlan, Action | TaskPlan/Action ID | TaskPlan/Action name | Pipe | Pipe ID | Pipe Name | TaskPlan/Action Name.Stream Name.pipe Name | sumEnqueueTime (seconds) | {Time take to add to queue} |
| Pipe | TaskPlan, Action | TaskPlan/Action ID | TaskPlan/Action name | Pipe | Pipe ID | Pipe Name | TaskPlan/Action Name.Stream Name.pipe Name | enqueueCount | {Pipe added to queue} |
| Pipe | TaskPlan, Action | TaskPlan/Action ID | TaskPlan/Action name | Pipe | Pipe ID | Pipe Name | TaskPlan/Action Name.Stream Name.pipe Name | sumDequeueTime (seconds) | {Time take to remove from queue} |
| Pipe | TaskPlan, Action | TaskPlan/Action ID | TaskPlan/Action name | Pipe | Pipe ID | Pipe Name | TaskPlan/Action Name.Stream Name.pipe Name | dequeueCount | {Pipe removed from queue} |
This is a record of Pipe Lookup Statistics
STATS_TYPE | INITIATOR_TYPE | INITIATOR_ID | INITIATOR_NAME | CONTEXT_TYPE | CONTEXT_ID | CONTEXT_NAME | FULL_CONTEXT | DATA_TYPE | DATA_VALUE |
|---|---|---|---|---|---|---|---|---|---|
| Pipe | TaskPlan, Action | TaskPlan/Action ID | TaskPlan/Action name | Pipe | Pipe ID | Pipe Name | TaskPlan/Action Name.Stream Name.pipe Name | cacheSize | {Size of lookup pipe cache} |
| Pipe | TaskPlan, Action | TaskPlan/Action ID | TaskPlan/Action name | Pipe | Pipe ID | Pipe Name | TaskPlan/Action Name.Stream Name.pipe Name | addedToCache | {Lookup pipe added to cache} |
| Pipe | TaskPlan, Action | TaskPlan/Action ID | TaskPlan/Action name | Pipe | Pipe ID | Pipe Name | TaskPlan/Action Name.Stream Name.pipe Name | removedFromCache | {Lookup pipe removed from cache} |
| Pipe | TaskPlan, Action | TaskPlan/Action ID | TaskPlan/Action name | Pipe | Pipe ID | Pipe Name | TaskPlan/Action Name.Stream Name.pipe Name | cacheHits | {Count a read done from cache} |
| Pipe | TaskPlan, Action | TaskPlan/Action ID | TaskPlan/Action name | Pipe | Pipe ID | Pipe Name | TaskPlan/Action Name.Stream Name.pipe Name | cacheMisses | {Count a read not yet from the cache} |
| Pipe | TaskPlan, Action | TaskPlan/Action ID | TaskPlan/Action name | Pipe | Pipe ID | Pipe Name | TaskPlan/Action Name.Stream Name.pipe Name | cacheParallelMisses | {Total reads from cache by two simulatneous processes} |
Database Statistics Record Structure
The table structure for STATS is as follows:
Member | Class | Description |
|---|---|---|
| stats_type | enum | Statistics Type: E.g. 'Activity, Pipe, PipeLookup, PipeQueue, Stream' |
| fromDtm | Datetime | The start time of the period that this measure refers to |
| toDtm | Datetime | The end time of the period that this measure refers to |
| initiator_type | String | The object initiating the activity - eg. "TaskPlan", or "Action" |
| initiator_id | Uid | The ID of the initiating TaskPlan or Action / other object. |
| initiator_name | String | The name of the initiating TaskPlan or Action / other object. |
| context_type | String | The activity triggered by the initiator. eg "Stream" |
| context_id | Uid | e.g. The "Streams" ID that Analysis was run on. |
| context_name | String | eg. The name of the stream that Analysis was run on.` |
| full_context | String | This a dotted notation indicating the full context. eg. TaskPlan1.Stream X.in |
| data_type | String | The type of statistics data being recorded for this stats type. Eg. totalRecordsWritten, totalWriteTime. |
| data_value | Double | The value of the statistic being recorded for the given data_type. |