...
Data Types that represent a rate, or amount of throughput are shown as the average rate per second e.g. the data type for the number of items generated in a stream is 'items/s' i.e. items per second. The value is calculated by dividing the total number or amount recorded in the sampling period and dividing it by the duration of the sampling period in seconds. The rate per second is shown rather than the absolute number so that if the sampling period is changed, the numbers (the rate per second) stay the same
Data Types include
| Data Type | Description |
|---|---|
| activities/s | Activities per second |
| seconds | A simple time value, e.g. a maximum wait time for a database statement to execute |
| seconds/s | Seconds per second. Stats that record times cumulative times (e.g. the total of internal wait times) are normalised per second. Where this applies to a single-threaded stats type, the value will always be between 0.0 and 1.0, where a value near 1.0 means that that part of the system is waiting nearly 100% of the time. |
| items | A number of items (records), e.g. the number of items in a pipe cache. |
| items/s | The number of items processed per second e.g. the number of items generated per second. |
| ops/s | A number of operations per second (e.g. databaes reads) |
| bytes | A total number of bytes, e.g. the number of bytes of Java memory used. |
| busy | The fraction of the time that a resource is busy e.g. the CPU utilisation. |
| task | A number of tasks |
Stats Type
...
Most Each Stats Types record a single Data Typetype of data. The data type is stored in the data as an indicator of the units of the data value for each stats type.
Activity
Activity stats record information about the high-level activities that the system was performing at any time. In general these are user-level actions e.g. running a Task Plan.
| Stats Type | Data Type | Description |
|---|---|---|
| activity.start | activities/s | The number of activities that started |
| activity.end | activities/s | The number of activities that finished |
| activity.time | seconds/s | The time spent running the activities |
System Performance
System stats record aspects of system behaviour.
| Stats Type | Data Type | Description |
|---|---|---|
| java.memory.free | bytes | The amount of java memory that is free |
| java.memory.used | bytes | The amount of java memory that is used |
| java.memory.total | bytes | The total amount of java memory |
| system.cpu | busy | The fraction of the time that the cpu is busy |
Database
Database stats record aggregate values for low-level database operations.
| Stats Type | Data Type | Description |
|---|---|---|
| data.exec.time | seconds/s | Time spent executing database statements. |
| data.read.time | seconds/s | Time spent reading from a database. |
| data.read.ops | ops/s | Number of database read operations / second |
| data.read.items | items/s | Number of items (records) read from database / second |
| data.write.time | seconds/s | Time spent writing to a database. |
| data.write.ops | ops/s | Number of database write operations / second |
| data.write.items | items/s | Number of items (records) written to database / second |
Data Generation
Data Generation stats record details of stream data generation.
| Stats Type | Data Type | Description |
|---|---|---|
| csf.create.time | seconds/s | Time spent creating candidate sets |
| csf.find.time | seconds/s | Time spent finding data for candidate sets |
| csf.process.time | seconds/s | Time spent processing candidate sets to create stream items. |
| generate.time | items/s | Number of items generated / second |
| generate.items | seconds/s | Time |
| spent per second generating items |
Data Output
Data Output stats record details of the output phase of Data Gneration, where items are written to an output queue so that they can be written out by one or more writers.
| Stats Type | Data Type | Description |
|---|---|---|
| output.enqueue.time | seconds/s | Time spent adding generated items to the output queue |
| output.dequeue.time | seconds/s | Time spent taking generated items from the output queue |
| output.dequeue.items | items/s | The number of items/s taken from the output queue |
| output.wait.time | seconds/s | Time spent waiting for the output writer |
| output.write.time | seconds/s | Time spent writing out items |
| output.write.items | items/s | The number of items written out / second |
| output.reject.items | items/s | The number of items rejected / second |
Pipes (Pull)
These stats types record various aspects of the behaviour of pull pipes
| Stats Type | Data Type | Description |
|---|---|---|
| pipe.pull.idle.time | seconds/s | Time spent idle |
| pipe.pull.prepare.time | seconds/s | Time spent preparing (creating pipe candidate sets) |
| pipe.pull.prepared.time | seconds/s | Time spent after preparation |
| pipe.pull.process.time | seconds/s | Time spent creating candidate sets |
| pipe.pull.processed.time | seconds/s | Time spent after processing |
| pipe.pull.read.time | seconds/s | Time spent reading |
| pipe.pull.readresponse.time | seconds/s | Time spent processing read responses |
| pipe.pull.submitted.time | seconds/s | Time spent submitting read requests |
Pipes (Lookups)
These stats types record various aspects of the behaviour of lookup pipes.
| Stats Type | Data Type | Description |
|---|---|---|
| lookup.added.items | items/s | The number of items added to a pipe cache per second |
| lookup.clash.ops | ops | The number of lookups where another process was reading data for the same lookup |
| lookup.miss.ops | ops/s | The number of lookups per second not satisfied by data already in the pipe cache |
| lookup.removed.items | items/s | The number of items removed from a pipe cache per second |
| lookup.size.items | items | The number of items in a pipe cache |
| pipe.lookup.ops | ops/s | The number of lookups / second |
| pipe.lookup.time | seconds/s | The time spent doing lookups / second |
Activity Statistics
Activity Statistics are recorded from TaskPlans executed, Actions triggered.
...
...
...
...
TaskPlan/Action ID
...
{Number of tasks started in this timeslot. The timeslot is set in the System Configuration > System Tuning Tab
Stats DB Logging Seconds field}
...
TaskPlan/Action ID
...
Stats DB Logging Seconds field}
...
TaskPlan/Action ID
...
TaskPlan/Action ID
...
Stats DB Logging Seconds field}
Stream Statistics
This is a record of data volumes written to Streams and Exporters
...
...
...
...
TaskPlan/Action ID
...
TaskPlan/Action ID
...
TaskPlan/Action ID
...
insertVolumeBytes
(bytes)
...
TaskPlan/Action ID
...
insertTimeTotal
(seconds)
...
TaskPlan/Action ID
...
TaskPlan/Action ID
...
TaskPlan/Action ID
...
updateVolumeBytes
(bytes)
...
TaskPlan/Action ID
...
updateTimeTotal
(seconds)
...
The timeslot is set in the System Configuration > System Tuning Tab
Stats DB Logging Seconds field
Pipe Statistics
This is a record of Pipe Read Statistics
...
...
...
...
TaskPlan/Action ID
...
TaskPlan/Action ID
...
readTime
(seconds)
...
TaskPlan/Action ID
...
readTimeMax
(seconds)
...
TaskPlan/Action ID
...
This is a record of Pipe Queue Statistics
...
...
...
...
TaskPlan/Action ID
...
sumEnqueueTime
(seconds)
...
TaskPlan/Action ID
...
TaskPlan/Action ID
...
sumDequeueTime
(seconds)
...
TaskPlan/Action ID
...
Work Queues
PhixFlow maintains a number of queues for asynchronous task work / task processing.
These statistics record the occupancy levels of these queues, comprising for each queue the number of tasks queued (waiting) + the number running.
| Stats Type | Data Type | Description |
|---|---|---|
| workqueue.cspt.size | tasks | The number of tasks in the CSPT work queue |
| workqueue.dgr.size | tasks | The number of tasks in the DGR work queue |
| workqueue.pdp.size | tasks | The number of tasks in the PDP work queue |
| workqueue.prepare.size | tasks | The number of tasks in the PREPARE work queue |
| workqueue.read.size | tasks | The number of tasks in the READ work queue |
| workqueue.write.size | tasks | The number of tasks in the WRITE work queue |
| workqueue.other.size | tasks | The number of tasks in the OTHER work queue |
| workqueue.export.size | tasks | The number of tasks in the EXPORT work queue |
| workqueue.view.size | tasks | The number of tasks in the VIEW work queue |
kup Statistics
STATS_TYPE | INITIATOR_TYPE | INITIATOR_ID | INITIATOR_NAME | CONTEXT_TYPE | CONTEXT_ID | CONTEXT_NAME | FULL_CONTEXT | DATA_TYPE | DATA_VALUE |
|---|---|---|---|---|---|---|---|---|---|
| Pipe | TaskPlan, Action | TaskPlan/Action ID | TaskPlan/Action name | Pipe | Pipe ID | Pipe Name | TaskPlan/Action Name.Stream Name.pipe Name | cacheSize | {Size of lookup pipe cache} |
| Pipe | TaskPlan, Action | TaskPlan/Action ID | TaskPlan/Action name | Pipe | Pipe ID | Pipe Name | TaskPlan/Action Name.Stream Name.pipe Name | addedToCache | {Lookup pipe added to cache} |
| Pipe | TaskPlan, Action | TaskPlan/Action ID | TaskPlan/Action name | Pipe | Pipe ID | Pipe Name | TaskPlan/Action Name.Stream Name.pipe Name | removedFromCache | {Lookup pipe removed from cache} |
| Pipe | TaskPlan, Action | TaskPlan/Action ID | TaskPlan/Action name | Pipe | Pipe ID | Pipe Name | TaskPlan/Action Name.Stream Name.pipe Name | cacheHits | {Count a read done from cache} |
| Pipe | TaskPlan, Action | TaskPlan/Action ID | TaskPlan/Action name | Pipe | Pipe ID | Pipe Name | TaskPlan/Action Name.Stream Name.pipe Name | cacheMisses | {Count a read not yet from the cache} |
| Pipe | TaskPlan, Action | TaskPlan/Action ID | TaskPlan/Action name | Pipe | Pipe ID | Pipe Name | TaskPlan/Action Name.Stream Name.pipe Name | cacheParallelMisses | {Total reads from cache by two simulatneous processes} |
...