Introduction

PhixFlow version 7.6.0 contains improvements in the areas of performance, usability improvements, and bug fixes, as listed below.

Contents

Installation and Upgrade

To install a new PhixFlow instance, follow the standard installation instructions.

To upgrade an existing PhixFlow instance:

1. First review Compatibility Guide and Upgrade Planning to check any changing requirements for database versions etc.
2. Follow the Standard Upgrade Instructions plus any special instructions for all intermediate releases listed in Compatibility Guide and Upgrade Planning.

Special notes

Discontinued Features

Graphical User Interface

The original flash-based Graphical User Interface (GUI) is no longer available.

Modelling Features

The following features are no longer supported at this release:

Alarms & Alarm Generators

WorkFlows

The Control Framework

Measurements

Database

We no longer support the following database versions. Please contact the support desk if you are currently using any of these versions and are not planning to change by then.

Oracle 11g

SQL Server 2012

MariaDB 10.0

Features and Improvements

Installation

Templates

There is a set of templates (Layout Templates, Styles, Palettes) provided with every release which needs to be imported during the install/update process. These files should be loaded as part of any fresh install, and they will create a new set of objects on each load. These objects can then be referenced, modified and moved from instance to instance like any user-configured objects.

phixflow-datasource.xml

PhixFlow now determines the database dialect automatically - it is no longer necessary to specify the dialect in the phixflow-datasource.xml file.

SQL Server

The PhixFlow install package now correctly includes both 32-bit and 64-bit versions of sqljdbc_auth.dll. One or other of these files is required to configure PhixFlow to use SQL Server integrated authentication.

Security

Web Security

Various web server security improvements.

User Group Administrators

It is now possible to configure certain users to be administrators for specific user groups. These group administrators are allowed to manage which users can access their groups but not other groups.

This feature makes use of a new privilege, Add and Delete Users from a User Group. See User Group Access Permissions for details.

Attribute-level Access

It is possible to restrict access to individual stream attributes based on user group membership (like the current stream-level access based on user-groups).

Users / Login

Users

User security questions are now not visible for other users

Upper case letters and spaces are not allowed in the username entry field of the user editor.

Login Dialog Improvements

Minor improvements to the login form.

Failed Login Reporting

Removed stack trace from login failure debug log message.

Active Directory

Added an additional field to show the active directory group with the instance name inserted when editing user groups or viewing a user.

Graphical User Interface

The HTML graphical user interface is now the only interface, and users will be taken to it automatically on login.

Repository

Repository objects are now sorted alphabetically

Cut-and-Paste from grids

You can now highlight the values in a grid cell and copy using Ctrl-C.

Toast Messages

Messages (info, warnings, errors) are now displayed as toast notifications. If the user clicks on the message, it's is displayed in the big window (as usual).

Browser Window / Tab

The browser tab title is now set to a combination of the application name and instance name.

Added the 'Load Application in a new tab' and the 'Load Application in a new window' options to applications list context menu.

Double-clicking a grid row

We no longer show the default stream item view when double clicking a grid row since the user can now double click to highlight the contents of the cell for copying. Any specific double click action defined on the view is still triggered however.

Text Editing

By default a tab in an expression field is now replaced by 4 rather than three spaces.

Form Input

Improved error message when user input to a form is too long.

Many minor visual and/or ergonomic changes.

File Download

Added a pop-up window which allows the user to download a prepared file. The download will not be blocked by pop-up blockers. This applies to all file download situations e.g. exporting configuration data, exporting stream data.

Design Mode

Applications / Packages

Applications and Packages are referred to collectively as Containers.

You can now right click on an item in the repository browser and select the option "Remove from container". This will remove the selected item from its current package or application. It will still be visible in the Full Repository list however.

On Analysis Models, we now tag objects with a 'container' icon if the object belongs to an Application different than the Model's Application.

In addition a number of changes have been made to improve the user's awareness of which objects are in which containers.

Disabled the zoom functionality from Analysis Models. Added navigation arrows instead.

Data Modelling

Pipes

The user can now delete pipes from the Stream Details' list of inputs

Streams

It is possible to flag a stream as to be updated in a single database transaction.

Stream Attributes

Many restrictions on changing a Stream Attribute from one data type to another when the Stream contains data have been lifted for MySQL databases - these restrictions were already largely not applied for Oracle and SQL Server. E.g. is is now possible to convert an Integer attribute to a Float or String, and visa versa, although a String to Float conversion will then only be successful if all of the existing values are valid representations of floating-point numbers.

It is now not possible to create/update a Stream Attribute with name starting with a numeric character

Stream Rollback

Improvements to dialogs and progress / error messages.

You can now right click on a stream in the repository browser to roll it back.

Attribute Expressions

The jumpTo function can now use a string as a parameter.

The toString function (or conversion of float stream attributes to strings) will not give the result in scientific form and will instead return the number up to 6 decimal points.

_error can now be used in attribute expressions. The variable refers to the last error generated by a try function.

Application Building

Dashboards

Dashboards now open locked by default even in design mode. Buttons on dashboards will now carry out updates if the dashboard is locked even in design mode. The previous locked status is remembered when switching back to design mode from app mode.

Components

There are two new internal variables that can be used in formatting rules for layout components: _edited (true if the form has unsaved data) and _selected (true if the current item is selected - mostly useful for card components to highlight selected cards).

You can now drag and drop layout components from one area to another in the same form from within the repository browser.

There is now a variable called _edited which can be used in formatting rules of layout components to change the style if the form contains unsaved edits.

Card Component

There is a new Card component. It displays a configured form for one record of the linked Stream.

To create a Card Component the user needs to drop a Stream onto an existing form. Then drag some components onto a Card (fields etc).

Card Components have separate header and body styles.

The user can now use the following variables while defining formatting rules on a Card Component.

• _current (refer to the current card's record)
• _selected (true if the current record is selected)
• _edited (true if the current record is edited)
• _itemNumber (index of the current record)

This allows to define a different style for each card, depending on the Card's values.

Card Container Template is now included in both default palettes.

Formatting Rules

Formatting rules and css properties now support read-only flag

String Components

String Components are now marked as 'multi-line' (or not) rather than by setting a number of lines.

The user can now set date format on dynamic text components

Actions

Improved error message for invalid URL expressions on Stream Actions missing quotes around string.

'Action in progress' indicators will now correctly show Action's Display Name

The "prevent parallel processing" flag is set to true by default on new stream actions.

There is a new 'Edited items' option on a Stream Action filter. The action will apply only edited items (i.e. on forms, cards etc.)

The user can specify a target which the action will be applied to. It means that the action can be applied to other views, not only to the parent view / form.

The user can now use a '_grid' variable in Action expressions which refers to an item currently selected on the Action's target grid.

The user can now drag an action onto a Stream View or a Dashboard header in order to create a button which fires the action

Dashboard rules can be copied to another action by drag and drop

If the user does not have permission to run the default action on a view they no longer see an error message when double clicking the row. An information message is added to a log file in the console however indicating that an attempt was made to run the action but it was disallowed since the user did not have the appropriate permissions.

Action validation will be now performed in design mode

Filters

Improved conversion of integer, float and string attributes when comparing different types in filters.

It is now possible to use _context variable in drop-downs background filters.

The user can now drag a context attribute onto dashboard to create a driving background filter

Views

When you double click on a streamset to see the contents of the stream the view now includes superseded items.

The user can now drop Stream Attributes onto a grid (not only on headers) in order to add a column to the grid.

When you drop a stream onto a dashboard area to create a new stream view, as well as opening the stream view editor to allow you to change the stream view properties the stream attribute popup list will also appear so that you can easily select the stream attributes you want on the grid. These can then be dragged either onto the Stream View Attribute list on the editor or directly onto the header of the stream view on the dashboard.

If you tick the box on a Stream View to show superseded records these are now shown with a light greyish purple background.

Stream View Attributes can now be marked as 'Frozen', meaning that they remain on the screen when the grid view is scrolled to the right.

Grid Formatting Rules can now refer to other views and context params.

Added the 'Group Spacing' field to the Stream View Editor, which allows to specify the spacing between bars groups on the bar-chart

Fixed inconsistency in bar charts colours

Attributes grids are now displayed on the Sort Order editor even if the Sort Order is not saved.

Grid buttons now support states (Mouse Over, Mouse Down)

Pivot View

Various Improvements.

Chart View

Multiple charting improvements, i.e. Allow the axis to be hidden, padding changes.

The user can now see the context menu on standalone charts.

Updated chart configuration design.

Charts can now use Aliases on all attributes, not only as values.

Styles

Style names are now checked for uniqueness.

Performance Improvements

The basic set of expression functions can now be calculated on the client-side. This causes actions which require expression processing (i.e. running a Stream Action, refreshing a dashboard etc.) to be much faster and more responsive.

Dashboard layouts are now cached after first load - this makes subsequent dashboard displays faster.

Layout caching is switchable: dropdown Fields have now the "Cache Data" checkbox, which allows the system to cache the dropdown's data after the first load. The cache is cleared if you switch apps or log out. Layout data is not cached if you are in Design Mode.

Localisation

Date/Time and Number Formats

Added Date and Number Format objects. Formats can be used to change how numbers and dates are displayed in grids and forms.

Formats can be applied on Components (Stream View Attribute for Grids, Layout Component for form elements), Applications and the System (in System Configuration). Formats on Components override those on Applications, which in turn override System formats.

Date and DateTime formats may be linked via Date Styles to the current Locale, allowing the effective date format to be determined by the Locale.

Connectivity

Database Collector

Field "Query String" on Database Collector has been renamed to "Statement Expression"

Database Exporter

Errors from database exporters will appear in the logs as warnings up to the number of errors set by "Errors Before Rollback" on the exporter. Errors below this limit will not cause the suspension of task plans with "Suspend on failure" ticked.

Datasource

Improved error messaging for problems connecting to Datasources.

Improved clarity on error messages caused by invalid check queries for datasource instances

Stack trace will not go into the log file for errors connecting to a datasource where the cause is known to PhixFlow (e.g. incorrect password)

File Collector Error Handling

Improved error message when a file collector cannot move a file as access to the directory is denied.

Change how imported files are moved to the archive / error directory to reduce unnecessary copying and improve reporting in the event of failure.

File Exporter

You can now refer to _inputMultipler.<attributeName> in file exporter expressions. This means the input multiplier can simply be a pipe name and each _inputMultiplier value is then a record from that pipe.

Improved error messages when system is unable to open a file during export.

The Excel Template field is now an expression with access to the Input Multiplier value, which allows the exporter to use different templates for different exported files in a single exporter run.

The default Excel export file format (for new installations) is .xlsx

For File Exporters the Line Separator Characters may include escaped Unicode characters.

Multi-input File Exporters

File Exporters now provide a new file type, 'Excel Spreadsheet (Multiple Inputs)', which support exporting data from multiple input pipes to a single Excel spreadsheet. The export must still be triggered by a Push pipe, but can now read data from other input pipes. The exporter must have an Excel template which is annotated to define which pipe data is written into which region of the template.

Email Configuration Improvements

Email Account configuration has a number of improvements:

• Select inbound protocol (IMAP / POP3)
• Select Encryption (None / STARTTLS / TLS/SSL)
• Select Authentication (the ability to send/receive emails with/without a username or password)
• Easier access to debugging
• Default port based on protocol / encryption

Configuration Export / Import / Copy

General

Improved Import/export reliability

Improved help for configuration export.

Password-protected Exports

Confidential fields (passwords, security questions etc.) are now either encrypted in or omitted from configuration export files.

When the user exports configuration data, he is prompted to supply a password; if he does so, all confidential fields associated with the exported objects are encrypted using the supplied password; if not, confidential fields are not written to the export file. On import, if the user provides the same password, the confidential fields are written to the database; if not, the confidential fields in the import file are ignored and the non-confidential configuration data is imported (the confidential fields will be unchanged or blank). Providing an incorrect password is an error.

Compressed Exports

Configuration exports can now be automatically compressed. This can give a significant improvement in download/upload times.

Duplicate renaming

Duplicate objects are renamed on copy / import. All name changes are reported to the console.

Copy-on-Import

It is possible to flag an export as copy-on-import, in which case when imported the contained objects are created as copies rather than replacing the existing versions of the objects.

Cancellation

It is now possible to cancel a long-running configuration export or import from the console.

Reporting

Import now reports all major objects imported.

Operational

Console

The system console now opens maximised by default.

Email Queue

Emails sent by Task Plans or for Password Reset are now queued and sent later if they cannot be sent straight away. This means

1. that emails are not lost if the email server is down at the time when the emails are generated, and
2. the emails can include attachments generated by processes outside PhixFlow.

The list of emails is visible in the Console and serves as both an operational interface and as an audit log of emails sent.

The System Task will delete old emails according to the retention period set in System Configuration.

Logging

Various minor changes to improve server-side logging.

Bug Fixes

Installation

General

Updated recommended settings for pound reverse proxy.

Security

Permissions

Fixed problems with default double-click actions permissions

Audit

Added translations to make audit logs more readable.

Users / Login

Users

Fixed problem with saving user without changing the password

A user can no longer be created without setting a password policy.

Login

Logging in as an external (active directory) user now correctly ignores existing user records if they are marked as deleted.

Removed spurious local user login failed messages when an active directory user logs in.

Improved the way the login screen handles a blank username

Fixed problem with requests being sent twice if the user was logged out

Now when setting a first name, last name or username for a local user, leading or trailing white space will automatically be removed.

Password reset requests are now always seen as successful by the user while the admins can see the cause of any failure in the Tomcat log files.

Graphical User Interface

Tool tips

Fixed problem with tooltips not showing on areas containing other items.

Dropdown Menus / Lists

Fixed problem with dropdowns appearing in wrong place if the calling field has non standard height.

Read only dropdowns are now correctly translated

File Upload

Fixed problem with multiple files names overriding progress bar on file upload forms.

Browser-specific

Fixed various problems with the Firefox browser.

Fixed problem with wrapping log messages in IE

Other

Fixed problem with editor windows (i.e. Filter Editor) appearing under a dashboard if the dashboard is set to 'Display on top'.

Fixed problem with read only check boxes.

Fixed an occasional issue where trying to open an object editor would fail.

Configuration details for an object in design mode now open correctly even if there is no repository browser open when the object is selected.

Editor grids preserves the currently applied sort order while refreshing

Fixed bug with model toolbox staying visible when mouse moved out of the model too quickly

Fixed bug with drag action being sent to all open filter editors

'Find in explorer' button now works correctly for things in packages

Added a clear error message when gui versions mismatch

Date and DateTime fields on design mode property editors now stretch to the full width of the property editor to allow for much longer date formats.

Fixed position of long messages on error windows

Fixed problems with resizing various columns in configuration panels

A currently open expression editor will be reused if the user wants to open a new editor for the same field

Fixed bug with an expression editor window saving result to wrong object editor

Dragging multiple objects now correctly preserve their sequence.

Modelling

Analysis Models

Fixed problem with confirmation windows while dragging Analysis Models onto new containers

Fixed problem with incorrect messages while deleting multiple objects

Fixed problem with 'Show Inputs' and 'Show outputs' buttons while clicking on self-referencing Stream.

Fixed problem with showing predecessors/successors on a modelling pane

Analysis Models don't allow static flag if pipe is from a collector/exporter.

Fixed bug with incorrect tooltip position while analysis model zoomed out

Analysis Model nodes are now being selected using double-click

Fixed bug with moving analysis model's children to new container

After dragging an analysis model onto a models list, the user will be asked if he wants to move related object to the parent container

Fixed problems with deleting pipes on a model

Datasource

Fixed problem with deleting pipes from Datasources while other pipes are selected

Database Collectors

Fixed problems with displaying data directly from database collectors.

Database Exporters

Improved help for Database Exporter Commit Size.

File Collectors

Improved Excel File Collector error reporting

Fixed bug with the 'Input Directory Expression' error for managed file collectors

Improved error message when a File Collector fails to move a processed file to an archive or error directory.

File Exporters

Fixed issue where Excel Template could not be configured for HTML File Exporter.

Fixed reporting when excel template expected but no template found

File Exporter input multiplier expression can now reference the _context variable.

Temporary variables are reset on file exporters each time a new input multiplier value is processed.

Exporting a Stream Set of size 0 to Excel will no longer cause an error. The exported Excel sheet will be blank.

HTTP Datasource

Fixed problem causing null error messages when failing to connect to Http Datasource

Improved help for datasources.

Pipes

Fixed problem deleting Pipe that has no input stream / collector.

The user can now delete input pipes from Database, File and Http Exporter editors

Fixed problem with opening pipes context menu

Aggregate pipe function names are now translated

Fixed problem with pipe editors when input data handler is null

Fixed problem with a possibility of having two active pipes with the same name

The Pipe fields To Date Offset and From Date Offset have been renamed To Offset and From Offset

Streams

Changing a stream attribute's name failed if one of the stream's out pipes had an output stream that in turn had an input pipe which had no input stream

Improved error message for when an archive filter fails

Fix bug which stopped you dropping order by fields from pipes onto the stream attribute list

Improved performance when making changes to Streams with a large number of Stream Sets

Fixed error message when Stream Attribute integer value is too long for the field size

Improved error reporting for failed type conversion

Fixed issue where stream attribute expression allowed _out. references to later attributes

Deleting a stream could fail in such a way that the stream configuration was left unchanged but the backing data table was deleted. This state was unrecoverable - it was not possible to retry the deletion, nor could the backing data table be re-created. Now, the system will automatically create a new, empty backing data table, allowing the user to re-try the delete or to continue modelling.

Fixed problem with rolling back superseding records

Fixed problem with deleting a Stream while assigned to an Area.

Fixed issue where adding a self referencing stream attribute caused an error unless the attribute had already been saved with a different attribute expression.

Application Building

Actions

Action buttons that do not have any records on a backing form but have also ticked the 'Copy By Values' option on the action no longer fail.

Fixed problem with buttons with General Action Type and nothing else configured causing errors

Failed stream actions now leave any created streamset in the 'Incomplete' state rather than the 'OK' state.

Streamsets created by stream actions now show up in the log file.

Card Component will now refresh after a Stream Action completion even if the Action is run from an other window.

Fixed bug with empty Action error messages.

Stream actions now ignore superseded records.

Allow any action to be added to the refresh actions list on a stream view - not just actions from the same stream.

Fixed bug with incorrect name of model displayed in the "Running Actions" bar

If you right click on a form button that has an action associated with it there is now a context menu option to show the stream action configuration.

Task Plans used in Task Plan Rules on Stream Actions can now be deleted.

Stream Actions can now insert records into empty partitioned streams. This previously caused an error.

Duplicate attributes now cannot be added to the Stream Action Attributes list.

Dashboards

Fixed problems with dashboard dropdown buttons

Fixed problem with dashboard driving filter referring to the parent view, which was causing refreshing to loop.

Fixed problem with creating a dashboard view while a view already exists on the chosen area

Fixed problems with using column context menu on dropdown views

Dropdown selection lists no longer remain open if you start scrolling outside the list.

Right aligned dropdown fields no longer overwrite the dropdown icon.

Reopening a dashboard is no longer needed while editing dashboard elements

Components

Fixed problem with the 'Use default filter' flag on StreamView Layout components.

Fixed problem with rendering a form after adding a shared style to the form's element

Dynamic text components now display new lines

The design mode context menus (to show the view and stream configuration) are no longer shown when you right click on the drop down lists which are shown when you click on a drop down field on a form.

The hyperlink to the parent of an area layout component now works.

Context Parameters

Fixed problem with using dates as context parameters

Context parameters name doesn't allow white spaces anymore.

Filters

Improved error message if stream attribute type is changed after a pipe filter is applied and the filter cannot be applied to the new type. Filter will continue to work if types are compatible.

Fixed problem with filters conditions not updating

Fixed problem with filter conditions changing their sequence while editing

If the user applies a filter on a editor grid and then edit some other object in the same editor, the filter will be applied and filter icons will be correctly displayed on the filtered column

Filter drop down list doesn't contain local fields anymore

Fixed bug with filter field large expression window

Filter condition field is now larger

The user can now set a driving filter on different streams using column header dragging

Added a friendly error message when trying to set up a background filter on a view without the attribute with matching name

Fixed problem with deleting / adding filter entries

Fixed bug with the size of an input field on quick filter editor

Fixed problem with calling _prompt filter from a standalone Stream View

Fixed problem with displaying filter editors after deleting Stream Attribute used in the filter

Improved filter condition error messages

_prompt filters now allow field names to have an _ characters

Views

Fixed problem with order of bars on bar-charts

Fixed issue where empty formatting rule expression caused an error when loading grid views.

Form view fields now show new lines if necessary.

Fixed problem with sorting strings starting from numbers

Fixed grid scrolling issues.

When viewing a view outside of a dashboard the view now shows the number of streamsets defined in the advanced tab.

Fixed problem with chart panes (i.e. card views) being reused on multiple dashboards

Fixed problem with 'filtered' icon indicator on Stream Data grids

Fixed problem with quick filter 'clear all' button

Fixed problem with creating Stream View Attributes while the Stream or the Stream View name contains '&'

Fixed problem with standalone views ignoring periods

Non numeric view attributes can no longer be given numeric aggregate functions.

Checkbox when hover over a selection column is now correctly displayed

Fixed bug with refreshing a grid driven by a form with no stream

Fixed problem with incorrect grid column sequence after dragging onto a grid

Fixed bug with float form fields not backed by a Stream Attribute

Chart Views

Fixed problem with displaying decimal values on charts

Fixed problem with distribution of values on the Line Chart (time)

'Label angle' option now works on charts

Formatting Rules

Fixed problem with formatting rule using background or background-color css properties.

In formatting rules on grids you can access _field.Labels to see the list of labels assigned to the current column in the grid. This attribute was not being set properly and so couldn't be used. This has now been fixed.

Style

Quick style setting are now being applied in conjunction with other css properties

Fixed problem with style elements not having a sequence number

Removed incorrect style reference from the tab header label template

Configuration Export / Import / Copy

Import / Copy

Fixed issue causing blank Task Plan Rules when copying them across Stream Actions

Scheduled task plans are now started automatically after configuration export/import.

Export/Import and Copy now correctly maintain used-on links from Streams, etc. to Analysis Models.

Copying/exporting internal non- top-level objects (e.g. Stream View, Stream Action) is no longer allowed as it gave unpredictable results.

Configuration import failed sometimes for Dashboards with user-specific properties (e.g. width / height)

Fixed problem with broken dashboard links after incomplete import.

Import/export now correctly exports/imports individual users

Operational

Console

Fixed problem with current task plans not visible in the System Console

Fixed a bug in filtering log files on the console by user name.

Task Plans

Fixed bug with refreshing a Task Plan Streams list after dragging a Stream on it

Performance Statistics

Fixed issue which prevented statistics saving to the database because the data value was infinity

Fixed problem whereby data errors caused unrelated performance statistics to be lost.