Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

Insert excerpt
_Banners
_Banners
nametaskPlan
nopaneltrue

This topic is for data modellers, application designers and system administrators. The pages in this topic explain how to create and run automated tasks in PhixFlow. It also explains strategies for tasks that affect

streams

tables

The pages in this topic are:

Child pages (Children Display)
depth4

Panel
borderColor#7da054
titleColorwhite
titleBGColor#7da054
borderStylesolid
titleSections on this page
Table of Contentsindent12pxstylenone

Overview

When working with data, applications and IT systems, there are routine processes that you need to run. PhixFlow makes it easy for you to set up and manage these processes using tasks and task plans. For example:

  • application designers or system administrators can automatically update data in applications
  • data
modelers
  • modellers can run analysis on models or specific
streams 
  • tables
  • administrators can clear old information from the PhixFlow system, by archiving data from:
    stream
      • record sets
      • processing logs
      • performance statistics.

    Use the task plan properties tab to configure Task Plans. Set the options and add individual Tasks.

    If you need to temporarily prevent scheduled task plans from running during system maintenance, see Preventing Task Plans From Running.

    Insert excerpt
    Task Plans
    Task Plans
    nopaneltrue

    Structuring Tasks

    When setting up task plans with tasks that affect

    streams

    tables, you must consider whether the

    streams

    tables have any dependencies. This affects your options when structuring tasks in task plans.

    Either

    Multiple tasks each

    task affects

    affecting one

    stream

    table

    If there are dependencies between

    streams

    tables, configure PhixFlow to have one task per

    stream

    table and configure each task to be mandatory. 

    1. Create a set of tasks.
      • Each task works on a single
    stream
      • table.
      • Set the tasks to be mandatory.
    1. Add the set of tasks to the task plan.

    PhixFlow processes the tasks, and therefore the

    streams

    tables, in order. PhixFlow must finish a task successfully before it starts to update the next

    stream

    table. This is the safest way to set up tasks that affect

    streams

    tables. Always use this method when there are dependencies between

    streams

    tables.

    Panel
    borderColor#01cff1
    borderWidth4
    titleBGColor#01cff1
    titleExample

    A model contains connected

    streams

    tables that should be run twice daily to update a model with the latest data from an external database. A scheduled task plan contains:

    • TaskA to run
    stream
    • table-a
    • TaskB to run
    stream
    • table-b
    • TaskC to run
    stream
    • table-c.

    PhixFlow will process TaskA until changes to

    stream

    table-a are finished. As this now has the latest data, when TaskB runs, it pulls the new data into

    stream

    table-b. Finally TaskC pulls new data from

    stream

    table-b into

    stream

    table-c.


    Or one task

    affects

    affecting multiple

    streams

    tables

    If there are no dependencies between

    streams

    tables, you can add multiple

    streams

    tables to the task. In this case you have no control over the order in which

    streams

    tables will be processed. Only use this method if there is you are sure there are no dependencies between the

    streams

    tables.

    1. Create a task
    2. Add multiple
    streams
    1. tables to a task
    2. Add the task to the task plan. 

    This has the advantage that you only need to configure one task to change multiple

    streams

    tables.

    Panel
    borderColor#01cff1
    borderWidth4
    titleBGColor#01cff1
    titleExample

    Three

    streams

    tables, a, b and c are in different models and have not data dependencies. A task plan to update

    the stream

    the table data contains a single task which runs analysis on

    stream

    table-a,

    stream

    table-b and

    stream

    table-c. PhixFlow processes all the

    streams

    tables concurrently.

    Mandatory Tasks

    The tasks are run in the order they are listed in the the task plan.

    You can specify that a task in a task plan is mandatory. This means the task must complete before the next task in the sequence can run. If a mandatory task fails, PhixFlow will not run the following tasks. You must resolve the issue preventing the task from running and then restart the task.

    Note

    Rerunning a task plan that previously failed means it continues that run from where it failed.

    If data related to an previous task has changed this is not reflected in the results of the rerun. 

    This means that after you have rerun a previously failed task, consider then running the task again, to ensure all data or processes are up-to-date.

    Anchor
    task-type
    task-type
    Types of Task

    Anchor
    analysisTask
    analysisTask
    Analysis Tasks

    Use an analysis task to run analysis on the

    stream

    table(s) in the task. 

    Anchor
    rollbackTask
    rollbackTask
    Rollback Tasks

    Use a rollback task to effectively undo run analysis

    on

    on a

    stream

    table

    When

    When you run a rollback task it rolls back all data in each of the listed

    streams

    tables, deleting the

    stream

    record-sets. The list of

    stream

    record-sets is empty and there are no data records in the

    stream

    tableFor information about how to rollback

    streams

    tables manually, see Rollback Recordsets.

    Anchor
    configExportTask
    configExportTask
    Configuration Export Tasks

    Use a configuration export task to export selected applications and packages or a full configuration. The exported file is saved to the download area;

    see 

    see Using the Download Area

    Anchor
    systemTask
    systemTask
    System Tasks

    Use a system task to perform system-wide housekeeping activities. These include deleting:

    • managed file entries
    • email entries
    • temporary files created by file exporters to send by email
    • data from incomplete
    stream setsorphaned stream views from the PhixFlow database
    • record-sets
    • log messages and optionally archiving them
    • from the PhixFlow database:
      • table views for which there is no table
      • tables for which there is no table object, when they are older than the period set in System Configuration → Delete Orphaned Table after Days. 
        Tables become orphaned when their table object is deleted or removed by an import process.
      • temporary tables created during publishing, when they are older than the period set in System Configuration → Delete Temp Tables after Days.
        Temporary tables are created by PhixFlow during the publishing process.
    Note

    The system task should be run regularly, for example daily

    ,

    or

    at a minimum every week. If incomplete stream sets are not deleted, they

    weekly. When the system task is not run regularly, incomplete record-sets can accumulate. If these are large, or many have accumulated, this can slow down PhixFlow's performance. Depending on your PhixFlow database, queries that have to exempt many incomplete

    stream

    record-sets can reach system limits. This can prevent PhixFlow and its applications from running.

    The archiving periods

    How long PhixFlow keeps system information, and whether to archive log messages before they are deleted, are controlled by the following parameters in System Configuration:

    Note

    Ensure that only one system task is running at any one time to avoid system errors.

    Anchor
    archiveTask
    archiveTask

    Archive

    Table-Data-Delete Tasks

    Use

    an archive

    a table-data-delete task to delete

    or archive stream data or stream

    data records and record-sets that you no longer need in PhixFlow.

    Archive data is saved into a zip file - to get at this? You can (in theory) reinstate archive data. Whether or not PhixFlow deletes or archives stream sets depends on how the archive settings in the stream are configured. 
    Running an archive

    How long PhixFlow keeps table data is configured in table properties → Data Retention Settings.
    Running a table-data-delete task on a large number of

    streams

    tables, or on

    a stream

    a table with a large data set, can take some time. We recommend scheduling

    archiving

    these tasks for times when the system is quiet, for example overnight

    .
    Note

    Phixflow can only run an archive task on streams that have Archive Settings specified. If a stream does not have any archive settings, PhixFlow never deletes or archives the data. Stream sets will accumulate, leading to space and performance issues.

    Archiving Data

    Archiving Specific Streams

    You can create an archiving task for a specific stream or streams. PhixFlow creates a single zip file containing the specified streams

    .

      For example, you might want to create a single archive for:
  • data from several related streams
  • a stream that contains a large amount of data  


  • a stream that you want to be able to restore from the backup. This is easier when the data is in its own zip.
  • Archiving Any Stream

    You can create a single archive task to delete or archive the data in streams:

    • that are not in another archive task
    • and have archiving settings configured.

    To do this, select the Archive All option

    When you run the task, PhixFlow:

    • deletes stream sets that are older than the specified archive settings in the stream
    • creates a zip archive of all the stream sets within the specified archive settings in the stream.
      This is a single archive containing all the data from all the streams processed.

    Archive tasks must have:

    • either at least one stream added
    • or have the Archive All option ticked.
    If streams

    When tables are deleted from the repository, their names

    will

    are automatically be removed from tasks that refer to them. This means a saved task can become empty

    /redundant

    Finding the Archive file

    When a stream is archived, PhixFlow logs information in the system console →  Archive Log. This includes information about the data archived and any archive file created. This is a single archive containing all the data from the streams processed.. Optionally, the archived data can be saved to a file. If this is done, the data can reloaded into PhixFlow using Restore Archive.

    Warning

    This should be a warning in the Streams page.

    Streams that do not have any Archive Settings will never have their data deleted or archived. This can cause large amounts of data to build up in PhixFlow, affecting performance. Remember to set your archive settings!!!

    For data in a stream to be deleted or archived, the stream must have Archive Settings applied. If a stream does not have any archive settings, the

    .

    PhixFlow will no longer run the task - log mesages??


    Note

    PhixFlow only runs a table-data-delete task on tables that have Data Retention Settings specified. If no data retention values are set, PhixFlow never deletes the data. table data will accumulate, leading to space and performance issues.

    See also Deleting Old Data

    Anchor
    disable
    disable