Skip to end of metadata
Go to start of metadata

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 77 Next »


This page is for a data modeller who needs to load data from an external source via HTTP.

Overview

An HTTP collector reads data from a HTTP Datasource. The collector defines how the data needed from the datasource is extracted to be used in PhixFlow.

To add a new HTTP collector to an analysis model:

  1. Go to the model's toolbar → Create group.
  2. Click  HTTP to expand the menu.
  3. Drag a  HTTP Collector onto the analysis model.

To add an existing HTTP collector to an analysis model, in the model diagram toolbar:

  1. Go to the model toolbar → List group.
  2. Click  HTTP to expand the menu.
  3. Click  HTTP Collector to open the list of available collectors.
  4. Drag an HTTP collector into the analysis model.


 Handling Newlines

The way that HTTP collector, exporter and header objects handle newlines in data depends on the data type.

String, XML and HTML

HTTP objects retain all newline characters in HTTP and XML data. To strip the newlines, add the following expression to the attribute expressions that use the data from the HTTP object

.replaceAll(in.value, "[\r\n]", ""

XML

As PhixFlow cannot include newlines in an attribute value, it automatically replaces newlines in JSON with a space. If you want the data to indicate where the newlines were, in the Statement Expression, specify the escape character \n.

In the following sample of JSON, the paragraph symbol  indicates a newline character.

{
  “line1” : “Single line of text" 
  “line2” : “Also a single 
line of text” 
 “line3” : “Multiple lines of↵
 text split over↵several lines” 
}

By default a HTTP collector retrieves...

Not sure how to deal with this for Headers and Exporters....

{“line1” : “Single line of text" “line2” : “Also a single line of text”, “line3” : “Multiple lines of text split over several lines” }

in the Statement Expression, specify ??? to return a string with the newline indicated by /n.

{“line1” : “Single line of text" “line2” : “Also a single line of text”, “line3” : “Multiple lines of/ntext split over/nseveral lines” }

HTTP collectors and exporters:

  • replace newlines with spaces in JSON data. To retain newlines, see Statement Expression.
  • retain all newline characters in HTTP and XML data. To strip the newlines, add the following expression to the attribute expressions that use the data from the HTTP collector or exporter.
.replaceAll(in.value, "[\r\n]", ""

JSON data automatically replaces newlines with spaces.

Table Values in a HTTP Collector

To drive the lookups made by a HTTP collector from a table, the two must be connected using a lookup pipe. For example, a URL for a server can either be captured or calculated in an attribute called "ServerURL". The URL is then passed via a lookup pike to the HTTP collector to be used in its URL Expression

If the pipe is called in, here is how the URL Expression would be written on the HTTP collector: 
  {substring(in.ServerUrl,9)}


Property Pane Toolbar

For information about the toolbar options, see the Common Properties page, Toolbars and Controls section.

 Properties Tab

Parent Details

If this item is within or belongs to another, its parent name is shown here. See the Parent Details section on the Common Properties page for more details.

Basic Settings

FieldDescription
NameEnter the name of the HTTP collector.
Enabled Tick when the configuration is complete and the collector is ready to be used.

Send Message

Define details of the HTTP request sent to the HTTP Datasource to get the data required.

FieldDescription
HTTP Request Method

Select one of the following HTTP methods to use for the request:

  • GET or POST
  • GET
  • POST
  • DELETE
  • OPTIONS
  • PUT
  • PATCH

We recommend that you select a method but if you do not, PhixFlow uses GET or POST by default. If the Send Message → Statement Expression:

  • evaluates to null or empty string, PhixFlow uses GET
  • is not empty, PhixFlow uses POST.

For information, see the w3schools page about HTTP methods.

URL Expression

The URL to be used, without the leading http:// prefix.  If this field is blank, the URL field on the HTTP Datasource Instance is used directly.

You can embed expressions in the URL by enclosing the expression in ${…}. Syntax rules depend on the response language; see Embedded Expressions.

Example

This expression adds to the base URL provided by the HTTP datasource instance:
    {_url}/sub1/sub2?param1=3

The HTTP collector follows any HTTP redirections and returns the final response.

Statement Expression

An expression to generate the data that will be sent by the collector to the datasource.


Example

XML Statement
<?xml version ="1.0"?>
<!DOCTYPE CORPORATE DASHBOARD "corpDash.dtd">
<results user="%USERNAME%" password="%PASSWORD%">
	<monthlyTotals region=${'"' + Region + '"'} division=${'"' + Division + '"'}>
		<totalBilled>${'"' + TotalBilled + '"'}</totalBilled>
		<totalCollected>${'"' + TotalCollected + '"'}</totalCollected>
	</monthlyTotals>
</results>
Example JSON Statement
{
	user: '${user}',
	code:  ${'{size: "big"}'}
	price: ${price},
	currency: '${currency}'
}

Embed expressions within a statement by enclosing the expression in ${…}. Syntax rules. see Embedded Expressions.

The statement is encoded based on the Content-Type Header settings.

  • If the Content-Type Header charset parameter is set, this is used.
  • If the Content-Type header is set, but no charset is specified, then PhixFlow uses a default character set dependent on the content type.
  • If no Content-Type Header is set, PhixFlow uses the default ISO-8859-1

For how to configure and use usernames and passwords for a site, see Authenticating for an API and  Secret Key and Local Secret
To reference:

  • HTTP datasource instance Username and Password properties, use the variables:  %USERNAME% and %PASSWORD%.
  • encrypted values, use  ${_datasource.key}  where key is the Name of the secret key.

See also HTTP datasource instance and Expressions and PhixScripts.

HTTP Headers

This section has a toolbar with standard buttonsThe grid contains a list of the HTTP headers defined for this collector. To add a HTTP header to the list, click  Create New. PhixFlow opens a new HTTP Header properties tab. To remove a HTTP header, use the  Delete in the toolbar.

Some headers will be set to default values if not provided. Automatically added headers may not appear in the debug log. The Content-Length header will be added to all requests and cannot be overridden by providing a value.

Response

Define the data response type/format that will be returned:

  • HTML: response type allows an XPath Expression to be specified in order to retrieve just specified sections of the data into XML structures.
  • XML: response type allows an XPath Expression to be specified in order to retrieve just specified sections of the data into XML structures. XML response types also support XML namespaces. The Xml Namspaces tab will be available when this response type is chosen.
  • String: response type will return the full data as a string value.

Please see Response Examples for how the returned data can be used and evaluated in the corresponding attribute expressions.

FieldDescription
Return  Type

Select the type of the expected response:

  • String
  • XML
  • HTTP
  • JSON

XPath or JsonPath

Available when Return Type is XML, HTTP or JSON.

The expression used to resolve or filter the data that comes back in the selected format.

Use XPath namespaces syntax for XML response types only.

XML Namespaces

Available when Return Type is XML or HTTP. This section has a toolbar with standard buttonsThe grid contains a list of the namespaces defined in an XML response.

To add a namespace to the list, click  Create New. PhixFlow opens a new XML Namespace panel. To remove a namespace, use the  Delete in the toolbar.

Analysis Models

If this item is used by an analysis model, its name is listed here. See the Common Properties page, Analysis Model section for more details.

Advanced

FieldDescription
HTTP DatasourceSelect the HTTP datasource that this collector will collect from. For how to add a new one, see HTTP DatasourceTo select from a list, click  Show the List of HTTP Datasources.
Datasource Instance Expression

The datasource to which this collector is connected may list multiple instances from which the data may be accessed. Each HTTP datasource instance is identified by a unique string. This expression should evaluate to a string which allows the collector to determine the specific instance to use. If the expression is blank then the collector will assume that there is only one instance and will use that one by default. If there is more than one instance and no expression is provided here then an error will be thrown during analysis since the collector will be unable to determine which source to use.

See also  HTTP datasource instance and Expressions and PhixScripts.

Timeout (secs)The number of seconds to wait for a response from the corresponding HTTP datasource before a timeout is recorded.
IconEnter the path for an image file that has been uploaded to the PhixFlow database. PhixFlow displays this icon in controls when the HTTP collector is used.
Allow Non-Scheduled Collection

 Tick to run the HTTP collector as part of any ad-hoc analysis run that requires this data. If not, it will only run as part of a scheduled Task Plan.

Log Traffic

You can set this option when system logging → Allow Logging is ticked; see System Logging Configuration.

PhixFlow always logs HTTP responses and requests for HTTP collectors, whatever is set here.

Analysis Models

If this item is used by an analysis model, its name is listed here. See the Common Properties page, Analysis Model section for more details.

Description

We recommend that you always enter a description to explain the purpose of this item.

 Audit Tab

Audit Summary

See the Common Properties page, Audit Summary section.

Response Examples


This example uses the following XML and HTML data.

XML Data

<?xml version ="1.0"?> <root xmlns:h="http://www.w3.org/TR/html4/"> <main page="PF Main Page" > <h:title h:name="PF Title">PF Title Text <h:datarow> <h:data h:initials="AA">Alistair Andrews</h:data> <h:data h:initials="BB">Bert Brown</h:data> </h:datarow> </h:title> <title name="Non namespace Title">Non namespace Title Text</title> </main> </root>

HTML Data

<html> <body nodename="Html Body"> <table> <tbody> <tr> <td initials="AA">Alistair Andrews</td> <td initials="BB">Bert Brown</td> </tr> </tbody> </table> </body> </html>

The data is being pointed to by either HTTP datasources or XML/HTML File collectors respectively.

The following table shows the different types of responses that can be returned from an HTTP Collector and how these can be used in the corresponding attribute expressions. A HTTP Collector response type of XML/HTML will mimic the responses from XML/HTML Collectors respectively.

Response TypeXPath ExpressionExplanation
Stringn/aA String response should be referenced in the attribute expressions as in.value Note that in.value will contain the complete string data referenced above.
XML
/root/main/h:title


The namspace prefix used here 'h' must be configured in the XML Namespace.

This XPath expression will bring back all elements matching the xpath expression including the parent/grandparents and all child elements/subelements. i.e

<root xmlns:h="http://www.w3.org/TR/html4/"> <main page="PF Main Page" > <h:title h:name="PF Title">PF Title Text <h:datarow> <h:data h:initials="AA">Alistair Andrews</h:data> <h:data h:initials="BB">Bert Brown</h:data> </h:datarow> </h:title> </main> </root>

The following examples show how to reference the returned xpaths html/xml data structure in attribute expressions:-
  • Xpath element text value: in.value -> returns 'PF Title Text'
  • Xpath element attibutes: in.h$name -> returns 'PF Title'
  • Xpath parent attributes: in.^.page -> returns 'PF Main Page'
  • Xpath child attributes: listToString(in.h$datarow.h$data.h$initials) -> returns 'AA,BB'
  • Xpath child attribute text values: listToString(in.h$datarow.h$data.value) -> returns 'Alistair Andrews,Bert Brown'

Note the use of

  • $ instead of our usual : namespace notation.
  • ^ to traverse to the immediate parent element.
  • the listToString function to handle multiple matching child elements/attributes.
HTML

/html/body/table


Namspaces are not supported in the Xpath expression for HTML response types.

This XPath expression will bring back all elements matching the xpath expression including the parent/grandparents and all child elements/subelements. i.e

<html> <body nodename="Html Body"> <table> <tbody> <tr> <td initials="AA">Alistair Andrews</td> <td initials="BB">Bert Brown</td> </tr> </tbody> </table> </body> </html>

The following examples show how to reference the returned xpaths html/xml data structure in attribute expressions:-
  • Xpath parent attributes: in.^.nodename -> returns 'Html Body'
  • Xpath child attributes: listToString(in.tbody.tr.td.initials) -> returns 'AA,BB'
  • Xpath child attribute text values: listToString(in.tbody.tr.td.value) -> returns 'Alistair Andrews,Bert Brown'

Note the use of:

  • ^ to traverse to the immediate parent element.
  • the listToString function to handle multiple matching child elements/attributes.
  • the optional html <tbody> tags. If these are not in your html data, then PhixFlow will insert them to conform with the HTML standards. Therefore when using absolute XPath expressions, the tbody tags need to be included even if they are not present in the incoming HTML data. That means /html/title/table/tr should be replaced with /html/title/table/tbody/tr or use //tr
    The same applies when referencing parent/child nodes within the attribute expressions.
JSON

todo-Fiona: example requested from ThomasS

Sections on this page

Learn More

For links to all pages in this topic, see Analysis Models for Batch Processing Data.


Terminology changes in progress

As part of the redesign of PhixFlow, we are changing the following terms:

dashboard → screen   
stream → table
stream attributes → attributes
stream item → record
stream set → recordset
stream view → view
stream item action → record-action 
stream action → table-action
driver class → database driver

  • No labels