Versions Compared

Old Version 31

changes.mady.by.user Fiona Sargeant

Saved on

compared with

New Version 32

changes.mady.by.user Fiona Sargeant

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

An HTTP Collector reads data from a HTTP Datasource. The collector defines how the data needed from the datasource will be extracted to be used in PhixFlow.

Stream Values in a HTTP Collector

To drive the lookups made by a HTTP Collector from a stream, the two must be connected using a lookup pipe. For example - If a URL for a server is captured, or calculated, on a stream in an attribute called "ServerURL" and passed to the HTTP Collector to be used in its URL Expression, the pipe connecting the two must be a lookup pipe. If the pipe is called in, here is how the URL Expression would be written on the HTTP Collector:  {substring(in.ServerUrl,9)}



Panel
borderColor#7da054
titleColorwhite
titleBGColor#7da054
borderStylesolid
titleSections on this page

Table of Contents
indent12px
stylenone


HTTP Collector Properties

Insert excerpt
_standard_settings
_standard_settings
nopaneltrue

Basic Settings

FieldDescription
NameName of the HTTP Collector.
EnabledTick when the configuration is complete and the collector is ready to be used.
HTTP Data SourceThe HTTP datasource that this collector will collect from.
HTTP Request Method


Excerpt

Select one of the following HTTP methods to use for the request:

  • GET or Post
  • GET
  • POST
  • PUT
  • DELETE
  • OPTIONS

We recommend that you select a method but if you do not, PhixFlow uses GET or POST by default. If the Send Message → Statement Expression:

  • evaluates to null or empty string, PhixFlow uses GET
  • is not empty, PhixFlow uses POST.

For information, see the w3schools page about HTTP methods.


IconThe Icon to display in controls when this collector is used.
Timeout (secs)The number of seconds to wait for a response from the corresponding HTTP datasource before a timeout is recorded.
Allow Non-Scheduled CollectionIf this is turned on, then the collector will run as part of any ad-hoc Analysis Engine run which requires this data. If not, it will only run as part of a scheduled Task Plan under the Analysis Engine.
Datasource Instance ExpressionThe datasource to which this collector is connected may list multiple instances from which the data may be accessed. Each HTTP Datasource Instance is identified by a unique string. This expression should evaluate to a string which allows the collector to determine the specific instance to use. If the expression is blank then the collector will assume that there is only one instance and will use that one by default. If there is more than one instance and no expression is provided here then an error will be thrown during analysis since the collector will be unable to determine which source to use.

Send Message

Define details of the HTTP request sent to the HTTP Datasource to get the data required.

FieldDescription
URL Expression

The URL to be used, without the leading http:// prefix. The URL may contain embedded expressions within { }. If this field is blank, the url field on the httpDatasourceInstance is used directly.

For Example, this expression adds to the base url provided by the HTTP Datasource Instance :

{_url}/sub1/sub2?param1=3

Statement ExpressionAn expression to generate the data that will be sent by the exporter to the datasource. For Example

<?xml version ="1.0"?> <!DOCTYPE CORPORATE DASHBOARD "corpDash.dtd"> <results user="%USERNAME%" password="%PASSWORD%"> <monthlyTotals region={'"' + Region + '"'} division={'"' + Division + '"'}> <totalBilled>{'"' + TotalBilled + '"'}<\totalBilled> <totalCollected>{'"' + TotalCollected + '"'}<\totalCollected> <monthlyTotals> <\results>

The username and password for the HTTP Datasource Instance are available as %USERNAME% and %PASSWORD%.

The data will be encoded using the charset parameter specified by the Content-Type Header if one is present. If no Content-Type Header is set then ISO-8859-1 will be used. If the Content-Type header is set, but does not specify a charset then PhixFlow will use a default character set dependant on the content type.

HTTP Headers

This section has a toolbar with standard buttons. The grid contains a list of the HTTP headers defined for this collector. To add a HTTP header to the list, click 

Insert excerpt
_add
_add
nopaneltrue
. PhixFlow opens a new HTTP Header properties tab. To remove a HTTP header, use the
Insert excerpt
_delete
_delete
nopaneltrue
 in the toolbar.

Response

Define the data response type/format that will be returned:

  • HTML: response type allows an XPath Expression to be specified in order to retrieve just specified sections of the data into XML structures.
  • XML: response type allows an XPath Expression to be specified in order to retrieve just specified sections of the data into XML structures. XML response types also support XML namespaces. The Xml Namspaces tab will be available when this response type is chosen.
  • String: response type will return the full data as a string value.

Please see Response Examples for how the returned data can be used and evaluated in the corresponding stream attribute expressions.

FieldDescription
Return  TypeThe type of the expected response : XML/HTTP/String
XPathThe XPath expression used to resolve or filter the data that comes back in XML or HTML format. Note that Xpath namespaces syntax can only be used for XML response types.

XML Namespaces

This section has a toolbar with standard buttons. The grid contains a list of the namespaces defined in an XML response.

To add a namespace to the list, click 

Insert excerpt
_add
_add
nopaneltrue
. PhixFlow opens a new XML Namespace properties tab. To remove a namespace, use the
Insert excerpt
_delete
_delete
nopaneltrue
 in the toolbar.

Anchor
responseExamples
responseExamples
Response Examples


This example uses the following XML and HTML data.

XML Data

<?xml version ="1.0"?> <root xmlns:h="http://www.w3.org/TR/html4/"> <main page="PF Main Page" > <h:title h:name="PF Title">PF Title Text <h:datarow> <h:data h:initials="AA">Alistair Andrews</h:data> <h:data h:initials="BB">Bert Brown</h:data> </h:datarow> </h:title> <title name="Non namespace Title">Non namespace Title Text</title> </main> </root>

HTML Data

<html> <body nodename="Html Body"> <table> <tbody> <tr> <td initials="AA">Alistair Andrews</td> <td initials="BB">Bert Brown</td> </tr> </tbody> </table> </body> </html>

The data is being pointed to by either HTTP datasources or XML/HTML File collectors respectively.

The following table shows the different types of responses that can be returned from an HTTP Collector and how these can be used in the corresponding stream attribute expressions. A HTTP Collector response type of XML/HTML will mimic the responses from XML/HTML Collectors respectively.

Response TypeXPath ExpressionExplanation
Stringn/aA String response should be referenced in the stream attribute expressions as in.value Note that in.value will contain the complete string data referenced above.
XML
/root/main/h:title


Note

The namspace prefix used here 'h' must be configured in the XML Namespace.


This XPath expression will bring back all elements matching the xpath expression including the parent/grandparents and all child elements/subelements. i.e

<root xmlns:h="http://www.w3.org/TR/html4/"> <main page="PF Main Page" > <h:title h:name="PF Title">PF Title Text <h:datarow> <h:data h:initials="AA">Alistair Andrews</h:data> <h:data h:initials="BB">Bert Brown</h:data> </h:datarow> </h:title> </main> </root>

The following examples show how to reference the returned xpaths html/xml data structure in stream attribute expressions:-
  • Xpath element text value: in.value -> returns 'PF Title Text'
  • Xpath element attibutes: in.h$name -> returns 'PF Title'
  • Xpath parent attributes: in.^.page -> returns 'PF Main Page'
  • Xpath child attributes: listToString(in.h$datarow.h$data.h$initials) -> returns 'AA,BB'
  • Xpath child attribute text values: listToString(in.h$datarow.h$data.value) -> returns 'Alistair Andrews,Bert Brown'

Note the use of

  • $ instead of our usual : namespace notation.
  • ^ to traverse to the immediate parent element.
  • the listToString function to handle multiple matching child elements/attributes.
HTML

/html/body/table


Note

Namspaces are not supported in the Xpath expression for HTML response types.


This XPath expression will bring back all elements matching the xpath expression including the parent/grandparents and all child elements/subelements. i.e

<html> <body nodename="Html Body"> <table> <tbody> <tr> <td initials="AA">Alistair Andrews</td> <td initials="BB">Bert Brown</td> </tr> </tbody> </table> </body> </html>

The following examples show how to reference the returned xpaths html/xml data structure in stream attribute expressions:-
  • Xpath parent attributes: in.^.nodename -> returns 'Html Body'
  • Xpath child attributes: listToString(in.tbody.tr.td.initials) -> returns 'AA,BB'
  • Xpath child attribute text values: listToString(in.tbody.tr.td.value) -> returns 'Alistair Andrews,Bert Brown'

Note the use of:

  • ^ to traverse to the immediate parent element.
  • the listToString function to handle multiple matching child elements/attributes.
  • the optional html <tbody> tags. If these are not in your html data, then PhixFlow will insert them to conform with the HTML standards. Therefore when using absolute XPath expressions, the tbody tags need to be included even if they are not present in the incoming HTML data. That means /html/title/table/tr should be replaced with /html/title/table/tbody/tr or use //tr
    The same applies when referencing parent/child nodes within the stream attribute expressions.

Advanced

FieldDescription
Log Traffic

This option is affected by system logging options; see System Logging Configuration.

  • Log HTTP Collector Connection Details : when ticked, PhixFlow always logs connections for HTTP collectors, whatever is set here.
  • Allow Logging when ticked, logging can be set here.

To change this property, you must have the Modify System Logging Configuration privilege.

Insert excerpt
_check_box_untick
_check_box_untick
nopaneltrue
 to prevent logging for this HTTP collector. Ticking this box has no effect when the system logging option Log HTTP Collector Connection Details is ticked.

Insert excerpt
_check_box_ticked
_check_box_ticked
nopaneltrue
 to log the details of the request and response for this HTTP collector. Ticking this box has no effect when the system logging option Allow Logging is not ticked.