This page is for a data modeller who needs to load data from an external source via HTTP.

Overview

An HTTP collector reads data from a HTTP Datasource. The collector defines how the data needed from the datasource is extracted to be used in PhixFlow.

.replaceAll(in.value, "[\r\n]", ""

{
  “line1” : “Single line of text" 
  “line2” : “Also a single 
line of text” 
 “line3” : “Multiple lines of↵
 text split over↵several lines” 
}

{“line1” : “Single line of text" “line2” : “Also a single line of text”, “line3” : “Multiple lines of text split over several lines” }

{“line1” : “Single line of text" “line2” : “Also a single line of text”, “line3” : “Multiple lines of/ntext split over/nseveral lines” }

.replaceAll(in.value, "[\r\n]", ""

To add a new HTTP collector to an analysis model:

1. Go to the model's toolbar → Create group.
2. Click  HTTP to expand the menu.
3. Drag a  HTTP Collector onto the analysis model.

To add an existing HTTP collector to an analysis model, in the model diagram toolbar:

1. Go to the model toolbar → List group.
2. Click  HTTP to expand the menu.
3. Click  HTTP Collector to open the list of available collectors.
4. Drag an HTTP collector into the analysis model.

Table Values in a HTTP Collector

To drive the lookups made by a HTTP collector from a table, the two must be connected using a lookup pipe. For example, a URL for a server can either be captured or calculated in an attribute called "ServerURL". The URL is then passed via a lookup pike to the HTTP collector to be used in its URL Expression. 

If the pipe is called in, here is how the URL Expression would be written on the HTTP collector: 
  {substring(in.ServerUrl,9)}

 Properties Tab

Parent Details

If this item is within or belongs to another, its parent name is shown here. See the Parent Details section on the Common Properties page for more details.

Basic Settings

Field	Description
Name	Enter the name of the HTTP collector.
Enabled	Tick when the configuration is complete and the collector is ready to be used.
HTTP Datasource	Select the HTTP datasource that this collector will collect from. For how to add a new one, see HTTP Datasource Properties.  To select from a list, click  Show the List of HTTP Datasources.
HTTP Request Method	Select one of the following HTTP methods to use for the request: GET or POST GET POST DELETE OPTIONS PUT PATCH We recommend that you select a method but if you do not, PhixFlow uses GET or POST by default. If the Send Message → Statement Expression: evaluates to null or empty string, PhixFlow uses GET is not empty, PhixFlow uses POST. For information, see the w3schools page about HTTP methods.
Icon	Enter the name of an icon to display in controls when this HTTP collector is used.
Timeout (secs)	The number of seconds to wait for a response from the corresponding HTTP datasource before a timeout is recorded.
Allow Non-Scheduled Collection	If this is turned on, then the collector will run as part of any ad-hoc Analysis Engine run which requires this data. If not, it will only run as part of a scheduled Task Plan under the Analysis Engine.
Datasource Instance Expression	The datasource to which this collector is connected may list multiple instances from which the data may be accessed. Each HTTP datasource instance is identified by a unique string. This expression should evaluate to a string which allows the collector to determine the specific instance to use. If the expression is blank then the collector will assume that there is only one instance and will use that one by default. If there is more than one instance and no expression is provided here then an error will be thrown during analysis since the collector will be unable to determine which source to use. See also  HTTP datasource instance and Expressions and PhixScripts.

Send Message

Define details of the HTTP request sent to the HTTP Datasource to get the data required.

HTTP Headers

This section has a toolbar with standard buttons. The grid contains a list of the HTTP headers defined for this collector. To add a HTTP header to the list, click  Create New. PhixFlow opens a new HTTP Header Properties properties tab. To remove a HTTP header, use the  Delete in the toolbar.

Some headers will be set to default values if not provided. Automatically added headers may not appear in the debug log. The Content-Length header will be added to all requests and cannot be overridden by providing a value.

Response

Define the data response type/format that will be returned:

• HTML: response type allows an XPath Expression to be specified in order to retrieve just specified sections of the data into XML structures.
• XML: response type allows an XPath Expression to be specified in order to retrieve just specified sections of the data into XML structures. XML response types also support XML namespaces. The Xml Namspaces tab will be available when this response type is chosen.
• String: response type will return the full data as a string value.

Please see Response Examples for how the returned data can be used and evaluated in the corresponding stream attribute expressions.

XML Namespaces

This section has a toolbar with standard buttons. The grid contains a list of the namespaces defined in an XML response.

To add a namespace to the list, click  Create New. PhixFlow opens a new XML Namespace Properties panel. To remove a namespace, use the  Delete in the toolbar.

Analysis Models

If this item is used by an analysis model, its name is listed here. See the Common Properties page, Analysis Model section for more details.

Advanced

Field	Description
Log Traffic	Tick to include HTTP traffic in the logs. This is useful for debugging problems.

Description

We recommend that you always enter a description to explain the purpose of this item.

 Audit Tab

 Audit Tab

Audit Summary

See the Common Properties page, Audit Summary section.

Response Examples

XML Data

<?xml version ="1.0"?> <root xmlns:h="http://www.w3.org/TR/html4/"> <main page="PF Main Page" > <h:title h:name="PF Title">PF Title Text <h:datarow> <h:data h:initials="AA">Alistair Andrews</h:data> <h:data h:initials="BB">Bert Brown</h:data> </h:datarow> </h:title> <title name="Non namespace Title">Non namespace Title Text</title> </main> </root>

HTML Data

<html> <body nodename="Html Body"> <table> <tbody> <tr> <td initials="AA">Alistair Andrews</td> <td initials="BB">Bert Brown</td> </tr> </tbody> </table> </body> </html>

The data is being pointed to by either HTTP datasources or XML/HTML File collectors respectively.

The following table shows the different types of responses that can be returned from an HTTP Collector and how these can be used in the corresponding stream attribute expressions. A HTTP Collector response type of XML/HTML will mimic the responses from XML/HTML Collectors respectively.

Advanced