This page is for a data modeller who needs to load data from an external source via HTTP.

Overview

An HTTP collector reads data from a HTTP Datasource. The collector defines how the data needed from the datasource is extracted to be used in PhixFlow.

To add a new HTTP collector to an analysis model:

1. Go to the model's toolbar → Create group.
2. Click  HTTP to expand the menu.
3. Drag a  HTTP Collector onto the analysis model.

To add an existing HTTP collector to an analysis model, in the model diagram toolbar:

1. Go to the model toolbar → List group.
2. Click  HTTP to expand the menu.
3. Click  HTTP Collector to open the list of available collectors.
4. Drag an HTTP collector into the analysis model.

.replaceAll(in.value, "[\r\n]", ""

{
  “line1” : “Single line of text" 
  “line2” : “Also a single 
line of text” 
 “line3” : “Multiple lines of↵
 text split over↵several lines” 
}

{“line1” : “Single line of text" “line2” : “Also a single line of text”, “line3” : “Multiple lines of text split over several lines” }

{“line1” : “Single line of text" “line2” : “Also a single line of text”, “line3” : “Multiple lines of/ntext split over/nseveral lines” }

.replaceAll(in.value, "[\r\n]", ""

Table Values in a HTTP Collector

To drive the lookups made by a HTTP collector from a table, the two must be connected using a lookup pipe. For example, a URL for a server can either be captured or calculated in an attribute called "ServerURL". The URL is then passed via a lookup pike to the HTTP collector to be used in its URL Expression. 

If the pipe is called in, here is how the URL Expression would be written on the HTTP collector: 
  {substring(in.ServerUrl,9)}

Property Pane Toolbar

For information about the toolbar options, see the Common Properties page, Toolbars and Controls section.

 Properties Tab

Parent Details

If this item is within or belongs to another, its parent name is shown here. See the Parent Details section on the Common Properties page for more details.

Basic Settings

Field	Description
Name	Enter the name of the HTTP collector.
Enabled	Tick when the configuration is complete and the collector is ready to be used.

Send Message

Define details of the HTTP request sent to the HTTP Datasource to get the data required.

<?xml version ="1.0"?>
<!DOCTYPE CORPORATE DASHBOARD "corpDash.dtd">
<results user="%USERNAME%" password="%PASSWORD%">
	<monthlyTotals region=${'"' + Region + '"'} division=${'"' + Division + '"'}>
		<totalBilled>${'"' + TotalBilled + '"'}</totalBilled>
		<totalCollected>${'"' + TotalCollected + '"'}</totalCollected>
	</monthlyTotals>
</results>

{
	user: '${user}',
	code:  ${'{size: "big"}'}
	price: ${price},
	currency: '${currency}'
}

HTTP Headers

This section has a toolbar with standard buttons. The grid contains a list of the HTTP headers defined for this collector. To add a HTTP header to the list, click  Create New. PhixFlow opens a new HTTP Header properties tab. To remove a HTTP header, use the  Delete in the toolbar.

Some headers will be set to default values if not provided. Automatically added headers may not appear in the debug log. The Content-Length header will be added to all requests and cannot be overridden by providing a value.

Response

Defines the data response type/format that will be returned and the desired location of the data.

Please see Response Examples for how the returned data can be used and evaluated in the corresponding attribute expressions.

XML Namespaces

Available when Return Type is XML or HTTP. This section has a toolbar with standard buttons. The grid contains a list of the namespaces defined in an XML response.

To add a namespace to the list, click  Create New. PhixFlow opens a new XML Namespace property pane. To remove a namespace, use the  Delete in the toolbar.

Analysis Models

If this item is used by an analysis model, its name is listed here. See the Common Properties page, Analysis Model section for more details.

Advanced

Field	Description
HTTP Datasource	Select the HTTP datasource that this collector will collect from. For how to add a new one, see HTTP Datasource.  To select from a list, click  Show the List of HTTP Datasources.
Datasource Instance Expression	The datasource to which this collector is connected may list multiple instances from which the data may be accessed. Each HTTP datasource instance is identified by a unique string. This expression should evaluate to a string which allows the collector to determine the specific instance to use. If the expression is blank then the collector will assume that there is only one instance and will use that one by default. If there is more than one instance and no expression is provided here then an error will be thrown during analysis since the collector will be unable to determine which source to use. See also  HTTP datasource instance and Expressions and PhixScripts.
Timeout (secs)	The number of seconds to wait for a response from the corresponding HTTP datasource before a timeout is recorded.
Icon	Enter the path for an image file that has been uploaded to the PhixFlow database. PhixFlow displays this icon in controls when the HTTP collector is used.
Allow Non-Scheduled Collection	Tick to run the HTTP collector as part of any ad-hoc analysis run that requires this data. If not, it will only run as part of a scheduled Task Plan.
Use Raw URL	If enabled the URL Template value is sent in exactly the format it is provided to the HTTP Node. If not enabled PhixFlow will transpose values to form a valid URL, such as replacing spaces with %20.
Log Traffic	You can set this option when system logging → Allow Logging is ticked; see System Logging Configuration. PhixFlow always logs HTTP responses and requests for HTTP collectors, whatever is set here.

Analysis Models

If this item is used by an analysis model, its name is listed here. See the Common Properties page, Analysis Model section for more details.

Description

We recommend that you always enter a description to explain the purpose of this item.

 Audit Tab

Audit Summary

See the Common Properties page, Audit Summary section.

Response Examples

XML Data

Standard

<root> 
	<main page="PF Main Page"> 
		<title name="PF Title">PF Title Text
			<datarow> 
				<data initials="AA">Alistair Andrews</data>
				<data initials="BB">Bert Brown</data> 
			</datarow> 
		</title> 
	</main> 
</root>

With Namespace

<root xmlns:h="http://example.com/schema"> 
	<main page="PF Main Page"> 
		<h:title name="PF Title">PF Title Text
			<h:datarow> 
				<h:data h:initials="AA">Alistair Andrews</data>
				<h:data h:initials="BB">Bert Brown</data> 
			</h:datarow> 
		</h:title> 
	</main> 
</root>

HTML Data

<html>
	<body nodename="Html Body"> 
		<table> 
			<tbody> 
				<tr> 
					<td initials="AA">Alistair Andrews</td> 
					<td initials="BB">Bert Brown</td> 
				</tr> 
			</tbody> 
		</table> 
	</body> 
</html>

JSON

{
	"main_page": {
		"page": "PF Main Page",
		"title" : {
			"name" : "PF Title Text,
			"data" : [
				{"initials": "AA", "value" : "Alistair Andrews"},
				{"initials": "BB", "value" : "Bert Brown"}
			]
		}
	}
}

The data is being pointed to by either HTTP datasources or XML/HTML File collectors respectively.

The following table shows the different types of responses that can be returned from an HTTP Collector and how these can be used in the corresponding attribute expressions. A HTTP Collector response type of XML/HTML will mimic the responses from XML/HTML Collectors respectively.