6 Enriching Data Sets

By the end of this chapter you will be able to:

  • Create lookup pipes
  • Enrich a data set using a lookup pipe

Copy data from merge

First, you will create a new table to copy over all data from your merge:

  1. Hover over the table Customer All Details, and press  Create New Table.
  2. Call this table: Customer Details with Region.
  3. Copy all attributes into your new table:
    1. Hover over Customer All Details and in the hover menu press  Show attributes for <table>
    2. Select all attributes in the list:
      1. Select the top attribute.
      2. Keep shift pressed down and select the bottom attribute in the list.
    3. Drag the attributes on top of the table Customer Details with Region.
  4. Run your new table and look at the data in it to verify that you have copied data from the merge table.
  5. Remember to save your model layout.

Here you have copied data from a merge table into a new table – but this method works for any type of input table. Copying data into a new table is a common starting point for a modelling step.

Create a lookup into reference data

Now you will add a lookup pipe from Regions by County to enrich the data in Customer Details with Region, by finding the region for each customer address:

  1. Hover over RegionsByCounty and press  Connector
  2. Move the pointer over the top of Customer Details with Region and select – this will attach the end of the pipe to the selected table.
  3. In the pipe details form that pops up, enter:
    • Name: region
    • Type: Look-up
    • Data To Read: leave as the default – Latest
    • Press  Save.
    • In the Order/Index section we will set up the fields to match across the two recordsets:
      1. Press  Show Attributes.
      2. Drag the attribute County from the left hand list (the input attributes from the table RegionsByCounty) into the list of Order/Index attributes.
      3. Drag the attribute County from the right hand list (the output attributes from the table Customer Details with Region) on top of the Order/Index attribute County.
        • In the Order/Index section, you should see that the Index Expression column has updated to _out.County 

        • This means that the lookup will match records in the reference data (RegionsByCounty) where County equals County in the output data (Customer Details with Region).
      4. Press  OK
    • Note that the pipe linking RegionsByCounty to Customer Details with Region is dashed; this indicates that this is a lookup pipe.
    • When adding lookup pipes remember that the arrow points in the direction that the data flows; here you are getting data from RegionsByCounty to add to Customer Details with Region.

Enrich a data set using a Lookup Pipe

Now you can add the region from RegionsByCounty to Customer Details with Region:

  1. Hover over RegionsByCounty and in the hover menu press  Show attributes for <table>
  2. The list of the table attributes will pop up.
  3. Drag the attribute Region from RegionsByCounty on top of Customer Details with Region.
  4. Run analysis on Customer Details with Region.
  5. View the data in the table. Check that the Region attribute has been populated with the value looked up from RegionsByCounty.