CenterView Help

Skip to end of metadata
Go to start of metadata

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

Version 1 Next »

Data Enrichment Options

Scenario

Very often, a data extract will contains lots of references, and keys and links, but not necessarily data elements that are readable or useful to users. For example, lots of database tables have "type" fields where 1=home, 2=office, 3=school ... etc. An output report would want to show the text value of type, rather than just the number.

There are numerous ways to do data enrichment, the most common methods are:

  • Merge All data and then filter out unwanted data
  • Lookup to DB, via a lookup pipe to the DB collector
  • Use a SQL IN clause via an Input Multiplier

Solution:

  • Create a stream to serve as the source of the lookup.
  • Create a DB collector to retrieve the data that needs to be "looked up"
  • Implement data merges based on the 3 methods listed above.

Note : where stream and data sizes are small, the lookup method will have a negligible impact on the overall model. However, where data volumes are very large, choosing the correct lookup / enrichment solution can greatly impact the performance of the model. Try to determine which of the 3 methods listed here should be used in the following instances :

  • Stream contains 5,000 records, and DB table for lookup contains 200,000,000 records.
  • Stream contains 10,000,000 records, and DB table for lookup contains 10,000,000 records with an expected hit rate of 5%
  • Stream contains 10,000,000 records, and DB table for lookup contains 10,000,000 records with an expected hit rate of 85%
  • Stream contains 100,000,000 records, and DB table for lookup contains 100,000 records
What other data volume scenarios might impact data enrichment?

See Also

  • href="../../../../../../doc/training/Exercises.pdf">Patterns and Exercise Training Document
  • href="http://www.centerview-online.com:8081/patterns">Centerview Online Patterns and Training
  • No labels