The enrichment module is a module dedicated to taking the data
from the parsers that have been normalized into the Metron
data format (e.g. a JSON Map structure with original_message and
timestamp) and
- Enriching messages with external data from data stores (e.g. hbase) by adding new fields based on existing fields in the messages.
- Marking messages as threats based on data in external data stores
- Marking threat alerts with a numeric triage level based on a set of Stellar rules.
There is currently one option for running enrichments in Metron, which is as a Storm topology.
- metron-enrichment-common - this module houses the prepackaged enrichment configuration by sensor. It also contains the core enrichment and threat intelligence processing functionality.
- metron-common-storm - this module is home to Storm-specific code such as Flux files and Storm Bolts.
Metron provides an HBase table for storing enrichments. The rowkeys are a combination of a salt (for managing RegionServer hotspotting), indicator (this would be the search value, e.g. "192.168.1.1"), and type (whois, geoip, etc.).
This approach performs well for both inserts and lookups, but poses a challenge when looking to get an
up-to-date list of all the current enrichments. This is of particular concern for the Management UI where
it's desirable to provide an accurate list of all available enrichment types.
A table scan is undesirable because it results in a performance hit for inserts and reads.
The alternative approach that mitigates these performance bottlenecks is to leverage a custom HBase
Coprocessor which will listen to postPut calls from the RegionServer, extract the enrichment type from the rowkey, and
perform an insert into a separate enrichment_list HBase table.
See more about configuring the coprocessor here Enrichment Coprocessor