Deduplicate mode

Specify a field in an incoming Events and only emit the Event if that field contains a unique value.


  • Define how many historical events from previous story runs should be used to determine if value in field is unique.

  • Specify a period of time to check for duplicates.

  • Specify the field in an incoming Event to use as a measure of uniqueness.

  • If an Event is deemed unique, it will be emitted. If it is deemed a duplicate, it will be ignored and a corresponding log entry will be created.

Configuration Options 

  • mode: 'deduplicate'

  • path: a JSON wrapped path the value of which should be used to determine the uniqueness of the event.

  • lookback: (Optional: must use either lookback or period) Number of past events (maximum of 1000) to examine for uniqueness.

  • period: (Optional: must use either lookback or period) Number of seconds to use for deduplication. When an event is received, subsequent events will not be emitted until this period has elapsed.

  • emit_duplicates: (Optional) Set to true if duplicate events should be emitted. 'unique_event': false will be included in emitted event.

  • tolerance: (Optional) The number of events with the same value to allow before deduplicating. Events of the same value will be treated as unique until the provided tolerance is reached.

Emitted Events 

  "unique_event": true

Example Configuration Options 

Check the last 100 received Events and only emit an Event if the value in the field is unique.

  "mode": "deduplicate",
  "lookback": "100",
  "path": "<<>>"
Was this helpful?