Bender provides an extendable Java framework for creating serverless ETL functions on AWS Lambda.
It handles the complex plumbing and provides the interfaces necessary to build modules for all aspects of the ETL
process. Check it out at https://github.com/nextdoor/bender.
{"foo": <BAR>}
Note that if BENDER_SKIP_VALIDATE=true env var is set then Bender will not validate the configuration at runtime.
Use this if you validate the configuration files with the CLI tool prior to deployment.
Handler configuration
For use with DynamoDB stream triggers. Set the function handler to "com.nextdoor.bender.handler.dynamodb.DynamodbHandler::handler". The following IAM permissions are also required: dynamodb:DescribeStream, dynamodb:GetRecords, dynamodb:GetShardIterator, and dynamodb:ListStreams. DynamoDB stream events use AttributeValues (see https://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_AttributeValue.html) which uses data type as name, but such construct is converted to standard JSON by DynamodbHandler. For example, {"value":{"N":"123"}} becomes {"value":123}.
If an uncaught exception occurs fail the function
Adds Lambda function resource tags to reporters' metrics. Note that the lambda:ListTags IAM permission is required.
Additional tags to add to reporters' metrics. Note Lambda function resource tags take precedence.
{
"type": "object",
"additionalProperties": false,
"properties": {
"key": {
"type": "string"
},
"value": {
"type": "string"
}
},
"required": [
"key",
"value"
]
}
Maximum queue size used to buffer raw data prior to deserialization. This adds back pressure that ensures Bender does not read quicker than it can process and send to your desination (transport). Increasing the buffer will increase memory pressure and risk of OOM but will have the benefit of increasing throughput.
{
"type": "object",
"additionalProperties": false,
"description": "For use with DynamoDB stream triggers. Set the function handler to \"com.nextdoor.bender.handler.dynamodb.DynamodbHandler::handler\". The following IAM permissions are also required: dynamodb:DescribeStream, dynamodb:GetRecords, dynamodb:GetShardIterator, and dynamodb:ListStreams. DynamoDB stream events use AttributeValues (see https://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_AttributeValue.html) which uses data type as name, but such construct is converted to standard JSON by DynamodbHandler. For example, {\"value\":{\"N\":\"123\"}} becomes {\"value\":123}.",
"properties": {
"type": {
"type": "string",
"enum": [
"DynamodbHandler"
],
"default": "DynamodbHandler"
},
"fail_on_exception": {
"type": "boolean",
"default": true,
"description": "If an uncaught exception occurs fail the function"
},
"include_function_tags": {
"type": "boolean",
"default": false,
"description": "Adds Lambda function resource tags to reporters' metrics. Note that the lambda:ListTags IAM permission is required."
},
"metric_tags": {
"type": "array",
"items": {
"$ref": "#/definitions/Tag"
},
"description": "Additional tags to add to reporters' metrics. Note Lambda function resource tags take precedence."
},
"queue_size": {
"type": "integer",
"default": 500,
"description": "Maximum queue size used to buffer raw data prior to deserialization. This adds back pressure that ensures Bender does not read quicker than it can process and send to your desination (transport). Increasing the buffer will increase memory pressure and risk of OOM but will have the benefit of increasing throughput."
}
},
"title": "DynamodbHandler",
"required": [
"type"
]
}
For use with S3 Object creation Lambda triggers. You will also need to set the function handler to "com.nextdoor.bender.handler.s3.S3Handler::handler". The following IAM permissions are required: s3:GetObject and lambda:InvokeFunction. Note you will also need to provide permissoins to the S3 bucket to invoke your function.
If an uncaught exception occurs fail the function
Adds Lambda function resource tags to reporters' metrics. Note that the lambda:ListTags IAM permission is required.
Additional tags to add to reporters' metrics. Note Lambda function resource tags take precedence.
{
"type": "object",
"additionalProperties": false,
"properties": {
"key": {
"type": "string"
},
"value": {
"type": "string"
}
},
"required": [
"key",
"value"
]
}
Maximum queue size used to buffer raw data prior to deserialization. This adds back pressure that ensures Bender does not read quicker than it can process and send to your desination (transport). Increasing the buffer will increase memory pressure and risk of OOM but will have the benefit of increasing throughput.
SNS Topic to publish function falures to. Do not use this for function retries. Instead use DLQs http://docs.aws.amazon.com/lambda/latest/dg/dlq.html. This requires IAM permission SNS:Publish.
Logs the original S3 notification that triggered the function.
{
"type": "object",
"additionalProperties": false,
"description": "For use with S3 Object creation Lambda triggers. You will also need to set the function handler to \"com.nextdoor.bender.handler.s3.S3Handler::handler\". The following IAM permissions are required: s3:GetObject and lambda:InvokeFunction. Note you will also need to provide permissoins to the S3 bucket to invoke your function.",
"properties": {
"type": {
"type": "string",
"enum": [
"S3Handler"
],
"default": "S3Handler"
},
"fail_on_exception": {
"type": "boolean",
"default": true,
"description": "If an uncaught exception occurs fail the function"
},
"include_function_tags": {
"type": "boolean",
"default": false,
"description": "Adds Lambda function resource tags to reporters' metrics. Note that the lambda:ListTags IAM permission is required."
},
"metric_tags": {
"type": "array",
"items": {
"$ref": "#/definitions/Tag"
},
"description": "Additional tags to add to reporters' metrics. Note Lambda function resource tags take precedence."
},
"queue_size": {
"type": "integer",
"default": 500,
"description": "Maximum queue size used to buffer raw data prior to deserialization. This adds back pressure that ensures Bender does not read quicker than it can process and send to your desination (transport). Increasing the buffer will increase memory pressure and risk of OOM but will have the benefit of increasing throughput."
},
"sns_notification_arn": {
"type": "string",
"description": "SNS Topic to publish function falures to. Do not use this for function retries. Instead use DLQs http://docs.aws.amazon.com/lambda/latest/dg/dlq.html. This requires IAM permission SNS:Publish."
},
"log_s3_trigger": {
"type": "boolean",
"default": false,
"description": "Logs the original S3 notification that triggered the function."
}
},
"title": "S3Handler",
"required": [
"type"
]
}
Similar to the S3Handler but reads SNS notifications which contain S3 Object creation events. This is may be required if you have multiple functions running against the same bucket. For more information about S3 SNS triggers see https://aws.amazon.com/blogs/compute/fanout-s3-event-notifications-to-multiple-endpoints/.You will need to set the function handler to "com.nextdoor.bender.handler.s3.SNSS3Handler::handler". The following IAM permissions are required: s3:GetObject and lambda:InvokeFunction. Note you will also need to provide permissions to S3 bucket to publish to SNS and SNS to invoke your function.
If an uncaught exception occurs fail the function
Adds Lambda function resource tags to reporters' metrics. Note that the lambda:ListTags IAM permission is required.
Additional tags to add to reporters' metrics. Note Lambda function resource tags take precedence.
{
"type": "object",
"additionalProperties": false,
"properties": {
"key": {
"type": "string"
},
"value": {
"type": "string"
}
},
"required": [
"key",
"value"
]
}
Maximum queue size used to buffer raw data prior to deserialization. This adds back pressure that ensures Bender does not read quicker than it can process and send to your desination (transport). Increasing the buffer will increase memory pressure and risk of OOM but will have the benefit of increasing throughput.
SNS Topic to publish function falures to. Do not use this for function retries. Instead use DLQs http://docs.aws.amazon.com/lambda/latest/dg/dlq.html. This requires IAM permission SNS:Publish.
Logs the original SNS notification that triggered the function.
{
"type": "object",
"additionalProperties": false,
"description": "Similar to the S3Handler but reads SNS notifications which contain S3 Object creation events. This is may be required if you have multiple functions running against the same bucket. For more information about S3 SNS triggers see https://aws.amazon.com/blogs/compute/fanout-s3-event-notifications-to-multiple-endpoints/.You will need to set the function handler to \"com.nextdoor.bender.handler.s3.SNSS3Handler::handler\". The following IAM permissions are required: s3:GetObject and lambda:InvokeFunction. Note you will also need to provide permissions to S3 bucket to publish to SNS and SNS to invoke your function.",
"properties": {
"type": {
"type": "string",
"enum": [
"SNSS3Handler"
],
"default": "SNSS3Handler"
},
"fail_on_exception": {
"type": "boolean",
"default": true,
"description": "If an uncaught exception occurs fail the function"
},
"include_function_tags": {
"type": "boolean",
"default": false,
"description": "Adds Lambda function resource tags to reporters' metrics. Note that the lambda:ListTags IAM permission is required."
},
"metric_tags": {
"type": "array",
"items": {
"$ref": "#/definitions/Tag"
},
"description": "Additional tags to add to reporters' metrics. Note Lambda function resource tags take precedence."
},
"queue_size": {
"type": "integer",
"default": 500,
"description": "Maximum queue size used to buffer raw data prior to deserialization. This adds back pressure that ensures Bender does not read quicker than it can process and send to your desination (transport). Increasing the buffer will increase memory pressure and risk of OOM but will have the benefit of increasing throughput."
},
"sns_notification_arn": {
"type": "string",
"description": "SNS Topic to publish function falures to. Do not use this for function retries. Instead use DLQs http://docs.aws.amazon.com/lambda/latest/dg/dlq.html. This requires IAM permission SNS:Publish."
},
"log_sns_trigger": {
"type": "boolean",
"default": false,
"description": "Logs the original SNS notification that triggered the function."
}
},
"title": "SNSS3Handler",
"required": [
"type"
]
}
For use with Kinesis triggers. Set the function handler to "com.nextdoor.bender.handler.kinesis.KinesisHandler::handler". The following IAM permissions are also required: kinesis:DescribeStream, kinesis:ListStreams, kinesis:GetShardIterator, kinesis:GetRecords, and kinesis:ListTagsForStream.
If an uncaught exception occurs fail the function
Adds Lambda function resource tags to reporters' metrics. Note that the lambda:ListTags IAM permission is required.
Additional tags to add to reporters' metrics. Note Lambda function resource tags take precedence.
{
"type": "object",
"additionalProperties": false,
"properties": {
"key": {
"type": "string"
},
"value": {
"type": "string"
}
},
"required": [
"key",
"value"
]
}
Maximum queue size used to buffer raw data prior to deserialization. This adds back pressure that ensures Bender does not read quicker than it can process and send to your desination (transport). Increasing the buffer will increase memory pressure and risk of OOM but will have the benefit of increasing throughput.
Whether to add kinesis shardid to the event partitions list. The key is "shardid" and value will look like "shardId-000000000000". Note that partitioning must be either enabled or supported by the transport you use. Not all transporters support partitioning.
If this flag is set to true, then the Kinesis handler will assume all incoming Kinesis record's data will be a gzip that needs to be decompressed before it's stored as a string in the KinesisInternalEvent. One example use case is how using a CloudWatch log subscription filter on Kinesis results in data being stored as a zip that is base64 encoded. The Kinesis Java SDK willtake care of decoding so this flag ensures the gzip is inflated.
This sets the buffer size (default 1024) when Kinesis data is a gzip and needs to be expanded. This flag is only used if the getAssumeKinesisDataIsGzipped config is set as true.
{
"type": "object",
"additionalProperties": false,
"description": "For use with Kinesis triggers. Set the function handler to \"com.nextdoor.bender.handler.kinesis.KinesisHandler::handler\". The following IAM permissions are also required: kinesis:DescribeStream, kinesis:ListStreams, kinesis:GetShardIterator, kinesis:GetRecords, and kinesis:ListTagsForStream.",
"properties": {
"type": {
"type": "string",
"enum": [
"KinesisHandler"
],
"default": "KinesisHandler"
},
"fail_on_exception": {
"type": "boolean",
"default": true,
"description": "If an uncaught exception occurs fail the function"
},
"include_function_tags": {
"type": "boolean",
"default": false,
"description": "Adds Lambda function resource tags to reporters' metrics. Note that the lambda:ListTags IAM permission is required."
},
"metric_tags": {
"type": "array",
"items": {
"$ref": "#/definitions/Tag"
},
"description": "Additional tags to add to reporters' metrics. Note Lambda function resource tags take precedence."
},
"queue_size": {
"type": "integer",
"default": 500,
"description": "Maximum queue size used to buffer raw data prior to deserialization. This adds back pressure that ensures Bender does not read quicker than it can process and send to your desination (transport). Increasing the buffer will increase memory pressure and risk of OOM but will have the benefit of increasing throughput."
},
"add_kinesis_shard_to_partitions": {
"type": "boolean",
"default": false,
"description": "Whether to add kinesis shardid to the event partitions list. The key is \"__shardid__\" and value will look like \"shardId-000000000000\". Note that partitioning must be either enabled or supported by the transport you use. Not all transporters support partitioning."
},
"decompress": {
"type": "boolean",
"default": false,
"description": "If this flag is set to true, then the Kinesis handler will assume all incoming Kinesis record's data will be a gzip that needs to be decompressed before it's stored as a string in the KinesisInternalEvent. One example use case is how using a CloudWatch log subscription filter on Kinesis results in data being stored as a zip that is base64 encoded. The Kinesis Java SDK willtake care of decoding so this flag ensures the gzip is inflated."
},
"buffer_size": {
"type": "integer",
"description": "This sets the buffer size (default 1024) when Kinesis data is a gzip and needs to be expanded. This flag is only used if the getAssumeKinesisDataIsGzipped config is set as true."
},
"add_shardid_to_partitions": {
"type": "boolean"
}
},
"title": "KinesisHandler",
"required": [
"type"
]
}
Source configurations. This includes deserializer and operators.
Source name
Pattern to match source trigger against
Regex patterns to filter events by prior to deserialization
Filter events containing these Strings prior to deserialization
Deserializer configuration
Extracts fields from a line with regex groups. Note that number of groups in regex must match number of fields specified.
List of field names and their types
Name to give to field
Data type of field
{
"type": "object",
"additionalProperties": false,
"properties": {
"name": {
"type": "string",
"description": "Name to give to field"
},
"type": {
"type": "string",
"enum": [
"STRING",
"NUMBER",
"BOOLEAN"
],
"description": "Data type of field"
}
},
"required": [
"name",
"type"
]
}
Use the faster but less flexible regex library. See https://github.com/google/re2j
{
"type": "object",
"additionalProperties": false,
"description": "Extracts fields from a line with regex groups. Note that number of groups in regex must match number of fields specified.",
"properties": {
"type": {
"type": "string",
"enum": [
"Regex"
],
"default": "Regex"
},
"fields": {
"type": "array",
"items": {
"$ref": "#/definitions/ReFieldConfig"
},
"description": "List of field names and their types"
},
"regex": {
"type": "string",
"description": "Java regex. See https://docs.oracle.com/javase/8/docs/api/java/util/regex/Pattern.html"
},
"use_re2j": {
"type": "boolean",
"default": false,
"description": "Use the faster but less flexible regex library. See https://github.com/google/re2j"
}
},
"title": "Regex",
"required": [
"type",
"fields",
"regex"
]
}
Deserializes JSON without performing schema validation.
Configuration on how to interpret string fields which can also be deserialized as JSON. For example: {"foo": "{"bar": "baz"}"} will become {"foo": {"bar": "baz"}"}.
String field which contains a JSON object.
Field to put any data which preceded JSON object.
{
"type": "object",
"additionalProperties": false,
"properties": {
"field": {
"type": "string",
"description": "String field which contains a JSON object."
},
"prefix_field": {
"type": "string",
"description": "Field to put any data which preceded JSON object."
}
},
"required": [
"field"
]
}
Path to a JSON node which is promoted to root node. See https://github.com/jayway/JsonPath
{
"type": "object",
"additionalProperties": false,
"description": "Deserializes JSON without performing schema validation.",
"properties": {
"type": {
"type": "string",
"enum": [
"GenericJson"
],
"default": "GenericJson"
},
"nested_field_configs": {
"type": "array",
"items": {
"$ref": "#/definitions/FieldConfig"
},
"description": "Configuration on how to interpret string fields which can also be deserialized as JSON. For example: {\"foo\": \"{\"bar\": \"baz\"}\"} will become {\"foo\": {\"bar\": \"baz\"}\"}."
},
"root_node_override_path": {
"type": "string",
"description": "Path to a JSON node which is promoted to root node. See https://github.com/jayway/JsonPath"
}
},
"title": "GenericJson",
"required": [
"type"
]
}
Operation configuration
The fork operation allows nesting multiple operation pipelines within Bender. Each fork (pipeline) has its own operations and operates independently on a clone of the original event. If no filters are applied then this can result in multiple output events for each input event.
List of forks.
List of operations to perform.
circular reference
{
"description": "_circular reference_"
}
Provided a JSON object it will recursively lower case all fields.
{
"type": "object",
"additionalProperties": false,
"description": "Provided a JSON object it will recursively lower case all fields.",
"properties": {
"type": {
"type": "string",
"enum": [
"LowerCaseKeyOperation"
],
"default": "LowerCaseKeyOperation"
}
},
"title": "LowerCaseKeyOperation",
"required": [
"type"
]
}
Provided a JSON object and a path within the object it will promote the path's element to the root position. For example if the input is {"foo": {"bar": {"baz": 1}}} and specified path $.foo.bar the operation will produce {"baz": 1} as the new payload.
Path to a JSON node which is promoted to root node. See https://github.com/jayway/JsonPath
{
"type": "object",
"additionalProperties": false,
"description": "Provided a JSON object and a path within the object it will promote the path's element to the root position. For example if the input is {\"foo\": {\"bar\": {\"baz\": 1}}} and specified path $.foo.bar the operation will produce {\"baz\": 1} as the new payload.",
"properties": {
"type": {
"type": "string",
"enum": [
"JsonRootNodeOperation"
],
"default": "JsonRootNodeOperation"
},
"root_path": {
"type": "string",
"description": "Path to a JSON node which is promoted to root node. See https://github.com/jayway/JsonPath"
}
},
"title": "JsonRootNodeOperation",
"required": [
"type",
"root_path"
]
}
This operation allows substituting event fields with different sources such as other fields, static values, or metadata.
List of substitutions to perform.
Substitutes event field value for another event field value. Note the source field and destination field can be the same.
Name of the new field
Fail if destination field does not exist. This can be the case if attempting to insert into a nested destination which was not yet created.
Source fields to pull value from. If multiple fields are provided the first non-null valued one is used. Note that if no fields are found the value will be set to null.
Removes the source field when performing the substitution. Effectively making this a move operation.
Fail if source field is not found.
{
"type": "object",
"additionalProperties": false,
"description": "Substitutes event field value for another event field value. Note the source field and destination field can be the same.",
"properties": {
"type": {
"type": "string",
"enum": [
"FieldSubstitution"
],
"default": "FieldSubstitution"
},
"key": {
"type": "string",
"description": "Name of the new field"
},
"fail_dst_not_found": {
"type": "boolean",
"default": true,
"description": "Fail if destination field does not exist. This can be the case if attempting to insert into a nested destination which was not yet created."
},
"src_fields": {
"type": "array",
"items": {
"type": "string"
},
"description": "Source fields to pull value from. If multiple fields are provided the first non-null valued one is used. Note that if no fields are found the value will be set to null."
},
"remove_src_field": {
"type": "boolean",
"default": false,
"description": "Removes the source field when performing the substitution. Effectively making this a move operation."
},
"fail_src_not_found": {
"type": "boolean",
"default": true,
"description": "Fail if source field is not found."
}
},
"title": "FieldSubstitution",
"required": [
"type",
"key",
"src_fields"
]
}
Substitutes event field value for another event field value. Note the source field and destination field can be the same.
Fail if destination field does not exist. This can be the case if attempting to insert into a nested destination which was not yet created.
Regex pattern with match groups. See https://docs.oracle.com/javase/8/docs/api/java/util/regex/Pattern.html
Source fields to pull value from and apply regex to. If multiple fields are provided the first non-null valued one is used.
List of fields to create from matching regex groups.
Regex group name identifying the field.
Data type of match group field. If type coercion does not succeed then field is ignored.
Name or path of the new field.
{
"type": "object",
"additionalProperties": false,
"properties": {
"regex_group_name": {
"type": "string",
"description": "Regex group name identifying the field."
},
"type": {
"type": "string",
"enum": [
"STRING",
"NUMBER",
"BOOLEAN"
],
"description": "Data type of match group field. If type coercion does not succeed then field is ignored."
},
"key": {
"type": "string",
"description": "Name or path of the new field."
}
},
"required": [
"regex_group_name",
"type",
"key"
]
}
Removes the source field after applying this substitution.
Fail if source fields do not match regex or are not found.
{
"type": "object",
"additionalProperties": false,
"description": "Substitutes event field value for another event field value. Note the source field and destination field can be the same.",
"properties": {
"type": {
"type": "string",
"enum": [
"RegexSubstitution"
],
"default": "RegexSubstitution"
},
"fail_dst_not_found": {
"type": "boolean",
"default": true,
"description": "Fail if destination field does not exist. This can be the case if attempting to insert into a nested destination which was not yet created."
},
"pattern": {
"type": "string",
"description": "Regex pattern with match groups. See https://docs.oracle.com/javase/8/docs/api/java/util/regex/Pattern.html"
},
"src_fields": {
"type": "array",
"items": {
"type": "string"
},
"description": "Source fields to pull value from and apply regex to. If multiple fields are provided the first non-null valued one is used."
},
"fields": {
"type": "array",
"items": {
"$ref": "#/definitions/RegexSubField"
},
"description": "List of fields to create from matching regex groups."
},
"remove_src_field": {
"type": "boolean",
"default": false,
"description": "Removes the source field after applying this substitution."
},
"fail_src_not_found": {
"type": "boolean",
"default": true,
"description": "Fail if source fields do not match regex or are not found."
}
},
"title": "RegexSubstitution",
"required": [
"type",
"pattern",
"src_fields",
"fields"
]
}
Substitutes event field value for list of event metadata proprties.
Name of the new field
Fail if destination field does not exist. This can be the case if attempting to insert into a nested destination which was not yet created.
List of metadata fields to add. If non-specified then all are added.
List of metadata fields to exclude. If non-specified then none are excluded.
{
"type": "object",
"additionalProperties": false,
"description": "Substitutes event field value for list of event metadata proprties.",
"properties": {
"type": {
"type": "string",
"enum": [
"MetadataSubstitution"
],
"default": "MetadataSubstitution"
},
"key": {
"type": "string",
"description": "Name of the new field"
},
"fail_dst_not_found": {
"type": "boolean",
"default": true,
"description": "Fail if destination field does not exist. This can be the case if attempting to insert into a nested destination which was not yet created."
},
"includes": {
"type": "array",
"items": {
"type": "string"
},
"description": "List of metadata fields to add. If non-specified then all are added."
},
"excludes": {
"type": "array",
"items": {
"type": "string"
},
"description": "List of metadata fields to exclude. If non-specified then none are excluded."
}
},
"title": "MetadataSubstitution",
"required": [
"type",
"key"
]
}
The nested substitution helps build a Map object containing the result of other substitution.
Name of the new field
Fail if destination field does not exist. This can be the case if attempting to insert into a nested destination which was not yet created.
List of substitutions that will build up the final Map object.
Substitutes event field value for another event field value. Note the source field and destination field can be the same.
Name of the new field
Fail if destination field does not exist. This can be the case if attempting to insert into a nested destination which was not yet created.
Source fields to pull value from. If multiple fields are provided the first non-null valued one is used. Note that if no fields are found the value will be set to null.
Removes the source field when performing the substitution. Effectively making this a move operation.
Fail if source field is not found.
{
"type": "object",
"additionalProperties": false,
"description": "Substitutes event field value for another event field value. Note the source field and destination field can be the same.",
"properties": {
"type": {
"type": "string",
"enum": [
"FieldSubstitution"
],
"default": "FieldSubstitution"
},
"key": {
"type": "string",
"description": "Name of the new field"
},
"fail_dst_not_found": {
"type": "boolean",
"default": true,
"description": "Fail if destination field does not exist. This can be the case if attempting to insert into a nested destination which was not yet created."
},
"src_fields": {
"type": "array",
"items": {
"type": "string"
},
"description": "Source fields to pull value from. If multiple fields are provided the first non-null valued one is used. Note that if no fields are found the value will be set to null."
},
"remove_src_field": {
"type": "boolean",
"default": false,
"description": "Removes the source field when performing the substitution. Effectively making this a move operation."
},
"fail_src_not_found": {
"type": "boolean",
"default": true,
"description": "Fail if source field is not found."
}
},
"title": "FieldSubstitution",
"required": [
"type",
"key",
"src_fields"
]
}
Substitutes event field value for another event field value. Note the source field and destination field can be the same.
Fail if destination field does not exist. This can be the case if attempting to insert into a nested destination which was not yet created.
Regex pattern with match groups. See https://docs.oracle.com/javase/8/docs/api/java/util/regex/Pattern.html
Source fields to pull value from and apply regex to. If multiple fields are provided the first non-null valued one is used.
List of fields to create from matching regex groups.
Regex group name identifying the field.
Data type of match group field. If type coercion does not succeed then field is ignored.
Name or path of the new field.
{
"type": "object",
"additionalProperties": false,
"properties": {
"regex_group_name": {
"type": "string",
"description": "Regex group name identifying the field."
},
"type": {
"type": "string",
"enum": [
"STRING",
"NUMBER",
"BOOLEAN"
],
"description": "Data type of match group field. If type coercion does not succeed then field is ignored."
},
"key": {
"type": "string",
"description": "Name or path of the new field."
}
},
"required": [
"regex_group_name",
"type",
"key"
]
}
Removes the source field after applying this substitution.
Fail if source fields do not match regex or are not found.
{
"type": "object",
"additionalProperties": false,
"description": "Substitutes event field value for another event field value. Note the source field and destination field can be the same.",
"properties": {
"type": {
"type": "string",
"enum": [
"RegexSubstitution"
],
"default": "RegexSubstitution"
},
"fail_dst_not_found": {
"type": "boolean",
"default": true,
"description": "Fail if destination field does not exist. This can be the case if attempting to insert into a nested destination which was not yet created."
},
"pattern": {
"type": "string",
"description": "Regex pattern with match groups. See https://docs.oracle.com/javase/8/docs/api/java/util/regex/Pattern.html"
},
"src_fields": {
"type": "array",
"items": {
"type": "string"
},
"description": "Source fields to pull value from and apply regex to. If multiple fields are provided the first non-null valued one is used."
},
"fields": {
"type": "array",
"items": {
"$ref": "#/definitions/RegexSubField"
},
"description": "List of fields to create from matching regex groups."
},
"remove_src_field": {
"type": "boolean",
"default": false,
"description": "Removes the source field after applying this substitution."
},
"fail_src_not_found": {
"type": "boolean",
"default": true,
"description": "Fail if source fields do not match regex or are not found."
}
},
"title": "RegexSubstitution",
"required": [
"type",
"pattern",
"src_fields",
"fields"
]
}
Substitutes event field value for list of event metadata proprties.
Name of the new field
Fail if destination field does not exist. This can be the case if attempting to insert into a nested destination which was not yet created.
List of metadata fields to add. If non-specified then all are added.
List of metadata fields to exclude. If non-specified then none are excluded.
{
"type": "object",
"additionalProperties": false,
"description": "Substitutes event field value for list of event metadata proprties.",
"properties": {
"type": {
"type": "string",
"enum": [
"MetadataSubstitution"
],
"default": "MetadataSubstitution"
},
"key": {
"type": "string",
"description": "Name of the new field"
},
"fail_dst_not_found": {
"type": "boolean",
"default": true,
"description": "Fail if destination field does not exist. This can be the case if attempting to insert into a nested destination which was not yet created."
},
"includes": {
"type": "array",
"items": {
"type": "string"
},
"description": "List of metadata fields to add. If non-specified then all are added."
},
"excludes": {
"type": "array",
"items": {
"type": "string"
},
"description": "List of metadata fields to exclude. If non-specified then none are excluded."
}
},
"title": "MetadataSubstitution",
"required": [
"type",
"key"
]
}
circular reference
{
"description": "_circular reference_"
}
Creates a new string using variable replacement.
Name of the new field
Fail if destination field does not exist. This can be the case if attempting to insert into a nested destination which was not yet created.
String with variable indices to substitue. See https://docs.oracle.com/javase/8/docs/api/java/text/MessageFormat.html
List of variables used in string substitution. Index of variable relates to index in 'format' string.
Source fields to pull value from. If multiple fields are provided the first non-null valued one is used. Note that if no fields are found the value will be set to null.
Fail if source field is not found.
Removes the source field when performing the substitution.
{
"type": "object",
"additionalProperties": false,
"properties": {
"type": {
"type": "string",
"enum": [
"FieldVariable"
],
"default": "FieldVariable"
},
"src_fields": {
"type": "array",
"items": {
"type": "string"
},
"description": "Source fields to pull value from. If multiple fields are provided the first non-null valued one is used. Note that if no fields are found the value will be set to null."
},
"fail_src_not_found": {
"type": "boolean",
"default": true,
"description": "Fail if source field is not found."
},
"remove_src_field": {
"type": "boolean",
"default": false,
"description": "Removes the source field when performing the substitution."
}
},
"title": "FieldVariable",
"required": [
"type",
"src_fields"
]
}
Value to substitute.
{
"type": "object",
"additionalProperties": false,
"properties": {
"type": {
"type": "string",
"enum": [
"StaticVariable"
],
"default": "StaticVariable"
},
"value": {
"type": "string",
"description": "Value to substitute."
}
},
"title": "StaticVariable",
"required": [
"type",
"value"
]
}
{
"type": "object",
"additionalProperties": false,
"description": "Creates a new string using variable replacement.",
"properties": {
"type": {
"type": "string",
"enum": [
"FormattedSubstitution"
],
"default": "FormattedSubstitution"
},
"key": {
"type": "string",
"description": "Name of the new field"
},
"fail_dst_not_found": {
"type": "boolean",
"default": true,
"description": "Fail if destination field does not exist. This can be the case if attempting to insert into a nested destination which was not yet created."
},
"format": {
"type": "string",
"description": "String with variable indices to substitue. See https://docs.oracle.com/javase/8/docs/api/java/text/MessageFormat.html"
},
"variables": {
"type": "array",
"description": "List of variables used in string substitution. Index of variable relates to index in 'format' string.",
"anyOf": [
{
"$ref": "#/definitions/FieldVariable"
},
{
"$ref": "#/definitions/StaticVariable"
}
]
}
},
"title": "FormattedSubstitution",
"required": [
"type",
"key",
"format",
"variables"
]
}
Substitutes event field value for a static value.
Name of the new field
Fail if destination field does not exist. This can be the case if attempting to insert into a nested destination which was not yet created.
Value of the new field.
{
"type": "object",
"additionalProperties": false,
"description": "Substitutes event field value for a static value.",
"properties": {
"type": {
"type": "string",
"enum": [
"StaticSubstitution"
],
"default": "StaticSubstitution"
},
"key": {
"type": "string",
"description": "Name of the new field"
},
"fail_dst_not_found": {
"type": "boolean",
"default": true,
"description": "Fail if destination field does not exist. This can be the case if attempting to insert into a nested destination which was not yet created."
},
"value": {
"type": "string",
"description": "Value of the new field."
}
},
"title": "StaticSubstitution",
"required": [
"type",
"key",
"value"
]
}
Substitutes event field value for list of lambda function context proprties.
Name of the new field
Fail if destination field does not exist. This can be the case if attempting to insert into a nested destination which was not yet created.
List of context fields to add. If non-specified then all are added.
List of context fields to exclude. If non-specified then none are excluded.
{
"type": "object",
"additionalProperties": false,
"description": "Substitutes event field value for list of lambda function context proprties.",
"properties": {
"type": {
"type": "string",
"enum": [
"ContextSubstitution"
],
"default": "ContextSubstitution"
},
"key": {
"type": "string",
"description": "Name of the new field"
},
"fail_dst_not_found": {
"type": "boolean",
"default": true,
"description": "Fail if destination field does not exist. This can be the case if attempting to insert into a nested destination which was not yet created."
},
"includes": {
"type": "array",
"items": {
"type": "string"
},
"description": "List of context fields to add. If non-specified then all are added."
},
"excludes": {
"type": "array",
"items": {
"type": "string"
},
"description": "List of context fields to exclude. If non-specified then none are excluded."
}
},
"title": "ContextSubstitution",
"required": [
"type",
"key"
]
}
{
"type": "object",
"additionalProperties": false,
"description": "The nested substitution helps build a Map object containing the result of other substitution.",
"properties": {
"type": {
"type": "string",
"enum": [
"NestedSubstitution"
],
"default": "NestedSubstitution"
},
"key": {
"type": "string",
"description": "Name of the new field"
},
"fail_dst_not_found": {
"type": "boolean",
"default": true,
"description": "Fail if destination field does not exist. This can be the case if attempting to insert into a nested destination which was not yet created."
},
"substitutions": {
"type": "array",
"description": "List of substitutions that will build up the final Map object.",
"anyOf": [
{
"$ref": "#/definitions/FieldSubstitutionConfig"
},
{
"$ref": "#/definitions/RegexSubstitutionConfig"
},
{
"$ref": "#/definitions/MetadataSubstitutionConfig"
},
{
"$ref": "#/definitions/NestedSubstitutionConfig"
},
{
"$ref": "#/definitions/FormattedSubstitutionConfig"
},
{
"$ref": "#/definitions/StaticSubstitutionConfig"
},
{
"$ref": "#/definitions/ContextSubstitutionConfig"
}
]
}
},
"title": "NestedSubstitution",
"required": [
"type",
"key"
]
}
Creates a new string using variable replacement.
Name of the new field
Fail if destination field does not exist. This can be the case if attempting to insert into a nested destination which was not yet created.
String with variable indices to substitue. See https://docs.oracle.com/javase/8/docs/api/java/text/MessageFormat.html
List of variables used in string substitution. Index of variable relates to index in 'format' string.
Source fields to pull value from. If multiple fields are provided the first non-null valued one is used. Note that if no fields are found the value will be set to null.
Fail if source field is not found.
Removes the source field when performing the substitution.
{
"type": "object",
"additionalProperties": false,
"properties": {
"type": {
"type": "string",
"enum": [
"FieldVariable"
],
"default": "FieldVariable"
},
"src_fields": {
"type": "array",
"items": {
"type": "string"
},
"description": "Source fields to pull value from. If multiple fields are provided the first non-null valued one is used. Note that if no fields are found the value will be set to null."
},
"fail_src_not_found": {
"type": "boolean",
"default": true,
"description": "Fail if source field is not found."
},
"remove_src_field": {
"type": "boolean",
"default": false,
"description": "Removes the source field when performing the substitution."
}
},
"title": "FieldVariable",
"required": [
"type",
"src_fields"
]
}
Value to substitute.
{
"type": "object",
"additionalProperties": false,
"properties": {
"type": {
"type": "string",
"enum": [
"StaticVariable"
],
"default": "StaticVariable"
},
"value": {
"type": "string",
"description": "Value to substitute."
}
},
"title": "StaticVariable",
"required": [
"type",
"value"
]
}
{
"type": "object",
"additionalProperties": false,
"description": "Creates a new string using variable replacement.",
"properties": {
"type": {
"type": "string",
"enum": [
"FormattedSubstitution"
],
"default": "FormattedSubstitution"
},
"key": {
"type": "string",
"description": "Name of the new field"
},
"fail_dst_not_found": {
"type": "boolean",
"default": true,
"description": "Fail if destination field does not exist. This can be the case if attempting to insert into a nested destination which was not yet created."
},
"format": {
"type": "string",
"description": "String with variable indices to substitue. See https://docs.oracle.com/javase/8/docs/api/java/text/MessageFormat.html"
},
"variables": {
"type": "array",
"description": "List of variables used in string substitution. Index of variable relates to index in 'format' string.",
"anyOf": [
{
"$ref": "#/definitions/FieldVariable"
},
{
"$ref": "#/definitions/StaticVariable"
}
]
}
},
"title": "FormattedSubstitution",
"required": [
"type",
"key",
"format",
"variables"
]
}
Substitutes event field value for a static value.
Name of the new field
Fail if destination field does not exist. This can be the case if attempting to insert into a nested destination which was not yet created.
Value of the new field.
{
"type": "object",
"additionalProperties": false,
"description": "Substitutes event field value for a static value.",
"properties": {
"type": {
"type": "string",
"enum": [
"StaticSubstitution"
],
"default": "StaticSubstitution"
},
"key": {
"type": "string",
"description": "Name of the new field"
},
"fail_dst_not_found": {
"type": "boolean",
"default": true,
"description": "Fail if destination field does not exist. This can be the case if attempting to insert into a nested destination which was not yet created."
},
"value": {
"type": "string",
"description": "Value of the new field."
}
},
"title": "StaticSubstitution",
"required": [
"type",
"key",
"value"
]
}
Substitutes event field value for list of lambda function context proprties.
Name of the new field
Fail if destination field does not exist. This can be the case if attempting to insert into a nested destination which was not yet created.
List of context fields to add. If non-specified then all are added.
List of context fields to exclude. If non-specified then none are excluded.
{
"type": "object",
"additionalProperties": false,
"description": "Substitutes event field value for list of lambda function context proprties.",
"properties": {
"type": {
"type": "string",
"enum": [
"ContextSubstitution"
],
"default": "ContextSubstitution"
},
"key": {
"type": "string",
"description": "Name of the new field"
},
"fail_dst_not_found": {
"type": "boolean",
"default": true,
"description": "Fail if destination field does not exist. This can be the case if attempting to insert into a nested destination which was not yet created."
},
"includes": {
"type": "array",
"items": {
"type": "string"
},
"description": "List of context fields to add. If non-specified then all are added."
},
"excludes": {
"type": "array",
"items": {
"type": "string"
},
"description": "List of context fields to exclude. If non-specified then none are excluded."
}
},
"title": "ContextSubstitution",
"required": [
"type",
"key"
]
}
{
"type": "object",
"additionalProperties": false,
"description": "This operation allows substituting event fields with different sources such as other fields, static values, or metadata.",
"properties": {
"type": {
"type": "string",
"enum": [
"SubstitutionOperation"
],
"default": "SubstitutionOperation"
},
"substitutions": {
"type": "array",
"description": "List of substitutions to perform.",
"anyOf": [
{
"$ref": "#/definitions/FieldSubstitutionConfig"
},
{
"$ref": "#/definitions/RegexSubstitutionConfig"
},
{
"$ref": "#/definitions/MetadataSubstitutionConfig"
},
{
"$ref": "#/definitions/NestedSubstitutionConfig"
},
{
"$ref": "#/definitions/FormattedSubstitutionConfig"
},
{
"$ref": "#/definitions/StaticSubstitutionConfig"
},
{
"$ref": "#/definitions/ContextSubstitutionConfig"
}
]
}
},
"title": "SubstitutionOperation",
"required": [
"type",
"substitutions"
]
}
Sets the partition information for the Event using fields from the deserialized object. When using JSON use JsonPath format to specify fields. See https://github.com/jayway/JsonPath
Configuration to specify object fields that are treated as partitions
Value to use as the key for the partition.
Fields to use for the value of the partition.
Interpreter to use on the partition value.
Java date format to use when using a time based interpreter.
Basic string formatting.
Rounds time down to value prior to formatting. For example 300 would round down to nearest 5 minutes. If set to 0 then rounding is disabled.
{
"type": "object",
"additionalProperties": false,
"properties": {
"name": {
"type": "string",
"description": "Value to use as the key for the partition."
},
"sources": {
"type": "array",
"items": {
"type": "string"
},
"description": "Fields to use for the value of the partition."
},
"interpreter": {
"type": "string",
"default": "STRING",
"enum": [
"STRING",
"MILLISECONDS",
"SECONDS",
"STATIC"
],
"description": "Interpreter to use on the partition value."
},
"format": {
"type": "string",
"description": "Java date format to use when using a time based interpreter."
},
"string_format": {
"type": "string",
"default": "NONE",
"enum": [
"TOLOWER",
"TOUPPER",
"NONE"
],
"description": "Basic string formatting."
},
"seconds_to_round": {
"type": "integer",
"minimum": 0,
"maximum": 86400,
"default": 0,
"description": "Rounds time down to value prior to formatting. For example 300 would round down to nearest 5 minutes. If set to 0 then rounding is disabled."
}
},
"required": [
"name",
"interpreter"
]
}
{
"type": "object",
"additionalProperties": false,
"description": "Sets the partition information for the Event using fields from the deserialized object. When using JSON use JsonPath format to specify fields. See https://github.com/jayway/JsonPath",
"properties": {
"type": {
"type": "string",
"enum": [
"PartitionOperation"
],
"default": "PartitionOperation"
},
"partition_specs": {
"type": "array",
"items": {
"$ref": "#/definitions/PartitionSpec"
},
"description": "Configuration to specify object fields that are treated as partitions"
}
},
"title": "PartitionOperation",
"required": [
"type"
]
}
Provided a path to a JSON array it will produce new events with payloads coorepsonding to elements of the array. For example [{"foo": 1}, {"bar": 2}] will be turned into two seperate events with payloads of {"foo": 1} and {"bar": 2}.
If an array is found and split, this can specify additional fields to keep in the new JSON object to preserve common contexts, such as timestamps, accountId, etc.
{
"type": "object",
"additionalProperties": false,
"description": "Provided a path to a JSON array it will produce new events with payloads coorepsonding to elements of the array. For example [{\"foo\": 1}, {\"bar\": 2}] will be turned into two seperate events with payloads of {\"foo\": 1} and {\"bar\": 2}.",
"properties": {
"type": {
"type": "string",
"enum": [
"JsonArraySplitOperation"
],
"default": "JsonArraySplitOperation"
},
"path": {
"type": "string",
"description": "Path to a JSON node which is an array. See https://github.com/jayway/JsonPath"
},
"fields_to_keep": {
"type": "array",
"items": {
"type": "string"
},
"description": "If an array is found and split, this can specify additional fields to keep in the new JSON object to preserve common contexts, such as timestamps, accountId, etc."
}
},
"title": "JsonArraySplitOperation",
"required": [
"type",
"path"
]
}
Performs a URL decode on the specified fields.
fields
Number of times to perform URL decode on field. This is helpful if a field is encoded multiple times.
{
"type": "object",
"additionalProperties": false,
"description": "Performs a URL decode on the specified fields.",
"properties": {
"type": {
"type": "string",
"enum": [
"UrlDecodeOperation"
],
"default": "UrlDecodeOperation"
},
"fields": {
"type": "array",
"items": {
"type": "string"
},
"description": "fields"
},
"times": {
"type": "integer",
"minimum": 1,
"maximum": 65535,
"default": 1,
"description": "Number of times to perform URL decode on field. This is helpful if a field is encoded multiple times."
}
},
"title": "UrlDecodeOperation",
"required": [
"type",
"fields"
]
}
Deletes the inputted field in the deserialized object. When using JSON use JsonPath format to specify field. See https://github.com/jayway/JsonPath
Name of field key to be deleted.
{
"type": "object",
"additionalProperties": false,
"description": "Deletes the inputted field in the deserialized object. When using JSON use JsonPath format to specify field. See https://github.com/jayway/JsonPath",
"properties": {
"type": {
"type": "string",
"enum": [
"DeleteFieldOperation"
],
"default": "DeleteFieldOperation"
},
"key_field": {
"type": "string",
"description": "Name of field key to be deleted."
}
},
"title": "DeleteFieldOperation",
"required": [
"type",
"key_field"
]
}
Provided a JSON object it will remove any keys which have array values. This is helpful for use with ElasticSearch which indexes arrays in a particular manner not conducive to exploration. See: https://www.elastic.co/guide/en/elasticsearch/guide/current/complex-core-fields.html
{
"type": "object",
"additionalProperties": false,
"description": "Provided a JSON object it will remove any keys which have array values. This is helpful for use with ElasticSearch which indexes arrays in a particular manner not conducive to exploration. See: https://www.elastic.co/guide/en/elasticsearch/guide/current/complex-core-fields.html",
"properties": {
"type": {
"type": "string",
"enum": [
"JsonDropArraysOperation"
],
"default": "JsonDropArraysOperation"
}
},
"title": "JsonDropArraysOperation",
"required": [
"type"
]
}
Sets the Event timestamp using the provided field in the deserialized object. When using JSON use JsonPath format to specify field. See https://github.com/jayway/JsonPath
Name of field to use as time field
How to interpret time field
{
"type": "object",
"additionalProperties": false,
"description": "Sets the Event timestamp using the provided field in the deserialized object. When using JSON use JsonPath format to specify field. See https://github.com/jayway/JsonPath",
"properties": {
"type": {
"type": "string",
"enum": [
"TimeOperation"
],
"default": "TimeOperation"
},
"time_field": {
"type": "string",
"description": "Name of field to use as time field"
},
"time_field_type": {
"type": "string",
"enum": [
"SECONDS",
"MILLISECONDS",
"ISO8601"
],
"description": "How to interpret time field"
}
},
"title": "TimeOperation",
"required": [
"type",
"time_field",
"time_field_type"
]
}
This operation is used to remove certain events from the stream before continuing on to the destination. Each event is assessed by applying a JsonPath to its payload and matching the value against a regex Pattern. If exclude is true, events that match this criteria will be filtered out. If exclude is false, any events not matching this criteria will be filtered out. For example, say these two events are in the stream: {\"data\": \"one\", \"type\": \"bar\"} and {\"data\": \"one\", \"type\": \"baz\"}. With config values: regex = \"(bar)\", path = \"$.type\", and exclude = true, after filtering, the stream will only hold {\"data\": \"one\", \"type\": \"baz\"}. If instead, exclude = false, the stream would only hold {\"data\": \"one\", \"type\": \"bar\"} after filtering.
Regex to be matched against JSON objects. See https://docs.oracle.com/javase/8/docs/api/java/util/regex/Pattern.html
If true, matches will be filtered out. If false, non-matches will be filtered out.
{
"type": "object",
"additionalProperties": false,
"description": "This operation is used to remove certain events from the stream before continuing on to the destination. Each event is assessed by applying a JsonPath to its payload and matching the value against a regex Pattern. If exclude is true, events that match this criteria will be filtered out. If exclude is false, any events not matching this criteria will be filtered out. For example, say these two events are in the stream: {\\\"data\\\": \\\"one\\\", \\\"type\\\": \\\"bar\\\"} and {\\\"data\\\": \\\"one\\\", \\\"type\\\": \\\"baz\\\"}. With config values: regex = \\\"(bar)\\\", path = \\\"$.type\\\", and exclude = true, after filtering, the stream will only hold {\\\"data\\\": \\\"one\\\", \\\"type\\\": \\\"baz\\\"}. If instead, exclude = false, the stream would only hold {\\\"data\\\": \\\"one\\\", \\\"type\\\": \\\"bar\\\"} after filtering.",
"properties": {
"type": {
"type": "string",
"enum": [
"RegexFilterOperation"
],
"default": "RegexFilterOperation"
},
"regex": {
"type": "string",
"description": "Regex to be matched against JSON objects. See https://docs.oracle.com/javase/8/docs/api/java/util/regex/Pattern.html"
},
"path": {
"type": "string",
"description": "JsonPath to the element to be matched against. See https://github.com/jayway/JsonPath"
},
"exclude": {
"type": "boolean",
"default": true,
"description": "If true, matches will be filtered out. If false, non-matches will be filtered out."
}
},
"title": "RegexFilterOperation",
"required": [
"type",
"regex",
"path"
]
}
Looks up geo location provided an IP address and adds a map field to the payload that optionally contains country_name, country_iso_code, subdivision_name, subdivision_iso_code, city_name, postal_code, and location. For example
input = {"ip": "8.8.8.8"}
output = {"ip": "8.8.8.8", "geo_ip": {"location": {"lat": 37.751, "lon": -97.822}}}
Note 1: This operation requires the MaxMind GeoLite2 City Database. It is available at https://dev.maxmind.com/geoip/geoip2/geolite2/. After you download the database, extract the tarball and uploaded the .mmdb file to an S3 bucket accessible by your lambda function.
Note 2: If your ip address field contains a comma separated list of ip addresses (ie forwarded web requests) the first address in the list is used. This is typically the source address.
Field containing an IP address string in the payload
Field name to save the geo data to
S3 path to the GeoLite2 City database file. Prefix with s3://
List of geo properties to include
If ip lookup fails also fail the operation. Doing so will filter out the event.
{
"type": "object",
"additionalProperties": false,
"description": "Looks up geo location provided an IP address and adds a map field to the payload that optionally contains country_name, country_iso_code, subdivision_name, subdivision_iso_code, city_name, postal_code, and location. For example \n\n input = {\"ip\": \"8.8.8.8\"} \n\n output = {\"ip\": \"8.8.8.8\", \"geo_ip\": {\"location\": {\"lat\": 37.751, \"lon\": -97.822}}}\n\n\n\nNote 1: This operation requires the MaxMind GeoLite2 City Database. It is available at https://dev.maxmind.com/geoip/geoip2/geolite2/. After you download the database, extract the tarball and uploaded the .mmdb file to an S3 bucket accessible by your lambda function.\n\nNote 2: If your ip address field contains a comma separated list of ip addresses (ie forwarded web requests) the first address in the list is used. This is typically the source address.",
"properties": {
"type": {
"type": "string",
"enum": [
"GeoIpOperation"
],
"default": "GeoIpOperation"
},
"src_field_name": {
"type": "string",
"description": "Field containing an IP address string in the payload"
},
"dst_field_name": {
"type": "string",
"default": "geo_ip",
"description": "Field name to save the geo data to"
},
"geo_lite_db": {
"type": "string",
"description": "S3 path to the GeoLite2 City database file. Prefix with s3://"
},
"geo_properties": {
"type": "array",
"items": {
"type": "string",
"enum": [
"COUNTRY_NAME",
"COUNTRY_ISO_CODE",
"SUBDIVISION_NAME",
"SUBDIVISION_ISO_CODE",
"CITY_NAME",
"POSTAL_CODE",
"LOCATION"
]
},
"description": "List of geo properties to include"
},
"fail_on_not_found": {
"type": "boolean",
"default": false,
"description": "If ip lookup fails also fail the operation. Doing so will filter out the event."
}
},
"title": "GeoIpOperation",
"required": [
"type",
"src_field_name",
"dst_field_name",
"geo_lite_db"
]
}
Provided a deeply nested JSON Object, it will flatten out the object into keys with a specific separator (dot by default). For example, if the input is {"foo": {"bar": {"baz": 1}}} the operation will produce {"foo.bar.baz": 1} as the new payload.
Separator to be used between nested key names (typically a dot(.))
{
"type": "object",
"additionalProperties": false,
"description": "Provided a deeply nested JSON Object, it will flatten out the object into keys with a specific separator (dot by default). For example, if the input is {\"foo\": {\"bar\": {\"baz\": 1}}} the operation will produce {\"foo.bar.baz\": 1} as the new payload.",
"properties": {
"type": {
"type": "string",
"enum": [
"FlattenOperation"
],
"default": "FlattenOperation"
},
"separator": {
"type": "string",
"default": ".",
"description": "Separator to be used between nested key names (typically a dot(.))"
}
},
"title": "FlattenOperation",
"required": [
"type"
]
}
If pass is set to false then events are filtered out.
If 'false', event is filtered out.
{
"type": "object",
"additionalProperties": false,
"description": "If pass is set to false then events are filtered out.",
"properties": {
"type": {
"type": "string",
"enum": [
"BasicFilterOperation"
],
"default": "BasicFilterOperation"
},
"pass": {
"type": "boolean",
"default": false,
"description": "If 'false', event is filtered out."
}
},
"title": "BasicFilterOperation",
"required": [
"type"
]
}
Uses a regex to find matching parts of a key name and replaces with value or drops the key/value if a match is found. This is typically used to sanitize key names.
If a match is found then replace match with this value. Defaults to emptry string.
If a match is found then drop the key/value.
{
"type": "object",
"additionalProperties": false,
"description": "Uses a regex to find matching parts of a key name and replaces with value or drops the key/value if a match is found. This is typically used to sanitize key names.",
"properties": {
"type": {
"type": "string",
"enum": [
"KeyNameReplacementOperation"
],
"default": "KeyNameReplacementOperation"
},
"regex": {
"type": "string",
"description": "Java regex. See https://docs.oracle.com/javase/8/docs/api/java/util/regex/Pattern.html"
},
"replacement": {
"type": "string",
"default": "",
"description": "If a match is found then replace match with this value. Defaults to emptry string."
},
"drop": {
"type": "boolean",
"default": false,
"description": "If a match is found then drop the key/value."
}
},
"title": "KeyNameReplacementOperation",
"required": [
"type",
"regex"
]
}
The conditional operation allows for 'if else' style branches in the flow of Bender operations. Conditions are evaluated in order of appearance and the first matching condition will receive the event.
List of conditions and their operations.
List of operations to perform.
circular reference
{
"description": "_circular reference_"
}
Provided a JSON object it will recursively lower case all fields.
{
"type": "object",
"additionalProperties": false,
"description": "Provided a JSON object it will recursively lower case all fields.",
"properties": {
"type": {
"type": "string",
"enum": [
"LowerCaseKeyOperation"
],
"default": "LowerCaseKeyOperation"
}
},
"title": "LowerCaseKeyOperation",
"required": [
"type"
]
}
Provided a JSON object and a path within the object it will promote the path's element to the root position. For example if the input is {"foo": {"bar": {"baz": 1}}} and specified path $.foo.bar the operation will produce {"baz": 1} as the new payload.
Path to a JSON node which is promoted to root node. See https://github.com/jayway/JsonPath
{
"type": "object",
"additionalProperties": false,
"description": "Provided a JSON object and a path within the object it will promote the path's element to the root position. For example if the input is {\"foo\": {\"bar\": {\"baz\": 1}}} and specified path $.foo.bar the operation will produce {\"baz\": 1} as the new payload.",
"properties": {
"type": {
"type": "string",
"enum": [
"JsonRootNodeOperation"
],
"default": "JsonRootNodeOperation"
},
"root_path": {
"type": "string",
"description": "Path to a JSON node which is promoted to root node. See https://github.com/jayway/JsonPath"
}
},
"title": "JsonRootNodeOperation",
"required": [
"type",
"root_path"
]
}
This operation allows substituting event fields with different sources such as other fields, static values, or metadata.
List of substitutions to perform.
Substitutes event field value for another event field value. Note the source field and destination field can be the same.
Name of the new field
Fail if destination field does not exist. This can be the case if attempting to insert into a nested destination which was not yet created.
Source fields to pull value from. If multiple fields are provided the first non-null valued one is used. Note that if no fields are found the value will be set to null.
Removes the source field when performing the substitution. Effectively making this a move operation.
Fail if source field is not found.
{
"type": "object",
"additionalProperties": false,
"description": "Substitutes event field value for another event field value. Note the source field and destination field can be the same.",
"properties": {
"type": {
"type": "string",
"enum": [
"FieldSubstitution"
],
"default": "FieldSubstitution"
},
"key": {
"type": "string",
"description": "Name of the new field"
},
"fail_dst_not_found": {
"type": "boolean",
"default": true,
"description": "Fail if destination field does not exist. This can be the case if attempting to insert into a nested destination which was not yet created."
},
"src_fields": {
"type": "array",
"items": {
"type": "string"
},
"description": "Source fields to pull value from. If multiple fields are provided the first non-null valued one is used. Note that if no fields are found the value will be set to null."
},
"remove_src_field": {
"type": "boolean",
"default": false,
"description": "Removes the source field when performing the substitution. Effectively making this a move operation."
},
"fail_src_not_found": {
"type": "boolean",
"default": true,
"description": "Fail if source field is not found."
}
},
"title": "FieldSubstitution",
"required": [
"type",
"key",
"src_fields"
]
}
Substitutes event field value for another event field value. Note the source field and destination field can be the same.
Fail if destination field does not exist. This can be the case if attempting to insert into a nested destination which was not yet created.
Regex pattern with match groups. See https://docs.oracle.com/javase/8/docs/api/java/util/regex/Pattern.html
Source fields to pull value from and apply regex to. If multiple fields are provided the first non-null valued one is used.
List of fields to create from matching regex groups.
Regex group name identifying the field.
Data type of match group field. If type coercion does not succeed then field is ignored.
Name or path of the new field.
{
"type": "object",
"additionalProperties": false,
"properties": {
"regex_group_name": {
"type": "string",
"description": "Regex group name identifying the field."
},
"type": {
"type": "string",
"enum": [
"STRING",
"NUMBER",
"BOOLEAN"
],
"description": "Data type of match group field. If type coercion does not succeed then field is ignored."
},
"key": {
"type": "string",
"description": "Name or path of the new field."
}
},
"required": [
"regex_group_name",
"type",
"key"
]
}
Removes the source field after applying this substitution.
Fail if source fields do not match regex or are not found.
{
"type": "object",
"additionalProperties": false,
"description": "Substitutes event field value for another event field value. Note the source field and destination field can be the same.",
"properties": {
"type": {
"type": "string",
"enum": [
"RegexSubstitution"
],
"default": "RegexSubstitution"
},
"fail_dst_not_found": {
"type": "boolean",
"default": true,
"description": "Fail if destination field does not exist. This can be the case if attempting to insert into a nested destination which was not yet created."
},
"pattern": {
"type": "string",
"description": "Regex pattern with match groups. See https://docs.oracle.com/javase/8/docs/api/java/util/regex/Pattern.html"
},
"src_fields": {
"type": "array",
"items": {
"type": "string"
},
"description": "Source fields to pull value from and apply regex to. If multiple fields are provided the first non-null valued one is used."
},
"fields": {
"type": "array",
"items": {
"$ref": "#/definitions/RegexSubField"
},
"description": "List of fields to create from matching regex groups."
},
"remove_src_field": {
"type": "boolean",
"default": false,
"description": "Removes the source field after applying this substitution."
},
"fail_src_not_found": {
"type": "boolean",
"default": true,
"description": "Fail if source fields do not match regex or are not found."
}
},
"title": "RegexSubstitution",
"required": [
"type",
"pattern",
"src_fields",
"fields"
]
}
Substitutes event field value for list of event metadata proprties.
Name of the new field
Fail if destination field does not exist. This can be the case if attempting to insert into a nested destination which was not yet created.
List of metadata fields to add. If non-specified then all are added.
List of metadata fields to exclude. If non-specified then none are excluded.
{
"type": "object",
"additionalProperties": false,
"description": "Substitutes event field value for list of event metadata proprties.",
"properties": {
"type": {
"type": "string",
"enum": [
"MetadataSubstitution"
],
"default": "MetadataSubstitution"
},
"key": {
"type": "string",
"description": "Name of the new field"
},
"fail_dst_not_found": {
"type": "boolean",
"default": true,
"description": "Fail if destination field does not exist. This can be the case if attempting to insert into a nested destination which was not yet created."
},
"includes": {
"type": "array",
"items": {
"type": "string"
},
"description": "List of metadata fields to add. If non-specified then all are added."
},
"excludes": {
"type": "array",
"items": {
"type": "string"
},
"description": "List of metadata fields to exclude. If non-specified then none are excluded."
}
},
"title": "MetadataSubstitution",
"required": [
"type",
"key"
]
}
The nested substitution helps build a Map object containing the result of other substitution.
Name of the new field
Fail if destination field does not exist. This can be the case if attempting to insert into a nested destination which was not yet created.
List of substitutions that will build up the final Map object.
Substitutes event field value for another event field value. Note the source field and destination field can be the same.
Name of the new field
Fail if destination field does not exist. This can be the case if attempting to insert into a nested destination which was not yet created.
Source fields to pull value from. If multiple fields are provided the first non-null valued one is used. Note that if no fields are found the value will be set to null.
Removes the source field when performing the substitution. Effectively making this a move operation.
Fail if source field is not found.
{
"type": "object",
"additionalProperties": false,
"description": "Substitutes event field value for another event field value. Note the source field and destination field can be the same.",
"properties": {
"type": {
"type": "string",
"enum": [
"FieldSubstitution"
],
"default": "FieldSubstitution"
},
"key": {
"type": "string",
"description": "Name of the new field"
},
"fail_dst_not_found": {
"type": "boolean",
"default": true,
"description": "Fail if destination field does not exist. This can be the case if attempting to insert into a nested destination which was not yet created."
},
"src_fields": {
"type": "array",
"items": {
"type": "string"
},
"description": "Source fields to pull value from. If multiple fields are provided the first non-null valued one is used. Note that if no fields are found the value will be set to null."
},
"remove_src_field": {
"type": "boolean",
"default": false,
"description": "Removes the source field when performing the substitution. Effectively making this a move operation."
},
"fail_src_not_found": {
"type": "boolean",
"default": true,
"description": "Fail if source field is not found."
}
},
"title": "FieldSubstitution",
"required": [
"type",
"key",
"src_fields"
]
}
Substitutes event field value for another event field value. Note the source field and destination field can be the same.
Fail if destination field does not exist. This can be the case if attempting to insert into a nested destination which was not yet created.
Regex pattern with match groups. See https://docs.oracle.com/javase/8/docs/api/java/util/regex/Pattern.html
Source fields to pull value from and apply regex to. If multiple fields are provided the first non-null valued one is used.
List of fields to create from matching regex groups.
Regex group name identifying the field.
Data type of match group field. If type coercion does not succeed then field is ignored.
Name or path of the new field.
{
"type": "object",
"additionalProperties": false,
"properties": {
"regex_group_name": {
"type": "string",
"description": "Regex group name identifying the field."
},
"type": {
"type": "string",
"enum": [
"STRING",
"NUMBER",
"BOOLEAN"
],
"description": "Data type of match group field. If type coercion does not succeed then field is ignored."
},
"key": {
"type": "string",
"description": "Name or path of the new field."
}
},
"required": [
"regex_group_name",
"type",
"key"
]
}
Removes the source field after applying this substitution.
Fail if source fields do not match regex or are not found.
{
"type": "object",
"additionalProperties": false,
"description": "Substitutes event field value for another event field value. Note the source field and destination field can be the same.",
"properties": {
"type": {
"type": "string",
"enum": [
"RegexSubstitution"
],
"default": "RegexSubstitution"
},
"fail_dst_not_found": {
"type": "boolean",
"default": true,
"description": "Fail if destination field does not exist. This can be the case if attempting to insert into a nested destination which was not yet created."
},
"pattern": {
"type": "string",
"description": "Regex pattern with match groups. See https://docs.oracle.com/javase/8/docs/api/java/util/regex/Pattern.html"
},
"src_fields": {
"type": "array",
"items": {
"type": "string"
},
"description": "Source fields to pull value from and apply regex to. If multiple fields are provided the first non-null valued one is used."
},
"fields": {
"type": "array",
"items": {
"$ref": "#/definitions/RegexSubField"
},
"description": "List of fields to create from matching regex groups."
},
"remove_src_field": {
"type": "boolean",
"default": false,
"description": "Removes the source field after applying this substitution."
},
"fail_src_not_found": {
"type": "boolean",
"default": true,
"description": "Fail if source fields do not match regex or are not found."
}
},
"title": "RegexSubstitution",
"required": [
"type",
"pattern",
"src_fields",
"fields"
]
}
Substitutes event field value for list of event metadata proprties.
Name of the new field
Fail if destination field does not exist. This can be the case if attempting to insert into a nested destination which was not yet created.
List of metadata fields to add. If non-specified then all are added.
List of metadata fields to exclude. If non-specified then none are excluded.
{
"type": "object",
"additionalProperties": false,
"description": "Substitutes event field value for list of event metadata proprties.",
"properties": {
"type": {
"type": "string",
"enum": [
"MetadataSubstitution"
],
"default": "MetadataSubstitution"
},
"key": {
"type": "string",
"description": "Name of the new field"
},
"fail_dst_not_found": {
"type": "boolean",
"default": true,
"description": "Fail if destination field does not exist. This can be the case if attempting to insert into a nested destination which was not yet created."
},
"includes": {
"type": "array",
"items": {
"type": "string"
},
"description": "List of metadata fields to add. If non-specified then all are added."
},
"excludes": {
"type": "array",
"items": {
"type": "string"
},
"description": "List of metadata fields to exclude. If non-specified then none are excluded."
}
},
"title": "MetadataSubstitution",
"required": [
"type",
"key"
]
}
circular reference
{
"description": "_circular reference_"
}
Creates a new string using variable replacement.
Name of the new field
Fail if destination field does not exist. This can be the case if attempting to insert into a nested destination which was not yet created.
String with variable indices to substitue. See https://docs.oracle.com/javase/8/docs/api/java/text/MessageFormat.html
List of variables used in string substitution. Index of variable relates to index in 'format' string.
Source fields to pull value from. If multiple fields are provided the first non-null valued one is used. Note that if no fields are found the value will be set to null.
Fail if source field is not found.
Removes the source field when performing the substitution.
{
"type": "object",
"additionalProperties": false,
"properties": {
"type": {
"type": "string",
"enum": [
"FieldVariable"
],
"default": "FieldVariable"
},
"src_fields": {
"type": "array",
"items": {
"type": "string"
},
"description": "Source fields to pull value from. If multiple fields are provided the first non-null valued one is used. Note that if no fields are found the value will be set to null."
},
"fail_src_not_found": {
"type": "boolean",
"default": true,
"description": "Fail if source field is not found."
},
"remove_src_field": {
"type": "boolean",
"default": false,
"description": "Removes the source field when performing the substitution."
}
},
"title": "FieldVariable",
"required": [
"type",
"src_fields"
]
}
Value to substitute.
{
"type": "object",
"additionalProperties": false,
"properties": {
"type": {
"type": "string",
"enum": [
"StaticVariable"
],
"default": "StaticVariable"
},
"value": {
"type": "string",
"description": "Value to substitute."
}
},
"title": "StaticVariable",
"required": [
"type",
"value"
]
}
{
"type": "object",
"additionalProperties": false,
"description": "Creates a new string using variable replacement.",
"properties": {
"type": {
"type": "string",
"enum": [
"FormattedSubstitution"
],
"default": "FormattedSubstitution"
},
"key": {
"type": "string",
"description": "Name of the new field"
},
"fail_dst_not_found": {
"type": "boolean",
"default": true,
"description": "Fail if destination field does not exist. This can be the case if attempting to insert into a nested destination which was not yet created."
},
"format": {
"type": "string",
"description": "String with variable indices to substitue. See https://docs.oracle.com/javase/8/docs/api/java/text/MessageFormat.html"
},
"variables": {
"type": "array",
"description": "List of variables used in string substitution. Index of variable relates to index in 'format' string.",
"anyOf": [
{
"$ref": "#/definitions/FieldVariable"
},
{
"$ref": "#/definitions/StaticVariable"
}
]
}
},
"title": "FormattedSubstitution",
"required": [
"type",
"key",
"format",
"variables"
]
}
Substitutes event field value for a static value.
Name of the new field
Fail if destination field does not exist. This can be the case if attempting to insert into a nested destination which was not yet created.
Value of the new field.
{
"type": "object",
"additionalProperties": false,
"description": "Substitutes event field value for a static value.",
"properties": {
"type": {
"type": "string",
"enum": [
"StaticSubstitution"
],
"default": "StaticSubstitution"
},
"key": {
"type": "string",
"description": "Name of the new field"
},
"fail_dst_not_found": {
"type": "boolean",
"default": true,
"description": "Fail if destination field does not exist. This can be the case if attempting to insert into a nested destination which was not yet created."
},
"value": {
"type": "string",
"description": "Value of the new field."
}
},
"title": "StaticSubstitution",
"required": [
"type",
"key",
"value"
]
}
Substitutes event field value for list of lambda function context proprties.
Name of the new field
Fail if destination field does not exist. This can be the case if attempting to insert into a nested destination which was not yet created.
List of context fields to add. If non-specified then all are added.
List of context fields to exclude. If non-specified then none are excluded.
{
"type": "object",
"additionalProperties": false,
"description": "Substitutes event field value for list of lambda function context proprties.",
"properties": {
"type": {
"type": "string",
"enum": [
"ContextSubstitution"
],
"default": "ContextSubstitution"
},
"key": {
"type": "string",
"description": "Name of the new field"
},
"fail_dst_not_found": {
"type": "boolean",
"default": true,
"description": "Fail if destination field does not exist. This can be the case if attempting to insert into a nested destination which was not yet created."
},
"includes": {
"type": "array",
"items": {
"type": "string"
},
"description": "List of context fields to add. If non-specified then all are added."
},
"excludes": {
"type": "array",
"items": {
"type": "string"
},
"description": "List of context fields to exclude. If non-specified then none are excluded."
}
},
"title": "ContextSubstitution",
"required": [
"type",
"key"
]
}
{
"type": "object",
"additionalProperties": false,
"description": "The nested substitution helps build a Map object containing the result of other substitution.",
"properties": {
"type": {
"type": "string",
"enum": [
"NestedSubstitution"
],
"default": "NestedSubstitution"
},
"key": {
"type": "string",
"description": "Name of the new field"
},
"fail_dst_not_found": {
"type": "boolean",
"default": true,
"description": "Fail if destination field does not exist. This can be the case if attempting to insert into a nested destination which was not yet created."
},
"substitutions": {
"type": "array",
"description": "List of substitutions that will build up the final Map object.",
"anyOf": [
{
"$ref": "#/definitions/FieldSubstitutionConfig"
},
{
"$ref": "#/definitions/RegexSubstitutionConfig"
},
{
"$ref": "#/definitions/MetadataSubstitutionConfig"
},
{
"$ref": "#/definitions/NestedSubstitutionConfig"
},
{
"$ref": "#/definitions/FormattedSubstitutionConfig"
},
{
"$ref": "#/definitions/StaticSubstitutionConfig"
},
{
"$ref": "#/definitions/ContextSubstitutionConfig"
}
]
}
},
"title": "NestedSubstitution",
"required": [
"type",
"key"
]
}
Creates a new string using variable replacement.
Name of the new field
Fail if destination field does not exist. This can be the case if attempting to insert into a nested destination which was not yet created.
String with variable indices to substitue. See https://docs.oracle.com/javase/8/docs/api/java/text/MessageFormat.html
List of variables used in string substitution. Index of variable relates to index in 'format' string.
Source fields to pull value from. If multiple fields are provided the first non-null valued one is used. Note that if no fields are found the value will be set to null.
Fail if source field is not found.
Removes the source field when performing the substitution.
{
"type": "object",
"additionalProperties": false,
"properties": {
"type": {
"type": "string",
"enum": [
"FieldVariable"
],
"default": "FieldVariable"
},
"src_fields": {
"type": "array",
"items": {
"type": "string"
},
"description": "Source fields to pull value from. If multiple fields are provided the first non-null valued one is used. Note that if no fields are found the value will be set to null."
},
"fail_src_not_found": {
"type": "boolean",
"default": true,
"description": "Fail if source field is not found."
},
"remove_src_field": {
"type": "boolean",
"default": false,
"description": "Removes the source field when performing the substitution."
}
},
"title": "FieldVariable",
"required": [
"type",
"src_fields"
]
}
Value to substitute.
{
"type": "object",
"additionalProperties": false,
"properties": {
"type": {
"type": "string",
"enum": [
"StaticVariable"
],
"default": "StaticVariable"
},
"value": {
"type": "string",
"description": "Value to substitute."
}
},
"title": "StaticVariable",
"required": [
"type",
"value"
]
}
{
"type": "object",
"additionalProperties": false,
"description": "Creates a new string using variable replacement.",
"properties": {
"type": {
"type": "string",
"enum": [
"FormattedSubstitution"
],
"default": "FormattedSubstitution"
},
"key": {
"type": "string",
"description": "Name of the new field"
},
"fail_dst_not_found": {
"type": "boolean",
"default": true,
"description": "Fail if destination field does not exist. This can be the case if attempting to insert into a nested destination which was not yet created."
},
"format": {
"type": "string",
"description": "String with variable indices to substitue. See https://docs.oracle.com/javase/8/docs/api/java/text/MessageFormat.html"
},
"variables": {
"type": "array",
"description": "List of variables used in string substitution. Index of variable relates to index in 'format' string.",
"anyOf": [
{
"$ref": "#/definitions/FieldVariable"
},
{
"$ref": "#/definitions/StaticVariable"
}
]
}
},
"title": "FormattedSubstitution",
"required": [
"type",
"key",
"format",
"variables"
]
}
Substitutes event field value for a static value.
Name of the new field
Fail if destination field does not exist. This can be the case if attempting to insert into a nested destination which was not yet created.
Value of the new field.
{
"type": "object",
"additionalProperties": false,
"description": "Substitutes event field value for a static value.",
"properties": {
"type": {
"type": "string",
"enum": [
"StaticSubstitution"
],
"default": "StaticSubstitution"
},
"key": {
"type": "string",
"description": "Name of the new field"
},
"fail_dst_not_found": {
"type": "boolean",
"default": true,
"description": "Fail if destination field does not exist. This can be the case if attempting to insert into a nested destination which was not yet created."
},
"value": {
"type": "string",
"description": "Value of the new field."
}
},
"title": "StaticSubstitution",
"required": [
"type",
"key",
"value"
]
}
Substitutes event field value for list of lambda function context proprties.
Name of the new field
Fail if destination field does not exist. This can be the case if attempting to insert into a nested destination which was not yet created.
List of context fields to add. If non-specified then all are added.
List of context fields to exclude. If non-specified then none are excluded.
{
"type": "object",
"additionalProperties": false,
"description": "Substitutes event field value for list of lambda function context proprties.",
"properties": {
"type": {
"type": "string",
"enum": [
"ContextSubstitution"
],
"default": "ContextSubstitution"
},
"key": {
"type": "string",
"description": "Name of the new field"
},
"fail_dst_not_found": {
"type": "boolean",
"default": true,
"description": "Fail if destination field does not exist. This can be the case if attempting to insert into a nested destination which was not yet created."
},
"includes": {
"type": "array",
"items": {
"type": "string"
},
"description": "List of context fields to add. If non-specified then all are added."
},
"excludes": {
"type": "array",
"items": {
"type": "string"
},
"description": "List of context fields to exclude. If non-specified then none are excluded."
}
},
"title": "ContextSubstitution",
"required": [
"type",
"key"
]
}
{
"type": "object",
"additionalProperties": false,
"description": "This operation allows substituting event fields with different sources such as other fields, static values, or metadata.",
"properties": {
"type": {
"type": "string",
"enum": [
"SubstitutionOperation"
],
"default": "SubstitutionOperation"
},
"substitutions": {
"type": "array",
"description": "List of substitutions to perform.",
"anyOf": [
{
"$ref": "#/definitions/FieldSubstitutionConfig"
},
{
"$ref": "#/definitions/RegexSubstitutionConfig"
},
{
"$ref": "#/definitions/MetadataSubstitutionConfig"
},
{
"$ref": "#/definitions/NestedSubstitutionConfig"
},
{
"$ref": "#/definitions/FormattedSubstitutionConfig"
},
{
"$ref": "#/definitions/StaticSubstitutionConfig"
},
{
"$ref": "#/definitions/ContextSubstitutionConfig"
}
]
}
},
"title": "SubstitutionOperation",
"required": [
"type",
"substitutions"
]
}
Sets the partition information for the Event using fields from the deserialized object. When using JSON use JsonPath format to specify fields. See https://github.com/jayway/JsonPath
Configuration to specify object fields that are treated as partitions
Value to use as the key for the partition.
Fields to use for the value of the partition.
Interpreter to use on the partition value.
Java date format to use when using a time based interpreter.
Basic string formatting.
Rounds time down to value prior to formatting. For example 300 would round down to nearest 5 minutes. If set to 0 then rounding is disabled.
{
"type": "object",
"additionalProperties": false,
"properties": {
"name": {
"type": "string",
"description": "Value to use as the key for the partition."
},
"sources": {
"type": "array",
"items": {
"type": "string"
},
"description": "Fields to use for the value of the partition."
},
"interpreter": {
"type": "string",
"default": "STRING",
"enum": [
"STRING",
"MILLISECONDS",
"SECONDS",
"STATIC"
],
"description": "Interpreter to use on the partition value."
},
"format": {
"type": "string",
"description": "Java date format to use when using a time based interpreter."
},
"string_format": {
"type": "string",
"default": "NONE",
"enum": [
"TOLOWER",
"TOUPPER",
"NONE"
],
"description": "Basic string formatting."
},
"seconds_to_round": {
"type": "integer",
"minimum": 0,
"maximum": 86400,
"default": 0,
"description": "Rounds time down to value prior to formatting. For example 300 would round down to nearest 5 minutes. If set to 0 then rounding is disabled."
}
},
"required": [
"name",
"interpreter"
]
}
{
"type": "object",
"additionalProperties": false,
"description": "Sets the partition information for the Event using fields from the deserialized object. When using JSON use JsonPath format to specify fields. See https://github.com/jayway/JsonPath",
"properties": {
"type": {
"type": "string",
"enum": [
"PartitionOperation"
],
"default": "PartitionOperation"
},
"partition_specs": {
"type": "array",
"items": {
"$ref": "#/definitions/PartitionSpec"
},
"description": "Configuration to specify object fields that are treated as partitions"
}
},
"title": "PartitionOperation",
"required": [
"type"
]
}
Provided a path to a JSON array it will produce new events with payloads coorepsonding to elements of the array. For example [{"foo": 1}, {"bar": 2}] will be turned into two seperate events with payloads of {"foo": 1} and {"bar": 2}.
If an array is found and split, this can specify additional fields to keep in the new JSON object to preserve common contexts, such as timestamps, accountId, etc.
{
"type": "object",
"additionalProperties": false,
"description": "Provided a path to a JSON array it will produce new events with payloads coorepsonding to elements of the array. For example [{\"foo\": 1}, {\"bar\": 2}] will be turned into two seperate events with payloads of {\"foo\": 1} and {\"bar\": 2}.",
"properties": {
"type": {
"type": "string",
"enum": [
"JsonArraySplitOperation"
],
"default": "JsonArraySplitOperation"
},
"path": {
"type": "string",
"description": "Path to a JSON node which is an array. See https://github.com/jayway/JsonPath"
},
"fields_to_keep": {
"type": "array",
"items": {
"type": "string"
},
"description": "If an array is found and split, this can specify additional fields to keep in the new JSON object to preserve common contexts, such as timestamps, accountId, etc."
}
},
"title": "JsonArraySplitOperation",
"required": [
"type",
"path"
]
}
Performs a URL decode on the specified fields.
fields
Number of times to perform URL decode on field. This is helpful if a field is encoded multiple times.
{
"type": "object",
"additionalProperties": false,
"description": "Performs a URL decode on the specified fields.",
"properties": {
"type": {
"type": "string",
"enum": [
"UrlDecodeOperation"
],
"default": "UrlDecodeOperation"
},
"fields": {
"type": "array",
"items": {
"type": "string"
},
"description": "fields"
},
"times": {
"type": "integer",
"minimum": 1,
"maximum": 65535,
"default": 1,
"description": "Number of times to perform URL decode on field. This is helpful if a field is encoded multiple times."
}
},
"title": "UrlDecodeOperation",
"required": [
"type",
"fields"
]
}
Deletes the inputted field in the deserialized object. When using JSON use JsonPath format to specify field. See https://github.com/jayway/JsonPath
Name of field key to be deleted.
{
"type": "object",
"additionalProperties": false,
"description": "Deletes the inputted field in the deserialized object. When using JSON use JsonPath format to specify field. See https://github.com/jayway/JsonPath",
"properties": {
"type": {
"type": "string",
"enum": [
"DeleteFieldOperation"
],
"default": "DeleteFieldOperation"
},
"key_field": {
"type": "string",
"description": "Name of field key to be deleted."
}
},
"title": "DeleteFieldOperation",
"required": [
"type",
"key_field"
]
}
Provided a JSON object it will remove any keys which have array values. This is helpful for use with ElasticSearch which indexes arrays in a particular manner not conducive to exploration. See: https://www.elastic.co/guide/en/elasticsearch/guide/current/complex-core-fields.html
{
"type": "object",
"additionalProperties": false,
"description": "Provided a JSON object it will remove any keys which have array values. This is helpful for use with ElasticSearch which indexes arrays in a particular manner not conducive to exploration. See: https://www.elastic.co/guide/en/elasticsearch/guide/current/complex-core-fields.html",
"properties": {
"type": {
"type": "string",
"enum": [
"JsonDropArraysOperation"
],
"default": "JsonDropArraysOperation"
}
},
"title": "JsonDropArraysOperation",
"required": [
"type"
]
}
Sets the Event timestamp using the provided field in the deserialized object. When using JSON use JsonPath format to specify field. See https://github.com/jayway/JsonPath
Name of field to use as time field
How to interpret time field
{
"type": "object",
"additionalProperties": false,
"description": "Sets the Event timestamp using the provided field in the deserialized object. When using JSON use JsonPath format to specify field. See https://github.com/jayway/JsonPath",
"properties": {
"type": {
"type": "string",
"enum": [
"TimeOperation"
],
"default": "TimeOperation"
},
"time_field": {
"type": "string",
"description": "Name of field to use as time field"
},
"time_field_type": {
"type": "string",
"enum": [
"SECONDS",
"MILLISECONDS",
"ISO8601"
],
"description": "How to interpret time field"
}
},
"title": "TimeOperation",
"required": [
"type",
"time_field",
"time_field_type"
]
}
This operation is used to remove certain events from the stream before continuing on to the destination. Each event is assessed by applying a JsonPath to its payload and matching the value against a regex Pattern. If exclude is true, events that match this criteria will be filtered out. If exclude is false, any events not matching this criteria will be filtered out. For example, say these two events are in the stream: {\"data\": \"one\", \"type\": \"bar\"} and {\"data\": \"one\", \"type\": \"baz\"}. With config values: regex = \"(bar)\", path = \"$.type\", and exclude = true, after filtering, the stream will only hold {\"data\": \"one\", \"type\": \"baz\"}. If instead, exclude = false, the stream would only hold {\"data\": \"one\", \"type\": \"bar\"} after filtering.
Regex to be matched against JSON objects. See https://docs.oracle.com/javase/8/docs/api/java/util/regex/Pattern.html
If true, matches will be filtered out. If false, non-matches will be filtered out.
{
"type": "object",
"additionalProperties": false,
"description": "This operation is used to remove certain events from the stream before continuing on to the destination. Each event is assessed by applying a JsonPath to its payload and matching the value against a regex Pattern. If exclude is true, events that match this criteria will be filtered out. If exclude is false, any events not matching this criteria will be filtered out. For example, say these two events are in the stream: {\\\"data\\\": \\\"one\\\", \\\"type\\\": \\\"bar\\\"} and {\\\"data\\\": \\\"one\\\", \\\"type\\\": \\\"baz\\\"}. With config values: regex = \\\"(bar)\\\", path = \\\"$.type\\\", and exclude = true, after filtering, the stream will only hold {\\\"data\\\": \\\"one\\\", \\\"type\\\": \\\"baz\\\"}. If instead, exclude = false, the stream would only hold {\\\"data\\\": \\\"one\\\", \\\"type\\\": \\\"bar\\\"} after filtering.",
"properties": {
"type": {
"type": "string",
"enum": [
"RegexFilterOperation"
],
"default": "RegexFilterOperation"
},
"regex": {
"type": "string",
"description": "Regex to be matched against JSON objects. See https://docs.oracle.com/javase/8/docs/api/java/util/regex/Pattern.html"
},
"path": {
"type": "string",
"description": "JsonPath to the element to be matched against. See https://github.com/jayway/JsonPath"
},
"exclude": {
"type": "boolean",
"default": true,
"description": "If true, matches will be filtered out. If false, non-matches will be filtered out."
}
},
"title": "RegexFilterOperation",
"required": [
"type",
"regex",
"path"
]
}
Looks up geo location provided an IP address and adds a map field to the payload that optionally contains country_name, country_iso_code, subdivision_name, subdivision_iso_code, city_name, postal_code, and location. For example
input = {"ip": "8.8.8.8"}
output = {"ip": "8.8.8.8", "geo_ip": {"location": {"lat": 37.751, "lon": -97.822}}}
Note 1: This operation requires the MaxMind GeoLite2 City Database. It is available at https://dev.maxmind.com/geoip/geoip2/geolite2/. After you download the database, extract the tarball and uploaded the .mmdb file to an S3 bucket accessible by your lambda function.
Note 2: If your ip address field contains a comma separated list of ip addresses (ie forwarded web requests) the first address in the list is used. This is typically the source address.
Field containing an IP address string in the payload
Field name to save the geo data to
S3 path to the GeoLite2 City database file. Prefix with s3://
List of geo properties to include
If ip lookup fails also fail the operation. Doing so will filter out the event.
{
"type": "object",
"additionalProperties": false,
"description": "Looks up geo location provided an IP address and adds a map field to the payload that optionally contains country_name, country_iso_code, subdivision_name, subdivision_iso_code, city_name, postal_code, and location. For example \n\n input = {\"ip\": \"8.8.8.8\"} \n\n output = {\"ip\": \"8.8.8.8\", \"geo_ip\": {\"location\": {\"lat\": 37.751, \"lon\": -97.822}}}\n\n\n\nNote 1: This operation requires the MaxMind GeoLite2 City Database. It is available at https://dev.maxmind.com/geoip/geoip2/geolite2/. After you download the database, extract the tarball and uploaded the .mmdb file to an S3 bucket accessible by your lambda function.\n\nNote 2: If your ip address field contains a comma separated list of ip addresses (ie forwarded web requests) the first address in the list is used. This is typically the source address.",
"properties": {
"type": {
"type": "string",
"enum": [
"GeoIpOperation"
],
"default": "GeoIpOperation"
},
"src_field_name": {
"type": "string",
"description": "Field containing an IP address string in the payload"
},
"dst_field_name": {
"type": "string",
"default": "geo_ip",
"description": "Field name to save the geo data to"
},
"geo_lite_db": {
"type": "string",
"description": "S3 path to the GeoLite2 City database file. Prefix with s3://"
},
"geo_properties": {
"type": "array",
"items": {
"type": "string",
"enum": [
"COUNTRY_NAME",
"COUNTRY_ISO_CODE",
"SUBDIVISION_NAME",
"SUBDIVISION_ISO_CODE",
"CITY_NAME",
"POSTAL_CODE",
"LOCATION"
]
},
"description": "List of geo properties to include"
},
"fail_on_not_found": {
"type": "boolean",
"default": false,
"description": "If ip lookup fails also fail the operation. Doing so will filter out the event."
}
},
"title": "GeoIpOperation",
"required": [
"type",
"src_field_name",
"dst_field_name",
"geo_lite_db"
]
}
Provided a deeply nested JSON Object, it will flatten out the object into keys with a specific separator (dot by default). For example, if the input is {"foo": {"bar": {"baz": 1}}} the operation will produce {"foo.bar.baz": 1} as the new payload.
Separator to be used between nested key names (typically a dot(.))
{
"type": "object",
"additionalProperties": false,
"description": "Provided a deeply nested JSON Object, it will flatten out the object into keys with a specific separator (dot by default). For example, if the input is {\"foo\": {\"bar\": {\"baz\": 1}}} the operation will produce {\"foo.bar.baz\": 1} as the new payload.",
"properties": {
"type": {
"type": "string",
"enum": [
"FlattenOperation"
],
"default": "FlattenOperation"
},
"separator": {
"type": "string",
"default": ".",
"description": "Separator to be used between nested key names (typically a dot(.))"
}
},
"title": "FlattenOperation",
"required": [
"type"
]
}
If pass is set to false then events are filtered out.
If 'false', event is filtered out.
{
"type": "object",
"additionalProperties": false,
"description": "If pass is set to false then events are filtered out.",
"properties": {
"type": {
"type": "string",
"enum": [
"BasicFilterOperation"
],
"default": "BasicFilterOperation"
},
"pass": {
"type": "boolean",
"default": false,
"description": "If 'false', event is filtered out."
}
},
"title": "BasicFilterOperation",
"required": [
"type"
]
}
Uses a regex to find matching parts of a key name and replaces with value or drops the key/value if a match is found. This is typically used to sanitize key names.
If a match is found then replace match with this value. Defaults to emptry string.
If a match is found then drop the key/value.
{
"type": "object",
"additionalProperties": false,
"description": "Uses a regex to find matching parts of a key name and replaces with value or drops the key/value if a match is found. This is typically used to sanitize key names.",
"properties": {
"type": {
"type": "string",
"enum": [
"KeyNameReplacementOperation"
],
"default": "KeyNameReplacementOperation"
},
"regex": {
"type": "string",
"description": "Java regex. See https://docs.oracle.com/javase/8/docs/api/java/util/regex/Pattern.html"
},
"replacement": {
"type": "string",
"default": "",
"description": "If a match is found then replace match with this value. Defaults to emptry string."
},
"drop": {
"type": "boolean",
"default": false,
"description": "If a match is found then drop the key/value."
}
},
"title": "KeyNameReplacementOperation",
"required": [
"type",
"regex"
]
}
circular reference
{
"description": "_circular reference_"
}
Transforms JSON input into a GELF message format (See http://docs.graylog.org/en/2.4/pages/gelf.html details). Note that when selecting source fields use the JsonPath notation https://github.com/json-path/JsonPath.
Field containing host string
Field containing short_message string
Field containing full_message string
Field containing epoch timestamp in seconds with optional ms as decimal. Source field must be in numeric form (not a string) or Graylog will reject the input. Alternatively, use the TimeOperation to detect your timestamp field and properly parse it, and the GelfOperation will automatically use that to set the timestamp field properly. Use this setting only to override that behavior.
Field containing syslog level number
Field containing syslog facility string
Field containing the line in a file that caused the error (decimal); optional, deprecated. Send as additional field instead.
Field containing the file (with path if you want) that caused the error (string); optional, deprecated. Send as additional field instead.
{
"type": "object",
"additionalProperties": false,
"description": "Transforms JSON input into a GELF message format (See http://docs.graylog.org/en/2.4/pages/gelf.html details). Note that when selecting source fields use the JsonPath notation https://github.com/json-path/JsonPath.",
"properties": {
"type": {
"type": "string",
"enum": [
"GelfOperation"
],
"default": "GelfOperation"
},
"src_host_field": {
"type": "array",
"items": {
"type": "string"
},
"description": "Field containing host string"
},
"src_short_message_field": {
"type": "array",
"items": {
"type": "string"
},
"description": "Field containing short_message string"
},
"src_full_message_field": {
"type": "array",
"items": {
"type": "string"
},
"description": "Field containing full_message string"
},
"src_timestamp_field": {
"type": "array",
"items": {
"type": "string"
},
"description": "Field containing epoch timestamp in seconds with optional ms as decimal. Source field must be in numeric form (not a string) or Graylog will reject the input. Alternatively, use the TimeOperation to detect your timestamp field and properly parse it, and the GelfOperation will automatically use that to set the timestamp field properly. Use this setting only to override that behavior."
},
"src_level_field": {
"type": "array",
"items": {
"type": "string"
},
"description": "Field containing syslog level number"
},
"src_facility_field": {
"type": "array",
"items": {
"type": "string"
},
"description": "Field containing syslog facility string"
},
"src_line_number_field": {
"type": "array",
"items": {
"type": "string"
},
"description": "Field containing the line in a file that caused the error (decimal); optional, deprecated. Send as additional field instead."
},
"src_file_field": {
"type": "array",
"items": {
"type": "string"
},
"description": "Field containing the file (with path if you want) that caused the error (string); optional, deprecated. Send as additional field instead."
}
},
"title": "GelfOperation",
"required": [
"type",
"src_host_field",
"src_short_message_field"
]
}
Provided a JSON object it will recursively append the primitive type of the value to the key name. For example {"foo": "one", "bar": 2} will become {"foo__str": "one", "bar__long": 2}. The mapping is string:str, boolean:boolean, array:arr, number:long or __float. It also repalces "." with "_" in key names. This operation is particularily useful for modifying JSON which will be written to ElasticSearch which does not allow conflicting value types for keys or "." in key names.
{
"type": "object",
"additionalProperties": false,
"description": "Provided a JSON object it will recursively append the primitive type of the value to the key name. For example {\"foo\": \"one\", \"bar\": 2} will become {\"foo__str\": \"one\", \"bar__long\": 2}. The mapping is string:__str, boolean:__boolean, array:__arr, number:__long or __float. It also repalces \".\" with \"_\" in key names. This operation is particularily useful for modifying JSON which will be written to ElasticSearch which does not allow conflicting value types for keys or \".\" in key names.",
"properties": {
"type": {
"type": "string",
"enum": [
"JsonKeyNameOperation"
],
"default": "JsonKeyNameOperation"
}
},
"title": "JsonKeyNameOperation",
"required": [
"type"
]
}
Filter operation to evaluate events against.
This operation is used to remove certain events from the stream before continuing on to the destination. Each event is assessed by applying a JsonPath to its payload and matching the value against a regex Pattern. If exclude is true, events that match this criteria will be filtered out. If exclude is false, any events not matching this criteria will be filtered out. For example, say these two events are in the stream: {\"data\": \"one\", \"type\": \"bar\"} and {\"data\": \"one\", \"type\": \"baz\"}. With config values: regex = \"(bar)\", path = \"$.type\", and exclude = true, after filtering, the stream will only hold {\"data\": \"one\", \"type\": \"baz\"}. If instead, exclude = false, the stream would only hold {\"data\": \"one\", \"type\": \"bar\"} after filtering.
Regex to be matched against JSON objects. See https://docs.oracle.com/javase/8/docs/api/java/util/regex/Pattern.html
If true, matches will be filtered out. If false, non-matches will be filtered out.
{
"type": "object",
"additionalProperties": false,
"description": "This operation is used to remove certain events from the stream before continuing on to the destination. Each event is assessed by applying a JsonPath to its payload and matching the value against a regex Pattern. If exclude is true, events that match this criteria will be filtered out. If exclude is false, any events not matching this criteria will be filtered out. For example, say these two events are in the stream: {\\\"data\\\": \\\"one\\\", \\\"type\\\": \\\"bar\\\"} and {\\\"data\\\": \\\"one\\\", \\\"type\\\": \\\"baz\\\"}. With config values: regex = \\\"(bar)\\\", path = \\\"$.type\\\", and exclude = true, after filtering, the stream will only hold {\\\"data\\\": \\\"one\\\", \\\"type\\\": \\\"baz\\\"}. If instead, exclude = false, the stream would only hold {\\\"data\\\": \\\"one\\\", \\\"type\\\": \\\"bar\\\"} after filtering.",
"properties": {
"type": {
"type": "string",
"enum": [
"RegexFilterOperation"
],
"default": "RegexFilterOperation"
},
"regex": {
"type": "string",
"description": "Regex to be matched against JSON objects. See https://docs.oracle.com/javase/8/docs/api/java/util/regex/Pattern.html"
},
"path": {
"type": "string",
"description": "JsonPath to the element to be matched against. See https://github.com/jayway/JsonPath"
},
"exclude": {
"type": "boolean",
"default": true,
"description": "If true, matches will be filtered out. If false, non-matches will be filtered out."
}
},
"title": "RegexFilterOperation",
"required": [
"type",
"regex",
"path"
]
}
If pass is set to false then events are filtered out.
If 'false', event is filtered out.
{
"type": "object",
"additionalProperties": false,
"description": "If pass is set to false then events are filtered out.",
"properties": {
"type": {
"type": "string",
"enum": [
"BasicFilterOperation"
],
"default": "BasicFilterOperation"
},
"pass": {
"type": "boolean",
"default": false,
"description": "If 'false', event is filtered out."
}
},
"title": "BasicFilterOperation",
"required": [
"type"
]
}
{
"type": "object",
"additionalProperties": false,
"properties": {
"operations": {
"type": "array",
"description": "List of operations to perform.",
"anyOf": [
{
"$ref": "#/definitions/ForkOperationConfig"
},
{
"$ref": "#/definitions/LowerCaseKeyOperationConfig"
},
{
"$ref": "#/definitions/JsonRootNodeOperationConfig"
},
{
"$ref": "#/definitions/SubstitutionOperationConfig"
},
{
"$ref": "#/definitions/PartitionOperationConfig"
},
{
"$ref": "#/definitions/ArraySplitOperationConfig"
},
{
"$ref": "#/definitions/URLDecodeOperationConfig"
},
{
"$ref": "#/definitions/DeleteFieldOperationConfig"
},
{
"$ref": "#/definitions/DropArraysOperationConfig"
},
{
"$ref": "#/definitions/TimeOperationConfig"
},
{
"$ref": "#/definitions/RegexFilterOperationConfig"
},
{
"$ref": "#/definitions/GeoIpOperationConfig"
},
{
"$ref": "#/definitions/FlattenOperationConfig"
},
{
"$ref": "#/definitions/BasicFilterOperationConfig"
},
{
"$ref": "#/definitions/KeyNameReplacementOperationConfig"
},
{
"$ref": "#/definitions/ConditionalOperationConfig"
},
{
"$ref": "#/definitions/GelfOperationConfig"
},
{
"$ref": "#/definitions/KeyNameOperationConfig"
}
]
},
"condition": {
"oneOf": [
{
"$ref": "#/definitions/RegexFilterOperationConfig"
},
{
"$ref": "#/definitions/BasicFilterOperationConfig"
}
],
"description": "Filter operation to evaluate events against."
}
},
"required": [
"condition"
]
}
When true events that don't match any conditions will be filtered out.
{
"type": "object",
"additionalProperties": false,
"description": "The conditional operation allows for 'if else' style branches in the flow of Bender operations. Conditions are evaluated in order of appearance and the first matching condition will receive the event.",
"properties": {
"type": {
"type": "string",
"enum": [
"ConditionalOperation"
],
"default": "ConditionalOperation"
},
"conditions": {
"type": "array",
"items": {
"$ref": "#/definitions/Condition"
},
"description": "List of conditions and their operations."
},
"filter_non_match": {
"type": "boolean",
"default": false,
"description": "When true events that don't match any conditions will be filtered out."
}
},
"title": "ConditionalOperation",
"required": [
"type"
]
}
Transforms JSON input into a GELF message format (See http://docs.graylog.org/en/2.4/pages/gelf.html details). Note that when selecting source fields use the JsonPath notation https://github.com/json-path/JsonPath.
Field containing host string
Field containing short_message string
Field containing full_message string
Field containing epoch timestamp in seconds with optional ms as decimal. Source field must be in numeric form (not a string) or Graylog will reject the input. Alternatively, use the TimeOperation to detect your timestamp field and properly parse it, and the GelfOperation will automatically use that to set the timestamp field properly. Use this setting only to override that behavior.
Field containing syslog level number
Field containing syslog facility string
Field containing the line in a file that caused the error (decimal); optional, deprecated. Send as additional field instead.
Field containing the file (with path if you want) that caused the error (string); optional, deprecated. Send as additional field instead.
{
"type": "object",
"additionalProperties": false,
"description": "Transforms JSON input into a GELF message format (See http://docs.graylog.org/en/2.4/pages/gelf.html details). Note that when selecting source fields use the JsonPath notation https://github.com/json-path/JsonPath.",
"properties": {
"type": {
"type": "string",
"enum": [
"GelfOperation"
],
"default": "GelfOperation"
},
"src_host_field": {
"type": "array",
"items": {
"type": "string"
},
"description": "Field containing host string"
},
"src_short_message_field": {
"type": "array",
"items": {
"type": "string"
},
"description": "Field containing short_message string"
},
"src_full_message_field": {
"type": "array",
"items": {
"type": "string"
},
"description": "Field containing full_message string"
},
"src_timestamp_field": {
"type": "array",
"items": {
"type": "string"
},
"description": "Field containing epoch timestamp in seconds with optional ms as decimal. Source field must be in numeric form (not a string) or Graylog will reject the input. Alternatively, use the TimeOperation to detect your timestamp field and properly parse it, and the GelfOperation will automatically use that to set the timestamp field properly. Use this setting only to override that behavior."
},
"src_level_field": {
"type": "array",
"items": {
"type": "string"
},
"description": "Field containing syslog level number"
},
"src_facility_field": {
"type": "array",
"items": {
"type": "string"
},
"description": "Field containing syslog facility string"
},
"src_line_number_field": {
"type": "array",
"items": {
"type": "string"
},
"description": "Field containing the line in a file that caused the error (decimal); optional, deprecated. Send as additional field instead."
},
"src_file_field": {
"type": "array",
"items": {
"type": "string"
},
"description": "Field containing the file (with path if you want) that caused the error (string); optional, deprecated. Send as additional field instead."
}
},
"title": "GelfOperation",
"required": [
"type",
"src_host_field",
"src_short_message_field"
]
}
Provided a JSON object it will recursively append the primitive type of the value to the key name. For example {"foo": "one", "bar": 2} will become {"foo__str": "one", "bar__long": 2}. The mapping is string:str, boolean:boolean, array:arr, number:long or __float. It also repalces "." with "_" in key names. This operation is particularily useful for modifying JSON which will be written to ElasticSearch which does not allow conflicting value types for keys or "." in key names.
{
"type": "object",
"additionalProperties": false,
"description": "Provided a JSON object it will recursively append the primitive type of the value to the key name. For example {\"foo\": \"one\", \"bar\": 2} will become {\"foo__str\": \"one\", \"bar__long\": 2}. The mapping is string:__str, boolean:__boolean, array:__arr, number:__long or __float. It also repalces \".\" with \"_\" in key names. This operation is particularily useful for modifying JSON which will be written to ElasticSearch which does not allow conflicting value types for keys or \".\" in key names.",
"properties": {
"type": {
"type": "string",
"enum": [
"JsonKeyNameOperation"
],
"default": "JsonKeyNameOperation"
}
},
"title": "JsonKeyNameOperation",
"required": [
"type"
]
}
{
"type": "object",
"additionalProperties": false,
"properties": {
"operations": {
"type": "array",
"description": "List of operations to perform.",
"anyOf": [
{
"$ref": "#/definitions/ForkOperationConfig"
},
{
"$ref": "#/definitions/LowerCaseKeyOperationConfig"
},
{
"$ref": "#/definitions/JsonRootNodeOperationConfig"
},
{
"$ref": "#/definitions/SubstitutionOperationConfig"
},
{
"$ref": "#/definitions/PartitionOperationConfig"
},
{
"$ref": "#/definitions/ArraySplitOperationConfig"
},
{
"$ref": "#/definitions/URLDecodeOperationConfig"
},
{
"$ref": "#/definitions/DeleteFieldOperationConfig"
},
{
"$ref": "#/definitions/DropArraysOperationConfig"
},
{
"$ref": "#/definitions/TimeOperationConfig"
},
{
"$ref": "#/definitions/RegexFilterOperationConfig"
},
{
"$ref": "#/definitions/GeoIpOperationConfig"
},
{
"$ref": "#/definitions/FlattenOperationConfig"
},
{
"$ref": "#/definitions/BasicFilterOperationConfig"
},
{
"$ref": "#/definitions/KeyNameReplacementOperationConfig"
},
{
"$ref": "#/definitions/ConditionalOperationConfig"
},
{
"$ref": "#/definitions/GelfOperationConfig"
},
{
"$ref": "#/definitions/KeyNameOperationConfig"
}
]
}
}
}
{
"type": "object",
"additionalProperties": false,
"description": "The fork operation allows nesting multiple operation pipelines within Bender. Each fork (pipeline) has its own operations and operates independently on a clone of the original event. If no filters are applied then this can result in multiple output events for each input event.",
"properties": {
"type": {
"type": "string",
"enum": [
"ForkOperation"
],
"default": "ForkOperation"
},
"forks": {
"type": "array",
"items": {
"$ref": "#/definitions/Fork"
},
"description": "List of forks."
}
},
"title": "ForkOperation",
"required": [
"type"
]
}
Provided a JSON object it will recursively lower case all fields.
{
"type": "object",
"additionalProperties": false,
"description": "Provided a JSON object it will recursively lower case all fields.",
"properties": {
"type": {
"type": "string",
"enum": [
"LowerCaseKeyOperation"
],
"default": "LowerCaseKeyOperation"
}
},
"title": "LowerCaseKeyOperation",
"required": [
"type"
]
}
Provided a JSON object and a path within the object it will promote the path's element to the root position. For example if the input is {"foo": {"bar": {"baz": 1}}} and specified path $.foo.bar the operation will produce {"baz": 1} as the new payload.
Path to a JSON node which is promoted to root node. See https://github.com/jayway/JsonPath
{
"type": "object",
"additionalProperties": false,
"description": "Provided a JSON object and a path within the object it will promote the path's element to the root position. For example if the input is {\"foo\": {\"bar\": {\"baz\": 1}}} and specified path $.foo.bar the operation will produce {\"baz\": 1} as the new payload.",
"properties": {
"type": {
"type": "string",
"enum": [
"JsonRootNodeOperation"
],
"default": "JsonRootNodeOperation"
},
"root_path": {
"type": "string",
"description": "Path to a JSON node which is promoted to root node. See https://github.com/jayway/JsonPath"
}
},
"title": "JsonRootNodeOperation",
"required": [
"type",
"root_path"
]
}
This operation allows substituting event fields with different sources such as other fields, static values, or metadata.
List of substitutions to perform.
Substitutes event field value for another event field value. Note the source field and destination field can be the same.
Name of the new field
Fail if destination field does not exist. This can be the case if attempting to insert into a nested destination which was not yet created.
Source fields to pull value from. If multiple fields are provided the first non-null valued one is used. Note that if no fields are found the value will be set to null.
Removes the source field when performing the substitution. Effectively making this a move operation.
Fail if source field is not found.
{
"type": "object",
"additionalProperties": false,
"description": "Substitutes event field value for another event field value. Note the source field and destination field can be the same.",
"properties": {
"type": {
"type": "string",
"enum": [
"FieldSubstitution"
],
"default": "FieldSubstitution"
},
"key": {
"type": "string",
"description": "Name of the new field"
},
"fail_dst_not_found": {
"type": "boolean",
"default": true,
"description": "Fail if destination field does not exist. This can be the case if attempting to insert into a nested destination which was not yet created."
},
"src_fields": {
"type": "array",
"items": {
"type": "string"
},
"description": "Source fields to pull value from. If multiple fields are provided the first non-null valued one is used. Note that if no fields are found the value will be set to null."
},
"remove_src_field": {
"type": "boolean",
"default": false,
"description": "Removes the source field when performing the substitution. Effectively making this a move operation."
},
"fail_src_not_found": {
"type": "boolean",
"default": true,
"description": "Fail if source field is not found."
}
},
"title": "FieldSubstitution",
"required": [
"type",
"key",
"src_fields"
]
}
Substitutes event field value for another event field value. Note the source field and destination field can be the same.
Fail if destination field does not exist. This can be the case if attempting to insert into a nested destination which was not yet created.
Regex pattern with match groups. See https://docs.oracle.com/javase/8/docs/api/java/util/regex/Pattern.html
Source fields to pull value from and apply regex to. If multiple fields are provided the first non-null valued one is used.
List of fields to create from matching regex groups.
Regex group name identifying the field.
Data type of match group field. If type coercion does not succeed then field is ignored.
Name or path of the new field.
{
"type": "object",
"additionalProperties": false,
"properties": {
"regex_group_name": {
"type": "string",
"description": "Regex group name identifying the field."
},
"type": {
"type": "string",
"enum": [
"STRING",
"NUMBER",
"BOOLEAN"
],
"description": "Data type of match group field. If type coercion does not succeed then field is ignored."
},
"key": {
"type": "string",
"description": "Name or path of the new field."
}
},
"required": [
"regex_group_name",
"type",
"key"
]
}
Removes the source field after applying this substitution.
Fail if source fields do not match regex or are not found.
{
"type": "object",
"additionalProperties": false,
"description": "Substitutes event field value for another event field value. Note the source field and destination field can be the same.",
"properties": {
"type": {
"type": "string",
"enum": [
"RegexSubstitution"
],
"default": "RegexSubstitution"
},
"fail_dst_not_found": {
"type": "boolean",
"default": true,
"description": "Fail if destination field does not exist. This can be the case if attempting to insert into a nested destination which was not yet created."
},
"pattern": {
"type": "string",
"description": "Regex pattern with match groups. See https://docs.oracle.com/javase/8/docs/api/java/util/regex/Pattern.html"
},
"src_fields": {
"type": "array",
"items": {
"type": "string"
},
"description": "Source fields to pull value from and apply regex to. If multiple fields are provided the first non-null valued one is used."
},
"fields": {
"type": "array",
"items": {
"$ref": "#/definitions/RegexSubField"
},
"description": "List of fields to create from matching regex groups."
},
"remove_src_field": {
"type": "boolean",
"default": false,
"description": "Removes the source field after applying this substitution."
},
"fail_src_not_found": {
"type": "boolean",
"default": true,
"description": "Fail if source fields do not match regex or are not found."
}
},
"title": "RegexSubstitution",
"required": [
"type",
"pattern",
"src_fields",
"fields"
]
}
Substitutes event field value for list of event metadata proprties.
Name of the new field
Fail if destination field does not exist. This can be the case if attempting to insert into a nested destination which was not yet created.
List of metadata fields to add. If non-specified then all are added.
List of metadata fields to exclude. If non-specified then none are excluded.
{
"type": "object",
"additionalProperties": false,
"description": "Substitutes event field value for list of event metadata proprties.",
"properties": {
"type": {
"type": "string",
"enum": [
"MetadataSubstitution"
],
"default": "MetadataSubstitution"
},
"key": {
"type": "string",
"description": "Name of the new field"
},
"fail_dst_not_found": {
"type": "boolean",
"default": true,
"description": "Fail if destination field does not exist. This can be the case if attempting to insert into a nested destination which was not yet created."
},
"includes": {
"type": "array",
"items": {
"type": "string"
},
"description": "List of metadata fields to add. If non-specified then all are added."
},
"excludes": {
"type": "array",
"items": {
"type": "string"
},
"description": "List of metadata fields to exclude. If non-specified then none are excluded."
}
},
"title": "MetadataSubstitution",
"required": [
"type",
"key"
]
}
The nested substitution helps build a Map object containing the result of other substitution.
Name of the new field
Fail if destination field does not exist. This can be the case if attempting to insert into a nested destination which was not yet created.
List of substitutions that will build up the final Map object.
Substitutes event field value for another event field value. Note the source field and destination field can be the same.
Name of the new field
Fail if destination field does not exist. This can be the case if attempting to insert into a nested destination which was not yet created.
Source fields to pull value from. If multiple fields are provided the first non-null valued one is used. Note that if no fields are found the value will be set to null.
Removes the source field when performing the substitution. Effectively making this a move operation.
Fail if source field is not found.
{
"type": "object",
"additionalProperties": false,
"description": "Substitutes event field value for another event field value. Note the source field and destination field can be the same.",
"properties": {
"type": {
"type": "string",
"enum": [
"FieldSubstitution"
],
"default": "FieldSubstitution"
},
"key": {
"type": "string",
"description": "Name of the new field"
},
"fail_dst_not_found": {
"type": "boolean",
"default": true,
"description": "Fail if destination field does not exist. This can be the case if attempting to insert into a nested destination which was not yet created."
},
"src_fields": {
"type": "array",
"items": {
"type": "string"
},
"description": "Source fields to pull value from. If multiple fields are provided the first non-null valued one is used. Note that if no fields are found the value will be set to null."
},
"remove_src_field": {
"type": "boolean",
"default": false,
"description": "Removes the source field when performing the substitution. Effectively making this a move operation."
},
"fail_src_not_found": {
"type": "boolean",
"default": true,
"description": "Fail if source field is not found."
}
},
"title": "FieldSubstitution",
"required": [
"type",
"key",
"src_fields"
]
}
Substitutes event field value for another event field value. Note the source field and destination field can be the same.
Fail if destination field does not exist. This can be the case if attempting to insert into a nested destination which was not yet created.
Regex pattern with match groups. See https://docs.oracle.com/javase/8/docs/api/java/util/regex/Pattern.html
Source fields to pull value from and apply regex to. If multiple fields are provided the first non-null valued one is used.
List of fields to create from matching regex groups.
Regex group name identifying the field.
Data type of match group field. If type coercion does not succeed then field is ignored.
Name or path of the new field.
{
"type": "object",
"additionalProperties": false,
"properties": {
"regex_group_name": {
"type": "string",
"description": "Regex group name identifying the field."
},
"type": {
"type": "string",
"enum": [
"STRING",
"NUMBER",
"BOOLEAN"
],
"description": "Data type of match group field. If type coercion does not succeed then field is ignored."
},
"key": {
"type": "string",
"description": "Name or path of the new field."
}
},
"required": [
"regex_group_name",
"type",
"key"
]
}
Removes the source field after applying this substitution.
Fail if source fields do not match regex or are not found.
{
"type": "object",
"additionalProperties": false,
"description": "Substitutes event field value for another event field value. Note the source field and destination field can be the same.",
"properties": {
"type": {
"type": "string",
"enum": [
"RegexSubstitution"
],
"default": "RegexSubstitution"
},
"fail_dst_not_found": {
"type": "boolean",
"default": true,
"description": "Fail if destination field does not exist. This can be the case if attempting to insert into a nested destination which was not yet created."
},
"pattern": {
"type": "string",
"description": "Regex pattern with match groups. See https://docs.oracle.com/javase/8/docs/api/java/util/regex/Pattern.html"
},
"src_fields": {
"type": "array",
"items": {
"type": "string"
},
"description": "Source fields to pull value from and apply regex to. If multiple fields are provided the first non-null valued one is used."
},
"fields": {
"type": "array",
"items": {
"$ref": "#/definitions/RegexSubField"
},
"description": "List of fields to create from matching regex groups."
},
"remove_src_field": {
"type": "boolean",
"default": false,
"description": "Removes the source field after applying this substitution."
},
"fail_src_not_found": {
"type": "boolean",
"default": true,
"description": "Fail if source fields do not match regex or are not found."
}
},
"title": "RegexSubstitution",
"required": [
"type",
"pattern",
"src_fields",
"fields"
]
}
Substitutes event field value for list of event metadata proprties.
Name of the new field
Fail if destination field does not exist. This can be the case if attempting to insert into a nested destination which was not yet created.
List of metadata fields to add. If non-specified then all are added.
List of metadata fields to exclude. If non-specified then none are excluded.
{
"type": "object",
"additionalProperties": false,
"description": "Substitutes event field value for list of event metadata proprties.",
"properties": {
"type": {
"type": "string",
"enum": [
"MetadataSubstitution"
],
"default": "MetadataSubstitution"
},
"key": {
"type": "string",
"description": "Name of the new field"
},
"fail_dst_not_found": {
"type": "boolean",
"default": true,
"description": "Fail if destination field does not exist. This can be the case if attempting to insert into a nested destination which was not yet created."
},
"includes": {
"type": "array",
"items": {
"type": "string"
},
"description": "List of metadata fields to add. If non-specified then all are added."
},
"excludes": {
"type": "array",
"items": {
"type": "string"
},
"description": "List of metadata fields to exclude. If non-specified then none are excluded."
}
},
"title": "MetadataSubstitution",
"required": [
"type",
"key"
]
}
circular reference
{
"description": "_circular reference_"
}
Creates a new string using variable replacement.
Name of the new field
Fail if destination field does not exist. This can be the case if attempting to insert into a nested destination which was not yet created.
String with variable indices to substitue. See https://docs.oracle.com/javase/8/docs/api/java/text/MessageFormat.html
List of variables used in string substitution. Index of variable relates to index in 'format' string.
Source fields to pull value from. If multiple fields are provided the first non-null valued one is used. Note that if no fields are found the value will be set to null.
Fail if source field is not found.
Removes the source field when performing the substitution.
{
"type": "object",
"additionalProperties": false,
"properties": {
"type": {
"type": "string",
"enum": [
"FieldVariable"
],
"default": "FieldVariable"
},
"src_fields": {
"type": "array",
"items": {
"type": "string"
},
"description": "Source fields to pull value from. If multiple fields are provided the first non-null valued one is used. Note that if no fields are found the value will be set to null."
},
"fail_src_not_found": {
"type": "boolean",
"default": true,
"description": "Fail if source field is not found."
},
"remove_src_field": {
"type": "boolean",
"default": false,
"description": "Removes the source field when performing the substitution."
}
},
"title": "FieldVariable",
"required": [
"type",
"src_fields"
]
}
Value to substitute.
{
"type": "object",
"additionalProperties": false,
"properties": {
"type": {
"type": "string",
"enum": [
"StaticVariable"
],
"default": "StaticVariable"
},
"value": {
"type": "string",
"description": "Value to substitute."
}
},
"title": "StaticVariable",
"required": [
"type",
"value"
]
}
{
"type": "object",
"additionalProperties": false,
"description": "Creates a new string using variable replacement.",
"properties": {
"type": {
"type": "string",
"enum": [
"FormattedSubstitution"
],
"default": "FormattedSubstitution"
},
"key": {
"type": "string",
"description": "Name of the new field"
},
"fail_dst_not_found": {
"type": "boolean",
"default": true,
"description": "Fail if destination field does not exist. This can be the case if attempting to insert into a nested destination which was not yet created."
},
"format": {
"type": "string",
"description": "String with variable indices to substitue. See https://docs.oracle.com/javase/8/docs/api/java/text/MessageFormat.html"
},
"variables": {
"type": "array",
"description": "List of variables used in string substitution. Index of variable relates to index in 'format' string.",
"anyOf": [
{
"$ref": "#/definitions/FieldVariable"
},
{
"$ref": "#/definitions/StaticVariable"
}
]
}
},
"title": "FormattedSubstitution",
"required": [
"type",
"key",
"format",
"variables"
]
}
Substitutes event field value for a static value.
Name of the new field
Fail if destination field does not exist. This can be the case if attempting to insert into a nested destination which was not yet created.
Value of the new field.
{
"type": "object",
"additionalProperties": false,
"description": "Substitutes event field value for a static value.",
"properties": {
"type": {
"type": "string",
"enum": [
"StaticSubstitution"
],
"default": "StaticSubstitution"
},
"key": {
"type": "string",
"description": "Name of the new field"
},
"fail_dst_not_found": {
"type": "boolean",
"default": true,
"description": "Fail if destination field does not exist. This can be the case if attempting to insert into a nested destination which was not yet created."
},
"value": {
"type": "string",
"description": "Value of the new field."
}
},
"title": "StaticSubstitution",
"required": [
"type",
"key",
"value"
]
}
Substitutes event field value for list of lambda function context proprties.
Name of the new field
Fail if destination field does not exist. This can be the case if attempting to insert into a nested destination which was not yet created.
List of context fields to add. If non-specified then all are added.
List of context fields to exclude. If non-specified then none are excluded.
{
"type": "object",
"additionalProperties": false,
"description": "Substitutes event field value for list of lambda function context proprties.",
"properties": {
"type": {
"type": "string",
"enum": [
"ContextSubstitution"
],
"default": "ContextSubstitution"
},
"key": {
"type": "string",
"description": "Name of the new field"
},
"fail_dst_not_found": {
"type": "boolean",
"default": true,
"description": "Fail if destination field does not exist. This can be the case if attempting to insert into a nested destination which was not yet created."
},
"includes": {
"type": "array",
"items": {
"type": "string"
},
"description": "List of context fields to add. If non-specified then all are added."
},
"excludes": {
"type": "array",
"items": {
"type": "string"
},
"description": "List of context fields to exclude. If non-specified then none are excluded."
}
},
"title": "ContextSubstitution",
"required": [
"type",
"key"
]
}
{
"type": "object",
"additionalProperties": false,
"description": "The nested substitution helps build a Map object containing the result of other substitution.",
"properties": {
"type": {
"type": "string",
"enum": [
"NestedSubstitution"
],
"default": "NestedSubstitution"
},
"key": {
"type": "string",
"description": "Name of the new field"
},
"fail_dst_not_found": {
"type": "boolean",
"default": true,
"description": "Fail if destination field does not exist. This can be the case if attempting to insert into a nested destination which was not yet created."
},
"substitutions": {
"type": "array",
"description": "List of substitutions that will build up the final Map object.",
"anyOf": [
{
"$ref": "#/definitions/FieldSubstitutionConfig"
},
{
"$ref": "#/definitions/RegexSubstitutionConfig"
},
{
"$ref": "#/definitions/MetadataSubstitutionConfig"
},
{
"$ref": "#/definitions/NestedSubstitutionConfig"
},
{
"$ref": "#/definitions/FormattedSubstitutionConfig"
},
{
"$ref": "#/definitions/StaticSubstitutionConfig"
},
{
"$ref": "#/definitions/ContextSubstitutionConfig"
}
]
}
},
"title": "NestedSubstitution",
"required": [
"type",
"key"
]
}
Creates a new string using variable replacement.
Name of the new field
Fail if destination field does not exist. This can be the case if attempting to insert into a nested destination which was not yet created.
String with variable indices to substitue. See https://docs.oracle.com/javase/8/docs/api/java/text/MessageFormat.html
List of variables used in string substitution. Index of variable relates to index in 'format' string.
Source fields to pull value from. If multiple fields are provided the first non-null valued one is used. Note that if no fields are found the value will be set to null.
Fail if source field is not found.
Removes the source field when performing the substitution.
{
"type": "object",
"additionalProperties": false,
"properties": {
"type": {
"type": "string",
"enum": [
"FieldVariable"
],
"default": "FieldVariable"
},
"src_fields": {
"type": "array",
"items": {
"type": "string"
},
"description": "Source fields to pull value from. If multiple fields are provided the first non-null valued one is used. Note that if no fields are found the value will be set to null."
},
"fail_src_not_found": {
"type": "boolean",
"default": true,
"description": "Fail if source field is not found."
},
"remove_src_field": {
"type": "boolean",
"default": false,
"description": "Removes the source field when performing the substitution."
}
},
"title": "FieldVariable",
"required": [
"type",
"src_fields"
]
}
Value to substitute.
{
"type": "object",
"additionalProperties": false,
"properties": {
"type": {
"type": "string",
"enum": [
"StaticVariable"
],
"default": "StaticVariable"
},
"value": {
"type": "string",
"description": "Value to substitute."
}
},
"title": "StaticVariable",
"required": [
"type",
"value"
]
}
{
"type": "object",
"additionalProperties": false,
"description": "Creates a new string using variable replacement.",
"properties": {
"type": {
"type": "string",
"enum": [
"FormattedSubstitution"
],
"default": "FormattedSubstitution"
},
"key": {
"type": "string",
"description": "Name of the new field"
},
"fail_dst_not_found": {
"type": "boolean",
"default": true,
"description": "Fail if destination field does not exist. This can be the case if attempting to insert into a nested destination which was not yet created."
},
"format": {
"type": "string",
"description": "String with variable indices to substitue. See https://docs.oracle.com/javase/8/docs/api/java/text/MessageFormat.html"
},
"variables": {
"type": "array",
"description": "List of variables used in string substitution. Index of variable relates to index in 'format' string.",
"anyOf": [
{
"$ref": "#/definitions/FieldVariable"
},
{
"$ref": "#/definitions/StaticVariable"
}
]
}
},
"title": "FormattedSubstitution",
"required": [
"type",
"key",
"format",
"variables"
]
}
Substitutes event field value for a static value.
Name of the new field
Fail if destination field does not exist. This can be the case if attempting to insert into a nested destination which was not yet created.
Value of the new field.
{
"type": "object",
"additionalProperties": false,
"description": "Substitutes event field value for a static value.",
"properties": {
"type": {
"type": "string",
"enum": [
"StaticSubstitution"
],
"default": "StaticSubstitution"
},
"key": {
"type": "string",
"description": "Name of the new field"
},
"fail_dst_not_found": {
"type": "boolean",
"default": true,
"description": "Fail if destination field does not exist. This can be the case if attempting to insert into a nested destination which was not yet created."
},
"value": {
"type": "string",
"description": "Value of the new field."
}
},
"title": "StaticSubstitution",
"required": [
"type",
"key",
"value"
]
}
Substitutes event field value for list of lambda function context proprties.
Name of the new field
Fail if destination field does not exist. This can be the case if attempting to insert into a nested destination which was not yet created.
List of context fields to add. If non-specified then all are added.
List of context fields to exclude. If non-specified then none are excluded.
{
"type": "object",
"additionalProperties": false,
"description": "Substitutes event field value for list of lambda function context proprties.",
"properties": {
"type": {
"type": "string",
"enum": [
"ContextSubstitution"
],
"default": "ContextSubstitution"
},
"key": {
"type": "string",
"description": "Name of the new field"
},
"fail_dst_not_found": {
"type": "boolean",
"default": true,
"description": "Fail if destination field does not exist. This can be the case if attempting to insert into a nested destination which was not yet created."
},
"includes": {
"type": "array",
"items": {
"type": "string"
},
"description": "List of context fields to add. If non-specified then all are added."
},
"excludes": {
"type": "array",
"items": {
"type": "string"
},
"description": "List of context fields to exclude. If non-specified then none are excluded."
}
},
"title": "ContextSubstitution",
"required": [
"type",
"key"
]
}
{
"type": "object",
"additionalProperties": false,
"description": "This operation allows substituting event fields with different sources such as other fields, static values, or metadata.",
"properties": {
"type": {
"type": "string",
"enum": [
"SubstitutionOperation"
],
"default": "SubstitutionOperation"
},
"substitutions": {
"type": "array",
"description": "List of substitutions to perform.",
"anyOf": [
{
"$ref": "#/definitions/FieldSubstitutionConfig"
},
{
"$ref": "#/definitions/RegexSubstitutionConfig"
},
{
"$ref": "#/definitions/MetadataSubstitutionConfig"
},
{
"$ref": "#/definitions/NestedSubstitutionConfig"
},
{
"$ref": "#/definitions/FormattedSubstitutionConfig"
},
{
"$ref": "#/definitions/StaticSubstitutionConfig"
},
{
"$ref": "#/definitions/ContextSubstitutionConfig"
}
]
}
},
"title": "SubstitutionOperation",
"required": [
"type",
"substitutions"
]
}
Sets the partition information for the Event using fields from the deserialized object. When using JSON use JsonPath format to specify fields. See https://github.com/jayway/JsonPath
Configuration to specify object fields that are treated as partitions
Value to use as the key for the partition.
Fields to use for the value of the partition.
Interpreter to use on the partition value.
Java date format to use when using a time based interpreter.
Basic string formatting.
Rounds time down to value prior to formatting. For example 300 would round down to nearest 5 minutes. If set to 0 then rounding is disabled.
{
"type": "object",
"additionalProperties": false,
"properties": {
"name": {
"type": "string",
"description": "Value to use as the key for the partition."
},
"sources": {
"type": "array",
"items": {
"type": "string"
},
"description": "Fields to use for the value of the partition."
},
"interpreter": {
"type": "string",
"default": "STRING",
"enum": [
"STRING",
"MILLISECONDS",
"SECONDS",
"STATIC"
],
"description": "Interpreter to use on the partition value."
},
"format": {
"type": "string",
"description": "Java date format to use when using a time based interpreter."
},
"string_format": {
"type": "string",
"default": "NONE",
"enum": [
"TOLOWER",
"TOUPPER",
"NONE"
],
"description": "Basic string formatting."
},
"seconds_to_round": {
"type": "integer",
"minimum": 0,
"maximum": 86400,
"default": 0,
"description": "Rounds time down to value prior to formatting. For example 300 would round down to nearest 5 minutes. If set to 0 then rounding is disabled."
}
},
"required": [
"name",
"interpreter"
]
}
{
"type": "object",
"additionalProperties": false,
"description": "Sets the partition information for the Event using fields from the deserialized object. When using JSON use JsonPath format to specify fields. See https://github.com/jayway/JsonPath",
"properties": {
"type": {
"type": "string",
"enum": [
"PartitionOperation"
],
"default": "PartitionOperation"
},
"partition_specs": {
"type": "array",
"items": {
"$ref": "#/definitions/PartitionSpec"
},
"description": "Configuration to specify object fields that are treated as partitions"
}
},
"title": "PartitionOperation",
"required": [
"type"
]
}
Provided a path to a JSON array it will produce new events with payloads coorepsonding to elements of the array. For example [{"foo": 1}, {"bar": 2}] will be turned into two seperate events with payloads of {"foo": 1} and {"bar": 2}.
If an array is found and split, this can specify additional fields to keep in the new JSON object to preserve common contexts, such as timestamps, accountId, etc.
{
"type": "object",
"additionalProperties": false,
"description": "Provided a path to a JSON array it will produce new events with payloads coorepsonding to elements of the array. For example [{\"foo\": 1}, {\"bar\": 2}] will be turned into two seperate events with payloads of {\"foo\": 1} and {\"bar\": 2}.",
"properties": {
"type": {
"type": "string",
"enum": [
"JsonArraySplitOperation"
],
"default": "JsonArraySplitOperation"
},
"path": {
"type": "string",
"description": "Path to a JSON node which is an array. See https://github.com/jayway/JsonPath"
},
"fields_to_keep": {
"type": "array",
"items": {
"type": "string"
},
"description": "If an array is found and split, this can specify additional fields to keep in the new JSON object to preserve common contexts, such as timestamps, accountId, etc."
}
},
"title": "JsonArraySplitOperation",
"required": [
"type",
"path"
]
}
Performs a URL decode on the specified fields.
fields
Number of times to perform URL decode on field. This is helpful if a field is encoded multiple times.
{
"type": "object",
"additionalProperties": false,
"description": "Performs a URL decode on the specified fields.",
"properties": {
"type": {
"type": "string",
"enum": [
"UrlDecodeOperation"
],
"default": "UrlDecodeOperation"
},
"fields": {
"type": "array",
"items": {
"type": "string"
},
"description": "fields"
},
"times": {
"type": "integer",
"minimum": 1,
"maximum": 65535,
"default": 1,
"description": "Number of times to perform URL decode on field. This is helpful if a field is encoded multiple times."
}
},
"title": "UrlDecodeOperation",
"required": [
"type",
"fields"
]
}
Deletes the inputted field in the deserialized object. When using JSON use JsonPath format to specify field. See https://github.com/jayway/JsonPath
Name of field key to be deleted.
{
"type": "object",
"additionalProperties": false,
"description": "Deletes the inputted field in the deserialized object. When using JSON use JsonPath format to specify field. See https://github.com/jayway/JsonPath",
"properties": {
"type": {
"type": "string",
"enum": [
"DeleteFieldOperation"
],
"default": "DeleteFieldOperation"
},
"key_field": {
"type": "string",
"description": "Name of field key to be deleted."
}
},
"title": "DeleteFieldOperation",
"required": [
"type",
"key_field"
]
}
Provided a JSON object it will remove any keys which have array values. This is helpful for use with ElasticSearch which indexes arrays in a particular manner not conducive to exploration. See: https://www.elastic.co/guide/en/elasticsearch/guide/current/complex-core-fields.html
{
"type": "object",
"additionalProperties": false,
"description": "Provided a JSON object it will remove any keys which have array values. This is helpful for use with ElasticSearch which indexes arrays in a particular manner not conducive to exploration. See: https://www.elastic.co/guide/en/elasticsearch/guide/current/complex-core-fields.html",
"properties": {
"type": {
"type": "string",
"enum": [
"JsonDropArraysOperation"
],
"default": "JsonDropArraysOperation"
}
},
"title": "JsonDropArraysOperation",
"required": [
"type"
]
}
Sets the Event timestamp using the provided field in the deserialized object. When using JSON use JsonPath format to specify field. See https://github.com/jayway/JsonPath
Name of field to use as time field
How to interpret time field
{
"type": "object",
"additionalProperties": false,
"description": "Sets the Event timestamp using the provided field in the deserialized object. When using JSON use JsonPath format to specify field. See https://github.com/jayway/JsonPath",
"properties": {
"type": {
"type": "string",
"enum": [
"TimeOperation"
],
"default": "TimeOperation"
},
"time_field": {
"type": "string",
"description": "Name of field to use as time field"
},
"time_field_type": {
"type": "string",
"enum": [
"SECONDS",
"MILLISECONDS",
"ISO8601"
],
"description": "How to interpret time field"
}
},
"title": "TimeOperation",
"required": [
"type",
"time_field",
"time_field_type"
]
}
This operation is used to remove certain events from the stream before continuing on to the destination. Each event is assessed by applying a JsonPath to its payload and matching the value against a regex Pattern. If exclude is true, events that match this criteria will be filtered out. If exclude is false, any events not matching this criteria will be filtered out. For example, say these two events are in the stream: {\"data\": \"one\", \"type\": \"bar\"} and {\"data\": \"one\", \"type\": \"baz\"}. With config values: regex = \"(bar)\", path = \"$.type\", and exclude = true, after filtering, the stream will only hold {\"data\": \"one\", \"type\": \"baz\"}. If instead, exclude = false, the stream would only hold {\"data\": \"one\", \"type\": \"bar\"} after filtering.
Regex to be matched against JSON objects. See https://docs.oracle.com/javase/8/docs/api/java/util/regex/Pattern.html
If true, matches will be filtered out. If false, non-matches will be filtered out.
{
"type": "object",
"additionalProperties": false,
"description": "This operation is used to remove certain events from the stream before continuing on to the destination. Each event is assessed by applying a JsonPath to its payload and matching the value against a regex Pattern. If exclude is true, events that match this criteria will be filtered out. If exclude is false, any events not matching this criteria will be filtered out. For example, say these two events are in the stream: {\\\"data\\\": \\\"one\\\", \\\"type\\\": \\\"bar\\\"} and {\\\"data\\\": \\\"one\\\", \\\"type\\\": \\\"baz\\\"}. With config values: regex = \\\"(bar)\\\", path = \\\"$.type\\\", and exclude = true, after filtering, the stream will only hold {\\\"data\\\": \\\"one\\\", \\\"type\\\": \\\"baz\\\"}. If instead, exclude = false, the stream would only hold {\\\"data\\\": \\\"one\\\", \\\"type\\\": \\\"bar\\\"} after filtering.",
"properties": {
"type": {
"type": "string",
"enum": [
"RegexFilterOperation"
],
"default": "RegexFilterOperation"
},
"regex": {
"type": "string",
"description": "Regex to be matched against JSON objects. See https://docs.oracle.com/javase/8/docs/api/java/util/regex/Pattern.html"
},
"path": {
"type": "string",
"description": "JsonPath to the element to be matched against. See https://github.com/jayway/JsonPath"
},
"exclude": {
"type": "boolean",
"default": true,
"description": "If true, matches will be filtered out. If false, non-matches will be filtered out."
}
},
"title": "RegexFilterOperation",
"required": [
"type",
"regex",
"path"
]
}
Looks up geo location provided an IP address and adds a map field to the payload that optionally contains country_name, country_iso_code, subdivision_name, subdivision_iso_code, city_name, postal_code, and location. For example
input = {"ip": "8.8.8.8"}
output = {"ip": "8.8.8.8", "geo_ip": {"location": {"lat": 37.751, "lon": -97.822}}}
Note 1: This operation requires the MaxMind GeoLite2 City Database. It is available at https://dev.maxmind.com/geoip/geoip2/geolite2/. After you download the database, extract the tarball and uploaded the .mmdb file to an S3 bucket accessible by your lambda function.
Note 2: If your ip address field contains a comma separated list of ip addresses (ie forwarded web requests) the first address in the list is used. This is typically the source address.
Field containing an IP address string in the payload
Field name to save the geo data to
S3 path to the GeoLite2 City database file. Prefix with s3://
List of geo properties to include
If ip lookup fails also fail the operation. Doing so will filter out the event.
{
"type": "object",
"additionalProperties": false,
"description": "Looks up geo location provided an IP address and adds a map field to the payload that optionally contains country_name, country_iso_code, subdivision_name, subdivision_iso_code, city_name, postal_code, and location. For example \n\n input = {\"ip\": \"8.8.8.8\"} \n\n output = {\"ip\": \"8.8.8.8\", \"geo_ip\": {\"location\": {\"lat\": 37.751, \"lon\": -97.822}}}\n\n\n\nNote 1: This operation requires the MaxMind GeoLite2 City Database. It is available at https://dev.maxmind.com/geoip/geoip2/geolite2/. After you download the database, extract the tarball and uploaded the .mmdb file to an S3 bucket accessible by your lambda function.\n\nNote 2: If your ip address field contains a comma separated list of ip addresses (ie forwarded web requests) the first address in the list is used. This is typically the source address.",
"properties": {
"type": {
"type": "string",
"enum": [
"GeoIpOperation"
],
"default": "GeoIpOperation"
},
"src_field_name": {
"type": "string",
"description": "Field containing an IP address string in the payload"
},
"dst_field_name": {
"type": "string",
"default": "geo_ip",
"description": "Field name to save the geo data to"
},
"geo_lite_db": {
"type": "string",
"description": "S3 path to the GeoLite2 City database file. Prefix with s3://"
},
"geo_properties": {
"type": "array",
"items": {
"type": "string",
"enum": [
"COUNTRY_NAME",
"COUNTRY_ISO_CODE",
"SUBDIVISION_NAME",
"SUBDIVISION_ISO_CODE",
"CITY_NAME",
"POSTAL_CODE",
"LOCATION"
]
},
"description": "List of geo properties to include"
},
"fail_on_not_found": {
"type": "boolean",
"default": false,
"description": "If ip lookup fails also fail the operation. Doing so will filter out the event."
}
},
"title": "GeoIpOperation",
"required": [
"type",
"src_field_name",
"dst_field_name",
"geo_lite_db"
]
}
Provided a deeply nested JSON Object, it will flatten out the object into keys with a specific separator (dot by default). For example, if the input is {"foo": {"bar": {"baz": 1}}} the operation will produce {"foo.bar.baz": 1} as the new payload.
Separator to be used between nested key names (typically a dot(.))
{
"type": "object",
"additionalProperties": false,
"description": "Provided a deeply nested JSON Object, it will flatten out the object into keys with a specific separator (dot by default). For example, if the input is {\"foo\": {\"bar\": {\"baz\": 1}}} the operation will produce {\"foo.bar.baz\": 1} as the new payload.",
"properties": {
"type": {
"type": "string",
"enum": [
"FlattenOperation"
],
"default": "FlattenOperation"
},
"separator": {
"type": "string",
"default": ".",
"description": "Separator to be used between nested key names (typically a dot(.))"
}
},
"title": "FlattenOperation",
"required": [
"type"
]
}
If pass is set to false then events are filtered out.
If 'false', event is filtered out.
{
"type": "object",
"additionalProperties": false,
"description": "If pass is set to false then events are filtered out.",
"properties": {
"type": {
"type": "string",
"enum": [
"BasicFilterOperation"
],
"default": "BasicFilterOperation"
},
"pass": {
"type": "boolean",
"default": false,
"description": "If 'false', event is filtered out."
}
},
"title": "BasicFilterOperation",
"required": [
"type"
]
}
Uses a regex to find matching parts of a key name and replaces with value or drops the key/value if a match is found. This is typically used to sanitize key names.
If a match is found then replace match with this value. Defaults to emptry string.
If a match is found then drop the key/value.
{
"type": "object",
"additionalProperties": false,
"description": "Uses a regex to find matching parts of a key name and replaces with value or drops the key/value if a match is found. This is typically used to sanitize key names.",
"properties": {
"type": {
"type": "string",
"enum": [
"KeyNameReplacementOperation"
],
"default": "KeyNameReplacementOperation"
},
"regex": {
"type": "string",
"description": "Java regex. See https://docs.oracle.com/javase/8/docs/api/java/util/regex/Pattern.html"
},
"replacement": {
"type": "string",
"default": "",
"description": "If a match is found then replace match with this value. Defaults to emptry string."
},
"drop": {
"type": "boolean",
"default": false,
"description": "If a match is found then drop the key/value."
}
},
"title": "KeyNameReplacementOperation",
"required": [
"type",
"regex"
]
}
The conditional operation allows for 'if else' style branches in the flow of Bender operations. Conditions are evaluated in order of appearance and the first matching condition will receive the event.
List of conditions and their operations.
List of operations to perform.
The fork operation allows nesting multiple operation pipelines within Bender. Each fork (pipeline) has its own operations and operates independently on a clone of the original event. If no filters are applied then this can result in multiple output events for each input event.
List of forks.
List of operations to perform.
circular reference
{
"description": "_circular reference_"
}
Provided a JSON object it will recursively lower case all fields.
{
"type": "object",
"additionalProperties": false,
"description": "Provided a JSON object it will recursively lower case all fields.",
"properties": {
"type": {
"type": "string",
"enum": [
"LowerCaseKeyOperation"
],
"default": "LowerCaseKeyOperation"
}
},
"title": "LowerCaseKeyOperation",
"required": [
"type"
]
}
Provided a JSON object and a path within the object it will promote the path's element to the root position. For example if the input is {"foo": {"bar": {"baz": 1}}} and specified path $.foo.bar the operation will produce {"baz": 1} as the new payload.
Path to a JSON node which is promoted to root node. See https://github.com/jayway/JsonPath
{
"type": "object",
"additionalProperties": false,
"description": "Provided a JSON object and a path within the object it will promote the path's element to the root position. For example if the input is {\"foo\": {\"bar\": {\"baz\": 1}}} and specified path $.foo.bar the operation will produce {\"baz\": 1} as the new payload.",
"properties": {
"type": {
"type": "string",
"enum": [
"JsonRootNodeOperation"
],
"default": "JsonRootNodeOperation"
},
"root_path": {
"type": "string",
"description": "Path to a JSON node which is promoted to root node. See https://github.com/jayway/JsonPath"
}
},
"title": "JsonRootNodeOperation",
"required": [
"type",
"root_path"
]
}
This operation allows substituting event fields with different sources such as other fields, static values, or metadata.
List of substitutions to perform.
Substitutes event field value for another event field value. Note the source field and destination field can be the same.
Name of the new field
Fail if destination field does not exist. This can be the case if attempting to insert into a nested destination which was not yet created.
Source fields to pull value from. If multiple fields are provided the first non-null valued one is used. Note that if no fields are found the value will be set to null.
Removes the source field when performing the substitution. Effectively making this a move operation.
Fail if source field is not found.
{
"type": "object",
"additionalProperties": false,
"description": "Substitutes event field value for another event field value. Note the source field and destination field can be the same.",
"properties": {
"type": {
"type": "string",
"enum": [
"FieldSubstitution"
],
"default": "FieldSubstitution"
},
"key": {
"type": "string",
"description": "Name of the new field"
},
"fail_dst_not_found": {
"type": "boolean",
"default": true,
"description": "Fail if destination field does not exist. This can be the case if attempting to insert into a nested destination which was not yet created."
},
"src_fields": {
"type": "array",
"items": {
"type": "string"
},
"description": "Source fields to pull value from. If multiple fields are provided the first non-null valued one is used. Note that if no fields are found the value will be set to null."
},
"remove_src_field": {
"type": "boolean",
"default": false,
"description": "Removes the source field when performing the substitution. Effectively making this a move operation."
},
"fail_src_not_found": {
"type": "boolean",
"default": true,
"description": "Fail if source field is not found."
}
},
"title": "FieldSubstitution",
"required": [
"type",
"key",
"src_fields"
]
}
Substitutes event field value for another event field value. Note the source field and destination field can be the same.
Fail if destination field does not exist. This can be the case if attempting to insert into a nested destination which was not yet created.
Regex pattern with match groups. See https://docs.oracle.com/javase/8/docs/api/java/util/regex/Pattern.html
Source fields to pull value from and apply regex to. If multiple fields are provided the first non-null valued one is used.
List of fields to create from matching regex groups.
Regex group name identifying the field.
Data type of match group field. If type coercion does not succeed then field is ignored.
Name or path of the new field.
{
"type": "object",
"additionalProperties": false,
"properties": {
"regex_group_name": {
"type": "string",
"description": "Regex group name identifying the field."
},
"type": {
"type": "string",
"enum": [
"STRING",
"NUMBER",
"BOOLEAN"
],
"description": "Data type of match group field. If type coercion does not succeed then field is ignored."
},
"key": {
"type": "string",
"description": "Name or path of the new field."
}
},
"required": [
"regex_group_name",
"type",
"key"
]
}
Removes the source field after applying this substitution.
Fail if source fields do not match regex or are not found.
{
"type": "object",
"additionalProperties": false,
"description": "Substitutes event field value for another event field value. Note the source field and destination field can be the same.",
"properties": {
"type": {
"type": "string",
"enum": [
"RegexSubstitution"
],
"default": "RegexSubstitution"
},
"fail_dst_not_found": {
"type": "boolean",
"default": true,
"description": "Fail if destination field does not exist. This can be the case if attempting to insert into a nested destination which was not yet created."
},
"pattern": {
"type": "string",
"description": "Regex pattern with match groups. See https://docs.oracle.com/javase/8/docs/api/java/util/regex/Pattern.html"
},
"src_fields": {
"type": "array",
"items": {
"type": "string"
},
"description": "Source fields to pull value from and apply regex to. If multiple fields are provided the first non-null valued one is used."
},
"fields": {
"type": "array",
"items": {
"$ref": "#/definitions/RegexSubField"
},
"description": "List of fields to create from matching regex groups."
},
"remove_src_field": {
"type": "boolean",
"default": false,
"description": "Removes the source field after applying this substitution."
},
"fail_src_not_found": {
"type": "boolean",
"default": true,
"description": "Fail if source fields do not match regex or are not found."
}
},
"title": "RegexSubstitution",
"required": [
"type",
"pattern",
"src_fields",
"fields"
]
}
Substitutes event field value for list of event metadata proprties.
Name of the new field
Fail if destination field does not exist. This can be the case if attempting to insert into a nested destination which was not yet created.
List of metadata fields to add. If non-specified then all are added.
List of metadata fields to exclude. If non-specified then none are excluded.
{
"type": "object",
"additionalProperties": false,
"description": "Substitutes event field value for list of event metadata proprties.",
"properties": {
"type": {
"type": "string",
"enum": [
"MetadataSubstitution"
],
"default": "MetadataSubstitution"
},
"key": {
"type": "string",
"description": "Name of the new field"
},
"fail_dst_not_found": {
"type": "boolean",
"default": true,
"description": "Fail if destination field does not exist. This can be the case if attempting to insert into a nested destination which was not yet created."
},
"includes": {
"type": "array",
"items": {
"type": "string"
},
"description": "List of metadata fields to add. If non-specified then all are added."
},
"excludes": {
"type": "array",
"items": {
"type": "string"
},
"description": "List of metadata fields to exclude. If non-specified then none are excluded."
}
},
"title": "MetadataSubstitution",
"required": [
"type",
"key"
]
}
The nested substitution helps build a Map object containing the result of other substitution.
Name of the new field
Fail if destination field does not exist. This can be the case if attempting to insert into a nested destination which was not yet created.
List of substitutions that will build up the final Map object.
Substitutes event field value for another event field value. Note the source field and destination field can be the same.
Name of the new field
Fail if destination field does not exist. This can be the case if attempting to insert into a nested destination which was not yet created.
Source fields to pull value from. If multiple fields are provided the first non-null valued one is used. Note that if no fields are found the value will be set to null.
Removes the source field when performing the substitution. Effectively making this a move operation.
Fail if source field is not found.
{
"type": "object",
"additionalProperties": false,
"description": "Substitutes event field value for another event field value. Note the source field and destination field can be the same.",
"properties": {
"type": {
"type": "string",
"enum": [
"FieldSubstitution"
],
"default": "FieldSubstitution"
},
"key": {
"type": "string",
"description": "Name of the new field"
},
"fail_dst_not_found": {
"type": "boolean",
"default": true,
"description": "Fail if destination field does not exist. This can be the case if attempting to insert into a nested destination which was not yet created."
},
"src_fields": {
"type": "array",
"items": {
"type": "string"
},
"description": "Source fields to pull value from. If multiple fields are provided the first non-null valued one is used. Note that if no fields are found the value will be set to null."
},
"remove_src_field": {
"type": "boolean",
"default": false,
"description": "Removes the source field when performing the substitution. Effectively making this a move operation."
},
"fail_src_not_found": {
"type": "boolean",
"default": true,
"description": "Fail if source field is not found."
}
},
"title": "FieldSubstitution",
"required": [
"type",
"key",
"src_fields"
]
}
Substitutes event field value for another event field value. Note the source field and destination field can be the same.
Fail if destination field does not exist. This can be the case if attempting to insert into a nested destination which was not yet created.
Regex pattern with match groups. See https://docs.oracle.com/javase/8/docs/api/java/util/regex/Pattern.html
Source fields to pull value from and apply regex to. If multiple fields are provided the first non-null valued one is used.
List of fields to create from matching regex groups.
Regex group name identifying the field.
Data type of match group field. If type coercion does not succeed then field is ignored.
Name or path of the new field.
{
"type": "object",
"additionalProperties": false,
"properties": {
"regex_group_name": {
"type": "string",
"description": "Regex group name identifying the field."
},
"type": {
"type": "string",
"enum": [
"STRING",
"NUMBER",
"BOOLEAN"
],
"description": "Data type of match group field. If type coercion does not succeed then field is ignored."
},
"key": {
"type": "string",
"description": "Name or path of the new field."
}
},
"required": [
"regex_group_name",
"type",
"key"
]
}
Removes the source field after applying this substitution.
Fail if source fields do not match regex or are not found.
{
"type": "object",
"additionalProperties": false,
"description": "Substitutes event field value for another event field value. Note the source field and destination field can be the same.",
"properties": {
"type": {
"type": "string",
"enum": [
"RegexSubstitution"
],
"default": "RegexSubstitution"
},
"fail_dst_not_found": {
"type": "boolean",
"default": true,
"description": "Fail if destination field does not exist. This can be the case if attempting to insert into a nested destination which was not yet created."
},
"pattern": {
"type": "string",
"description": "Regex pattern with match groups. See https://docs.oracle.com/javase/8/docs/api/java/util/regex/Pattern.html"
},
"src_fields": {
"type": "array",
"items": {
"type": "string"
},
"description": "Source fields to pull value from and apply regex to. If multiple fields are provided the first non-null valued one is used."
},
"fields": {
"type": "array",
"items": {
"$ref": "#/definitions/RegexSubField"
},
"description": "List of fields to create from matching regex groups."
},
"remove_src_field": {
"type": "boolean",
"default": false,
"description": "Removes the source field after applying this substitution."
},
"fail_src_not_found": {
"type": "boolean",
"default": true,
"description": "Fail if source fields do not match regex or are not found."
}
},
"title": "RegexSubstitution",
"required": [
"type",
"pattern",
"src_fields",
"fields"
]
}
Substitutes event field value for list of event metadata proprties.
Name of the new field
Fail if destination field does not exist. This can be the case if attempting to insert into a nested destination which was not yet created.
List of metadata fields to add. If non-specified then all are added.
List of metadata fields to exclude. If non-specified then none are excluded.
{
"type": "object",
"additionalProperties": false,
"description": "Substitutes event field value for list of event metadata proprties.",
"properties": {
"type": {
"type": "string",
"enum": [
"MetadataSubstitution"
],
"default": "MetadataSubstitution"
},
"key": {
"type": "string",
"description": "Name of the new field"
},
"fail_dst_not_found": {
"type": "boolean",
"default": true,
"description": "Fail if destination field does not exist. This can be the case if attempting to insert into a nested destination which was not yet created."
},
"includes": {
"type": "array",
"items": {
"type": "string"
},
"description": "List of metadata fields to add. If non-specified then all are added."
},
"excludes": {
"type": "array",
"items": {
"type": "string"
},
"description": "List of metadata fields to exclude. If non-specified then none are excluded."
}
},
"title": "MetadataSubstitution",
"required": [
"type",
"key"
]
}
circular reference
{
"description": "_circular reference_"
}
Creates a new string using variable replacement.
Name of the new field
Fail if destination field does not exist. This can be the case if attempting to insert into a nested destination which was not yet created.
String with variable indices to substitue. See https://docs.oracle.com/javase/8/docs/api/java/text/MessageFormat.html
List of variables used in string substitution. Index of variable relates to index in 'format' string.
Source fields to pull value from. If multiple fields are provided the first non-null valued one is used. Note that if no fields are found the value will be set to null.
Fail if source field is not found.
Removes the source field when performing the substitution.
{
"type": "object",
"additionalProperties": false,
"properties": {
"type": {
"type": "string",
"enum": [
"FieldVariable"
],
"default": "FieldVariable"
},
"src_fields": {
"type": "array",
"items": {
"type": "string"
},
"description": "Source fields to pull value from. If multiple fields are provided the first non-null valued one is used. Note that if no fields are found the value will be set to null."
},
"fail_src_not_found": {
"type": "boolean",
"default": true,
"description": "Fail if source field is not found."
},
"remove_src_field": {
"type": "boolean",
"default": false,
"description": "Removes the source field when performing the substitution."
}
},
"title": "FieldVariable",
"required": [
"type",
"src_fields"
]
}
Value to substitute.
{
"type": "object",
"additionalProperties": false,
"properties": {
"type": {
"type": "string",
"enum": [
"StaticVariable"
],
"default": "StaticVariable"
},
"value": {
"type": "string",
"description": "Value to substitute."
}
},
"title": "StaticVariable",
"required": [
"type",
"value"
]
}
{
"type": "object",
"additionalProperties": false,
"description": "Creates a new string using variable replacement.",
"properties": {
"type": {
"type": "string",
"enum": [
"FormattedSubstitution"
],
"default": "FormattedSubstitution"
},
"key": {
"type": "string",
"description": "Name of the new field"
},
"fail_dst_not_found": {
"type": "boolean",
"default": true,
"description": "Fail if destination field does not exist. This can be the case if attempting to insert into a nested destination which was not yet created."
},
"format": {
"type": "string",
"description": "String with variable indices to substitue. See https://docs.oracle.com/javase/8/docs/api/java/text/MessageFormat.html"
},
"variables": {
"type": "array",
"description": "List of variables used in string substitution. Index of variable relates to index in 'format' string.",
"anyOf": [
{
"$ref": "#/definitions/FieldVariable"
},
{
"$ref": "#/definitions/StaticVariable"
}
]
}
},
"title": "FormattedSubstitution",
"required": [
"type",
"key",
"format",
"variables"
]
}
Substitutes event field value for a static value.
Name of the new field
Fail if destination field does not exist. This can be the case if attempting to insert into a nested destination which was not yet created.
Value of the new field.
{
"type": "object",
"additionalProperties": false,
"description": "Substitutes event field value for a static value.",
"properties": {
"type": {
"type": "string",
"enum": [
"StaticSubstitution"
],
"default": "StaticSubstitution"
},
"key": {
"type": "string",
"description": "Name of the new field"
},
"fail_dst_not_found": {
"type": "boolean",
"default": true,
"description": "Fail if destination field does not exist. This can be the case if attempting to insert into a nested destination which was not yet created."
},
"value": {
"type": "string",
"description": "Value of the new field."
}
},
"title": "StaticSubstitution",
"required": [
"type",
"key",
"value"
]
}
Substitutes event field value for list of lambda function context proprties.
Name of the new field
Fail if destination field does not exist. This can be the case if attempting to insert into a nested destination which was not yet created.
List of context fields to add. If non-specified then all are added.
List of context fields to exclude. If non-specified then none are excluded.
{
"type": "object",
"additionalProperties": false,
"description": "Substitutes event field value for list of lambda function context proprties.",
"properties": {
"type": {
"type": "string",
"enum": [
"ContextSubstitution"
],
"default": "ContextSubstitution"
},
"key": {
"type": "string",
"description": "Name of the new field"
},
"fail_dst_not_found": {
"type": "boolean",
"default": true,
"description": "Fail if destination field does not exist. This can be the case if attempting to insert into a nested destination which was not yet created."
},
"includes": {
"type": "array",
"items": {
"type": "string"
},
"description": "List of context fields to add. If non-specified then all are added."
},
"excludes": {
"type": "array",
"items": {
"type": "string"
},
"description": "List of context fields to exclude. If non-specified then none are excluded."
}
},
"title": "ContextSubstitution",
"required": [
"type",
"key"
]
}
{
"type": "object",
"additionalProperties": false,
"description": "The nested substitution helps build a Map object containing the result of other substitution.",
"properties": {
"type": {
"type": "string",
"enum": [
"NestedSubstitution"
],
"default": "NestedSubstitution"
},
"key": {
"type": "string",
"description": "Name of the new field"
},
"fail_dst_not_found": {
"type": "boolean",
"default": true,
"description": "Fail if destination field does not exist. This can be the case if attempting to insert into a nested destination which was not yet created."
},
"substitutions": {
"type": "array",
"description": "List of substitutions that will build up the final Map object.",
"anyOf": [
{
"$ref": "#/definitions/FieldSubstitutionConfig"
},
{
"$ref": "#/definitions/RegexSubstitutionConfig"
},
{
"$ref": "#/definitions/MetadataSubstitutionConfig"
},
{
"$ref": "#/definitions/NestedSubstitutionConfig"
},
{
"$ref": "#/definitions/FormattedSubstitutionConfig"
},
{
"$ref": "#/definitions/StaticSubstitutionConfig"
},
{
"$ref": "#/definitions/ContextSubstitutionConfig"
}
]
}
},
"title": "NestedSubstitution",
"required": [
"type",
"key"
]
}
Creates a new string using variable replacement.
Name of the new field
Fail if destination field does not exist. This can be the case if attempting to insert into a nested destination which was not yet created.
String with variable indices to substitue. See https://docs.oracle.com/javase/8/docs/api/java/text/MessageFormat.html
List of variables used in string substitution. Index of variable relates to index in 'format' string.
Source fields to pull value from. If multiple fields are provided the first non-null valued one is used. Note that if no fields are found the value will be set to null.
Fail if source field is not found.
Removes the source field when performing the substitution.
{
"type": "object",
"additionalProperties": false,
"properties": {
"type": {
"type": "string",
"enum": [
"FieldVariable"
],
"default": "FieldVariable"
},
"src_fields": {
"type": "array",
"items": {
"type": "string"
},
"description": "Source fields to pull value from. If multiple fields are provided the first non-null valued one is used. Note that if no fields are found the value will be set to null."
},
"fail_src_not_found": {
"type": "boolean",
"default": true,
"description": "Fail if source field is not found."
},
"remove_src_field": {
"type": "boolean",
"default": false,
"description": "Removes the source field when performing the substitution."
}
},
"title": "FieldVariable",
"required": [
"type",
"src_fields"
]
}
Value to substitute.
{
"type": "object",
"additionalProperties": false,
"properties": {
"type": {
"type": "string",
"enum": [
"StaticVariable"
],
"default": "StaticVariable"
},
"value": {
"type": "string",
"description": "Value to substitute."
}
},
"title": "StaticVariable",
"required": [
"type",
"value"
]
}
{
"type": "object",
"additionalProperties": false,
"description": "Creates a new string using variable replacement.",
"properties": {
"type": {
"type": "string",
"enum": [
"FormattedSubstitution"
],
"default": "FormattedSubstitution"
},
"key": {
"type": "string",
"description": "Name of the new field"
},
"fail_dst_not_found": {
"type": "boolean",
"default": true,
"description": "Fail if destination field does not exist. This can be the case if attempting to insert into a nested destination which was not yet created."
},
"format": {
"type": "string",
"description": "String with variable indices to substitue. See https://docs.oracle.com/javase/8/docs/api/java/text/MessageFormat.html"
},
"variables": {
"type": "array",
"description": "List of variables used in string substitution. Index of variable relates to index in 'format' string.",
"anyOf": [
{
"$ref": "#/definitions/FieldVariable"
},
{
"$ref": "#/definitions/StaticVariable"
}
]
}
},
"title": "FormattedSubstitution",
"required": [
"type",
"key",
"format",
"variables"
]
}
Substitutes event field value for a static value.
Name of the new field
Fail if destination field does not exist. This can be the case if attempting to insert into a nested destination which was not yet created.
Value of the new field.
{
"type": "object",
"additionalProperties": false,
"description": "Substitutes event field value for a static value.",
"properties": {
"type": {
"type": "string",
"enum": [
"StaticSubstitution"
],
"default": "StaticSubstitution"
},
"key": {
"type": "string",
"description": "Name of the new field"
},
"fail_dst_not_found": {
"type": "boolean",
"default": true,
"description": "Fail if destination field does not exist. This can be the case if attempting to insert into a nested destination which was not yet created."
},
"value": {
"type": "string",
"description": "Value of the new field."
}
},
"title": "StaticSubstitution",
"required": [
"type",
"key",
"value"
]
}
Substitutes event field value for list of lambda function context proprties.
Name of the new field
Fail if destination field does not exist. This can be the case if attempting to insert into a nested destination which was not yet created.
List of context fields to add. If non-specified then all are added.
List of context fields to exclude. If non-specified then none are excluded.
{
"type": "object",
"additionalProperties": false,
"description": "Substitutes event field value for list of lambda function context proprties.",
"properties": {
"type": {
"type": "string",
"enum": [
"ContextSubstitution"
],
"default": "ContextSubstitution"
},
"key": {
"type": "string",
"description": "Name of the new field"
},
"fail_dst_not_found": {
"type": "boolean",
"default": true,
"description": "Fail if destination field does not exist. This can be the case if attempting to insert into a nested destination which was not yet created."
},
"includes": {
"type": "array",
"items": {
"type": "string"
},
"description": "List of context fields to add. If non-specified then all are added."
},
"excludes": {
"type": "array",
"items": {
"type": "string"
},
"description": "List of context fields to exclude. If non-specified then none are excluded."
}
},
"title": "ContextSubstitution",
"required": [
"type",
"key"
]
}
{
"type": "object",
"additionalProperties": false,
"description": "This operation allows substituting event fields with different sources such as other fields, static values, or metadata.",
"properties": {
"type": {
"type": "string",
"enum": [
"SubstitutionOperation"
],
"default": "SubstitutionOperation"
},
"substitutions": {
"type": "array",
"description": "List of substitutions to perform.",
"anyOf": [
{
"$ref": "#/definitions/FieldSubstitutionConfig"
},
{
"$ref": "#/definitions/RegexSubstitutionConfig"
},
{
"$ref": "#/definitions/MetadataSubstitutionConfig"
},
{
"$ref": "#/definitions/NestedSubstitutionConfig"
},
{
"$ref": "#/definitions/FormattedSubstitutionConfig"
},
{
"$ref": "#/definitions/StaticSubstitutionConfig"
},
{
"$ref": "#/definitions/ContextSubstitutionConfig"
}
]
}
},
"title": "SubstitutionOperation",
"required": [
"type",
"substitutions"
]
}
Sets the partition information for the Event using fields from the deserialized object. When using JSON use JsonPath format to specify fields. See https://github.com/jayway/JsonPath
Configuration to specify object fields that are treated as partitions
Value to use as the key for the partition.
Fields to use for the value of the partition.
Interpreter to use on the partition value.
Java date format to use when using a time based interpreter.
Basic string formatting.
Rounds time down to value prior to formatting. For example 300 would round down to nearest 5 minutes. If set to 0 then rounding is disabled.
{
"type": "object",
"additionalProperties": false,
"properties": {
"name": {
"type": "string",
"description": "Value to use as the key for the partition."
},
"sources": {
"type": "array",
"items": {
"type": "string"
},
"description": "Fields to use for the value of the partition."
},
"interpreter": {
"type": "string",
"default": "STRING",
"enum": [
"STRING",
"MILLISECONDS",
"SECONDS",
"STATIC"
],
"description": "Interpreter to use on the partition value."
},
"format": {
"type": "string",
"description": "Java date format to use when using a time based interpreter."
},
"string_format": {
"type": "string",
"default": "NONE",
"enum": [
"TOLOWER",
"TOUPPER",
"NONE"
],
"description": "Basic string formatting."
},
"seconds_to_round": {
"type": "integer",
"minimum": 0,
"maximum": 86400,
"default": 0,
"description": "Rounds time down to value prior to formatting. For example 300 would round down to nearest 5 minutes. If set to 0 then rounding is disabled."
}
},
"required": [
"name",
"interpreter"
]
}
{
"type": "object",
"additionalProperties": false,
"description": "Sets the partition information for the Event using fields from the deserialized object. When using JSON use JsonPath format to specify fields. See https://github.com/jayway/JsonPath",
"properties": {
"type": {
"type": "string",
"enum": [
"PartitionOperation"
],
"default": "PartitionOperation"
},
"partition_specs": {
"type": "array",
"items": {
"$ref": "#/definitions/PartitionSpec"
},
"description": "Configuration to specify object fields that are treated as partitions"
}
},
"title": "PartitionOperation",
"required": [
"type"
]
}
Provided a path to a JSON array it will produce new events with payloads coorepsonding to elements of the array. For example [{"foo": 1}, {"bar": 2}] will be turned into two seperate events with payloads of {"foo": 1} and {"bar": 2}.
If an array is found and split, this can specify additional fields to keep in the new JSON object to preserve common contexts, such as timestamps, accountId, etc.
{
"type": "object",
"additionalProperties": false,
"description": "Provided a path to a JSON array it will produce new events with payloads coorepsonding to elements of the array. For example [{\"foo\": 1}, {\"bar\": 2}] will be turned into two seperate events with payloads of {\"foo\": 1} and {\"bar\": 2}.",
"properties": {
"type": {
"type": "string",
"enum": [
"JsonArraySplitOperation"
],
"default": "JsonArraySplitOperation"
},
"path": {
"type": "string",
"description": "Path to a JSON node which is an array. See https://github.com/jayway/JsonPath"
},
"fields_to_keep": {
"type": "array",
"items": {
"type": "string"
},
"description": "If an array is found and split, this can specify additional fields to keep in the new JSON object to preserve common contexts, such as timestamps, accountId, etc."
}
},
"title": "JsonArraySplitOperation",
"required": [
"type",
"path"
]
}
Performs a URL decode on the specified fields.
fields
Number of times to perform URL decode on field. This is helpful if a field is encoded multiple times.
{
"type": "object",
"additionalProperties": false,
"description": "Performs a URL decode on the specified fields.",
"properties": {
"type": {
"type": "string",
"enum": [
"UrlDecodeOperation"
],
"default": "UrlDecodeOperation"
},
"fields": {
"type": "array",
"items": {
"type": "string"
},
"description": "fields"
},
"times": {
"type": "integer",
"minimum": 1,
"maximum": 65535,
"default": 1,
"description": "Number of times to perform URL decode on field. This is helpful if a field is encoded multiple times."
}
},
"title": "UrlDecodeOperation",
"required": [
"type",
"fields"
]
}
Deletes the inputted field in the deserialized object. When using JSON use JsonPath format to specify field. See https://github.com/jayway/JsonPath
Name of field key to be deleted.
{
"type": "object",
"additionalProperties": false,
"description": "Deletes the inputted field in the deserialized object. When using JSON use JsonPath format to specify field. See https://github.com/jayway/JsonPath",
"properties": {
"type": {
"type": "string",
"enum": [
"DeleteFieldOperation"
],
"default": "DeleteFieldOperation"
},
"key_field": {
"type": "string",
"description": "Name of field key to be deleted."
}
},
"title": "DeleteFieldOperation",
"required": [
"type",
"key_field"
]
}
Provided a JSON object it will remove any keys which have array values. This is helpful for use with ElasticSearch which indexes arrays in a particular manner not conducive to exploration. See: https://www.elastic.co/guide/en/elasticsearch/guide/current/complex-core-fields.html
{
"type": "object",
"additionalProperties": false,
"description": "Provided a JSON object it will remove any keys which have array values. This is helpful for use with ElasticSearch which indexes arrays in a particular manner not conducive to exploration. See: https://www.elastic.co/guide/en/elasticsearch/guide/current/complex-core-fields.html",
"properties": {
"type": {
"type": "string",
"enum": [
"JsonDropArraysOperation"
],
"default": "JsonDropArraysOperation"
}
},
"title": "JsonDropArraysOperation",
"required": [
"type"
]
}
Sets the Event timestamp using the provided field in the deserialized object. When using JSON use JsonPath format to specify field. See https://github.com/jayway/JsonPath
Name of field to use as time field
How to interpret time field
{
"type": "object",
"additionalProperties": false,
"description": "Sets the Event timestamp using the provided field in the deserialized object. When using JSON use JsonPath format to specify field. See https://github.com/jayway/JsonPath",
"properties": {
"type": {
"type": "string",
"enum": [
"TimeOperation"
],
"default": "TimeOperation"
},
"time_field": {
"type": "string",
"description": "Name of field to use as time field"
},
"time_field_type": {
"type": "string",
"enum": [
"SECONDS",
"MILLISECONDS",
"ISO8601"
],
"description": "How to interpret time field"
}
},
"title": "TimeOperation",
"required": [
"type",
"time_field",
"time_field_type"
]
}
This operation is used to remove certain events from the stream before continuing on to the destination. Each event is assessed by applying a JsonPath to its payload and matching the value against a regex Pattern. If exclude is true, events that match this criteria will be filtered out. If exclude is false, any events not matching this criteria will be filtered out. For example, say these two events are in the stream: {\"data\": \"one\", \"type\": \"bar\"} and {\"data\": \"one\", \"type\": \"baz\"}. With config values: regex = \"(bar)\", path = \"$.type\", and exclude = true, after filtering, the stream will only hold {\"data\": \"one\", \"type\": \"baz\"}. If instead, exclude = false, the stream would only hold {\"data\": \"one\", \"type\": \"bar\"} after filtering.
Regex to be matched against JSON objects. See https://docs.oracle.com/javase/8/docs/api/java/util/regex/Pattern.html
If true, matches will be filtered out. If false, non-matches will be filtered out.
{
"type": "object",
"additionalProperties": false,
"description": "This operation is used to remove certain events from the stream before continuing on to the destination. Each event is assessed by applying a JsonPath to its payload and matching the value against a regex Pattern. If exclude is true, events that match this criteria will be filtered out. If exclude is false, any events not matching this criteria will be filtered out. For example, say these two events are in the stream: {\\\"data\\\": \\\"one\\\", \\\"type\\\": \\\"bar\\\"} and {\\\"data\\\": \\\"one\\\", \\\"type\\\": \\\"baz\\\"}. With config values: regex = \\\"(bar)\\\", path = \\\"$.type\\\", and exclude = true, after filtering, the stream will only hold {\\\"data\\\": \\\"one\\\", \\\"type\\\": \\\"baz\\\"}. If instead, exclude = false, the stream would only hold {\\\"data\\\": \\\"one\\\", \\\"type\\\": \\\"bar\\\"} after filtering.",
"properties": {
"type": {
"type": "string",
"enum": [
"RegexFilterOperation"
],
"default": "RegexFilterOperation"
},
"regex": {
"type": "string",
"description": "Regex to be matched against JSON objects. See https://docs.oracle.com/javase/8/docs/api/java/util/regex/Pattern.html"
},
"path": {
"type": "string",
"description": "JsonPath to the element to be matched against. See https://github.com/jayway/JsonPath"
},
"exclude": {
"type": "boolean",
"default": true,
"description": "If true, matches will be filtered out. If false, non-matches will be filtered out."
}
},
"title": "RegexFilterOperation",
"required": [
"type",
"regex",
"path"
]
}
Looks up geo location provided an IP address and adds a map field to the payload that optionally contains country_name, country_iso_code, subdivision_name, subdivision_iso_code, city_name, postal_code, and location. For example
input = {"ip": "8.8.8.8"}
output = {"ip": "8.8.8.8", "geo_ip": {"location": {"lat": 37.751, "lon": -97.822}}}
Note 1: This operation requires the MaxMind GeoLite2 City Database. It is available at https://dev.maxmind.com/geoip/geoip2/geolite2/. After you download the database, extract the tarball and uploaded the .mmdb file to an S3 bucket accessible by your lambda function.
Note 2: If your ip address field contains a comma separated list of ip addresses (ie forwarded web requests) the first address in the list is used. This is typically the source address.
Field containing an IP address string in the payload
Field name to save the geo data to
S3 path to the GeoLite2 City database file. Prefix with s3://
List of geo properties to include
If ip lookup fails also fail the operation. Doing so will filter out the event.
{
"type": "object",
"additionalProperties": false,
"description": "Looks up geo location provided an IP address and adds a map field to the payload that optionally contains country_name, country_iso_code, subdivision_name, subdivision_iso_code, city_name, postal_code, and location. For example \n\n input = {\"ip\": \"8.8.8.8\"} \n\n output = {\"ip\": \"8.8.8.8\", \"geo_ip\": {\"location\": {\"lat\": 37.751, \"lon\": -97.822}}}\n\n\n\nNote 1: This operation requires the MaxMind GeoLite2 City Database. It is available at https://dev.maxmind.com/geoip/geoip2/geolite2/. After you download the database, extract the tarball and uploaded the .mmdb file to an S3 bucket accessible by your lambda function.\n\nNote 2: If your ip address field contains a comma separated list of ip addresses (ie forwarded web requests) the first address in the list is used. This is typically the source address.",
"properties": {
"type": {
"type": "string",
"enum": [
"GeoIpOperation"
],
"default": "GeoIpOperation"
},
"src_field_name": {
"type": "string",
"description": "Field containing an IP address string in the payload"
},
"dst_field_name": {
"type": "string",
"default": "geo_ip",
"description": "Field name to save the geo data to"
},
"geo_lite_db": {
"type": "string",
"description": "S3 path to the GeoLite2 City database file. Prefix with s3://"
},
"geo_properties": {
"type": "array",
"items": {
"type": "string",
"enum": [
"COUNTRY_NAME",
"COUNTRY_ISO_CODE",
"SUBDIVISION_NAME",
"SUBDIVISION_ISO_CODE",
"CITY_NAME",
"POSTAL_CODE",
"LOCATION"
]
},
"description": "List of geo properties to include"
},
"fail_on_not_found": {
"type": "boolean",
"default": false,
"description": "If ip lookup fails also fail the operation. Doing so will filter out the event."
}
},
"title": "GeoIpOperation",
"required": [
"type",
"src_field_name",
"dst_field_name",
"geo_lite_db"
]
}
Provided a deeply nested JSON Object, it will flatten out the object into keys with a specific separator (dot by default). For example, if the input is {"foo": {"bar": {"baz": 1}}} the operation will produce {"foo.bar.baz": 1} as the new payload.
Separator to be used between nested key names (typically a dot(.))
{
"type": "object",
"additionalProperties": false,
"description": "Provided a deeply nested JSON Object, it will flatten out the object into keys with a specific separator (dot by default). For example, if the input is {\"foo\": {\"bar\": {\"baz\": 1}}} the operation will produce {\"foo.bar.baz\": 1} as the new payload.",
"properties": {
"type": {
"type": "string",
"enum": [
"FlattenOperation"
],
"default": "FlattenOperation"
},
"separator": {
"type": "string",
"default": ".",
"description": "Separator to be used between nested key names (typically a dot(.))"
}
},
"title": "FlattenOperation",
"required": [
"type"
]
}
If pass is set to false then events are filtered out.
If 'false', event is filtered out.
{
"type": "object",
"additionalProperties": false,
"description": "If pass is set to false then events are filtered out.",
"properties": {
"type": {
"type": "string",
"enum": [
"BasicFilterOperation"
],
"default": "BasicFilterOperation"
},
"pass": {
"type": "boolean",
"default": false,
"description": "If 'false', event is filtered out."
}
},
"title": "BasicFilterOperation",
"required": [
"type"
]
}
Uses a regex to find matching parts of a key name and replaces with value or drops the key/value if a match is found. This is typically used to sanitize key names.
If a match is found then replace match with this value. Defaults to emptry string.
If a match is found then drop the key/value.
{
"type": "object",
"additionalProperties": false,
"description": "Uses a regex to find matching parts of a key name and replaces with value or drops the key/value if a match is found. This is typically used to sanitize key names.",
"properties": {
"type": {
"type": "string",
"enum": [
"KeyNameReplacementOperation"
],
"default": "KeyNameReplacementOperation"
},
"regex": {
"type": "string",
"description": "Java regex. See https://docs.oracle.com/javase/8/docs/api/java/util/regex/Pattern.html"
},
"replacement": {
"type": "string",
"default": "",
"description": "If a match is found then replace match with this value. Defaults to emptry string."
},
"drop": {
"type": "boolean",
"default": false,
"description": "If a match is found then drop the key/value."
}
},
"title": "KeyNameReplacementOperation",
"required": [
"type",
"regex"
]
}
circular reference
{
"description": "_circular reference_"
}
Transforms JSON input into a GELF message format (See http://docs.graylog.org/en/2.4/pages/gelf.html details). Note that when selecting source fields use the JsonPath notation https://github.com/json-path/JsonPath.
Field containing host string
Field containing short_message string
Field containing full_message string
Field containing epoch timestamp in seconds with optional ms as decimal. Source field must be in numeric form (not a string) or Graylog will reject the input. Alternatively, use the TimeOperation to detect your timestamp field and properly parse it, and the GelfOperation will automatically use that to set the timestamp field properly. Use this setting only to override that behavior.
Field containing syslog level number
Field containing syslog facility string
Field containing the line in a file that caused the error (decimal); optional, deprecated. Send as additional field instead.
Field containing the file (with path if you want) that caused the error (string); optional, deprecated. Send as additional field instead.
{
"type": "object",
"additionalProperties": false,
"description": "Transforms JSON input into a GELF message format (See http://docs.graylog.org/en/2.4/pages/gelf.html details). Note that when selecting source fields use the JsonPath notation https://github.com/json-path/JsonPath.",
"properties": {
"type": {
"type": "string",
"enum": [
"GelfOperation"
],
"default": "GelfOperation"
},
"src_host_field": {
"type": "array",
"items": {
"type": "string"
},
"description": "Field containing host string"
},
"src_short_message_field": {
"type": "array",
"items": {
"type": "string"
},
"description": "Field containing short_message string"
},
"src_full_message_field": {
"type": "array",
"items": {
"type": "string"
},
"description": "Field containing full_message string"
},
"src_timestamp_field": {
"type": "array",
"items": {
"type": "string"
},
"description": "Field containing epoch timestamp in seconds with optional ms as decimal. Source field must be in numeric form (not a string) or Graylog will reject the input. Alternatively, use the TimeOperation to detect your timestamp field and properly parse it, and the GelfOperation will automatically use that to set the timestamp field properly. Use this setting only to override that behavior."
},
"src_level_field": {
"type": "array",
"items": {
"type": "string"
},
"description": "Field containing syslog level number"
},
"src_facility_field": {
"type": "array",
"items": {
"type": "string"
},
"description": "Field containing syslog facility string"
},
"src_line_number_field": {
"type": "array",
"items": {
"type": "string"
},
"description": "Field containing the line in a file that caused the error (decimal); optional, deprecated. Send as additional field instead."
},
"src_file_field": {
"type": "array",
"items": {
"type": "string"
},
"description": "Field containing the file (with path if you want) that caused the error (string); optional, deprecated. Send as additional field instead."
}
},
"title": "GelfOperation",
"required": [
"type",
"src_host_field",
"src_short_message_field"
]
}
Provided a JSON object it will recursively append the primitive type of the value to the key name. For example {"foo": "one", "bar": 2} will become {"foo__str": "one", "bar__long": 2}. The mapping is string:str, boolean:boolean, array:arr, number:long or __float. It also repalces "." with "_" in key names. This operation is particularily useful for modifying JSON which will be written to ElasticSearch which does not allow conflicting value types for keys or "." in key names.
{
"type": "object",
"additionalProperties": false,
"description": "Provided a JSON object it will recursively append the primitive type of the value to the key name. For example {\"foo\": \"one\", \"bar\": 2} will become {\"foo__str\": \"one\", \"bar__long\": 2}. The mapping is string:__str, boolean:__boolean, array:__arr, number:__long or __float. It also repalces \".\" with \"_\" in key names. This operation is particularily useful for modifying JSON which will be written to ElasticSearch which does not allow conflicting value types for keys or \".\" in key names.",
"properties": {
"type": {
"type": "string",
"enum": [
"JsonKeyNameOperation"
],
"default": "JsonKeyNameOperation"
}
},
"title": "JsonKeyNameOperation",
"required": [
"type"
]
}
{
"type": "object",
"additionalProperties": false,
"properties": {
"operations": {
"type": "array",
"description": "List of operations to perform.",
"anyOf": [
{
"$ref": "#/definitions/ForkOperationConfig"
},
{
"$ref": "#/definitions/LowerCaseKeyOperationConfig"
},
{
"$ref": "#/definitions/JsonRootNodeOperationConfig"
},
{
"$ref": "#/definitions/SubstitutionOperationConfig"
},
{
"$ref": "#/definitions/PartitionOperationConfig"
},
{
"$ref": "#/definitions/ArraySplitOperationConfig"
},
{
"$ref": "#/definitions/URLDecodeOperationConfig"
},
{
"$ref": "#/definitions/DeleteFieldOperationConfig"
},
{
"$ref": "#/definitions/DropArraysOperationConfig"
},
{
"$ref": "#/definitions/TimeOperationConfig"
},
{
"$ref": "#/definitions/RegexFilterOperationConfig"
},
{
"$ref": "#/definitions/GeoIpOperationConfig"
},
{
"$ref": "#/definitions/FlattenOperationConfig"
},
{
"$ref": "#/definitions/BasicFilterOperationConfig"
},
{
"$ref": "#/definitions/KeyNameReplacementOperationConfig"
},
{
"$ref": "#/definitions/ConditionalOperationConfig"
},
{
"$ref": "#/definitions/GelfOperationConfig"
},
{
"$ref": "#/definitions/KeyNameOperationConfig"
}
]
}
}
}
{
"type": "object",
"additionalProperties": false,
"description": "The fork operation allows nesting multiple operation pipelines within Bender. Each fork (pipeline) has its own operations and operates independently on a clone of the original event. If no filters are applied then this can result in multiple output events for each input event.",
"properties": {
"type": {
"type": "string",
"enum": [
"ForkOperation"
],
"default": "ForkOperation"
},
"forks": {
"type": "array",
"items": {
"$ref": "#/definitions/Fork"
},
"description": "List of forks."
}
},
"title": "ForkOperation",
"required": [
"type"
]
}
Provided a JSON object it will recursively lower case all fields.
{
"type": "object",
"additionalProperties": false,
"description": "Provided a JSON object it will recursively lower case all fields.",
"properties": {
"type": {
"type": "string",
"enum": [
"LowerCaseKeyOperation"
],
"default": "LowerCaseKeyOperation"
}
},
"title": "LowerCaseKeyOperation",
"required": [
"type"
]
}
Provided a JSON object and a path within the object it will promote the path's element to the root position. For example if the input is {"foo": {"bar": {"baz": 1}}} and specified path $.foo.bar the operation will produce {"baz": 1} as the new payload.
Path to a JSON node which is promoted to root node. See https://github.com/jayway/JsonPath
{
"type": "object",
"additionalProperties": false,
"description": "Provided a JSON object and a path within the object it will promote the path's element to the root position. For example if the input is {\"foo\": {\"bar\": {\"baz\": 1}}} and specified path $.foo.bar the operation will produce {\"baz\": 1} as the new payload.",
"properties": {
"type": {
"type": "string",
"enum": [
"JsonRootNodeOperation"
],
"default": "JsonRootNodeOperation"
},
"root_path": {
"type": "string",
"description": "Path to a JSON node which is promoted to root node. See https://github.com/jayway/JsonPath"
}
},
"title": "JsonRootNodeOperation",
"required": [
"type",
"root_path"
]
}
This operation allows substituting event fields with different sources such as other fields, static values, or metadata.
List of substitutions to perform.
Substitutes event field value for another event field value. Note the source field and destination field can be the same.
Name of the new field
Fail if destination field does not exist. This can be the case if attempting to insert into a nested destination which was not yet created.
Source fields to pull value from. If multiple fields are provided the first non-null valued one is used. Note that if no fields are found the value will be set to null.
Removes the source field when performing the substitution. Effectively making this a move operation.
Fail if source field is not found.
{
"type": "object",
"additionalProperties": false,
"description": "Substitutes event field value for another event field value. Note the source field and destination field can be the same.",
"properties": {
"type": {
"type": "string",
"enum": [
"FieldSubstitution"
],
"default": "FieldSubstitution"
},
"key": {
"type": "string",
"description": "Name of the new field"
},
"fail_dst_not_found": {
"type": "boolean",
"default": true,
"description": "Fail if destination field does not exist. This can be the case if attempting to insert into a nested destination which was not yet created."
},
"src_fields": {
"type": "array",
"items": {
"type": "string"
},
"description": "Source fields to pull value from. If multiple fields are provided the first non-null valued one is used. Note that if no fields are found the value will be set to null."
},
"remove_src_field": {
"type": "boolean",
"default": false,
"description": "Removes the source field when performing the substitution. Effectively making this a move operation."
},
"fail_src_not_found": {
"type": "boolean",
"default": true,
"description": "Fail if source field is not found."
}
},
"title": "FieldSubstitution",
"required": [
"type",
"key",
"src_fields"
]
}
Substitutes event field value for another event field value. Note the source field and destination field can be the same.
Fail if destination field does not exist. This can be the case if attempting to insert into a nested destination which was not yet created.
Regex pattern with match groups. See https://docs.oracle.com/javase/8/docs/api/java/util/regex/Pattern.html
Source fields to pull value from and apply regex to. If multiple fields are provided the first non-null valued one is used.
List of fields to create from matching regex groups.
Regex group name identifying the field.
Data type of match group field. If type coercion does not succeed then field is ignored.
Name or path of the new field.
{
"type": "object",
"additionalProperties": false,
"properties": {
"regex_group_name": {
"type": "string",
"description": "Regex group name identifying the field."
},
"type": {
"type": "string",
"enum": [
"STRING",
"NUMBER",
"BOOLEAN"
],
"description": "Data type of match group field. If type coercion does not succeed then field is ignored."
},
"key": {
"type": "string",
"description": "Name or path of the new field."
}
},
"required": [
"regex_group_name",
"type",
"key"
]
}
Removes the source field after applying this substitution.
Fail if source fields do not match regex or are not found.
{
"type": "object",
"additionalProperties": false,
"description": "Substitutes event field value for another event field value. Note the source field and destination field can be the same.",
"properties": {
"type": {
"type": "string",
"enum": [
"RegexSubstitution"
],
"default": "RegexSubstitution"
},
"fail_dst_not_found": {
"type": "boolean",
"default": true,
"description": "Fail if destination field does not exist. This can be the case if attempting to insert into a nested destination which was not yet created."
},
"pattern": {
"type": "string",
"description": "Regex pattern with match groups. See https://docs.oracle.com/javase/8/docs/api/java/util/regex/Pattern.html"
},
"src_fields": {
"type": "array",
"items": {
"type": "string"
},
"description": "Source fields to pull value from and apply regex to. If multiple fields are provided the first non-null valued one is used."
},
"fields": {
"type": "array",
"items": {
"$ref": "#/definitions/RegexSubField"
},
"description": "List of fields to create from matching regex groups."
},
"remove_src_field": {
"type": "boolean",
"default": false,
"description": "Removes the source field after applying this substitution."
},
"fail_src_not_found": {
"type": "boolean",
"default": true,
"description": "Fail if source fields do not match regex or are not found."
}
},
"title": "RegexSubstitution",
"required": [
"type",
"pattern",
"src_fields",
"fields"
]
}
Substitutes event field value for list of event metadata proprties.
Name of the new field
Fail if destination field does not exist. This can be the case if attempting to insert into a nested destination which was not yet created.
List of metadata fields to add. If non-specified then all are added.
List of metadata fields to exclude. If non-specified then none are excluded.
{
"type": "object",
"additionalProperties": false,
"description": "Substitutes event field value for list of event metadata proprties.",
"properties": {
"type": {
"type": "string",
"enum": [
"MetadataSubstitution"
],
"default": "MetadataSubstitution"
},
"key": {
"type": "string",
"description": "Name of the new field"
},
"fail_dst_not_found": {
"type": "boolean",
"default": true,
"description": "Fail if destination field does not exist. This can be the case if attempting to insert into a nested destination which was not yet created."
},
"includes": {
"type": "array",
"items": {
"type": "string"
},
"description": "List of metadata fields to add. If non-specified then all are added."
},
"excludes": {
"type": "array",
"items": {
"type": "string"
},
"description": "List of metadata fields to exclude. If non-specified then none are excluded."
}
},
"title": "MetadataSubstitution",
"required": [
"type",
"key"
]
}
The nested substitution helps build a Map object containing the result of other substitution.
Name of the new field
Fail if destination field does not exist. This can be the case if attempting to insert into a nested destination which was not yet created.
List of substitutions that will build up the final Map object.
Substitutes event field value for another event field value. Note the source field and destination field can be the same.
Name of the new field
Fail if destination field does not exist. This can be the case if attempting to insert into a nested destination which was not yet created.
Source fields to pull value from. If multiple fields are provided the first non-null valued one is used. Note that if no fields are found the value will be set to null.
Removes the source field when performing the substitution. Effectively making this a move operation.
Fail if source field is not found.
{
"type": "object",
"additionalProperties": false,
"description": "Substitutes event field value for another event field value. Note the source field and destination field can be the same.",
"properties": {
"type": {
"type": "string",
"enum": [
"FieldSubstitution"
],
"default": "FieldSubstitution"
},
"key": {
"type": "string",
"description": "Name of the new field"
},
"fail_dst_not_found": {
"type": "boolean",
"default": true,
"description": "Fail if destination field does not exist. This can be the case if attempting to insert into a nested destination which was not yet created."
},
"src_fields": {
"type": "array",
"items": {
"type": "string"
},
"description": "Source fields to pull value from. If multiple fields are provided the first non-null valued one is used. Note that if no fields are found the value will be set to null."
},
"remove_src_field": {
"type": "boolean",
"default": false,
"description": "Removes the source field when performing the substitution. Effectively making this a move operation."
},
"fail_src_not_found": {
"type": "boolean",
"default": true,
"description": "Fail if source field is not found."
}
},
"title": "FieldSubstitution",
"required": [
"type",
"key",
"src_fields"
]
}
Substitutes event field value for another event field value. Note the source field and destination field can be the same.
Fail if destination field does not exist. This can be the case if attempting to insert into a nested destination which was not yet created.
Regex pattern with match groups. See https://docs.oracle.com/javase/8/docs/api/java/util/regex/Pattern.html
Source fields to pull value from and apply regex to. If multiple fields are provided the first non-null valued one is used.
List of fields to create from matching regex groups.
Regex group name identifying the field.
Data type of match group field. If type coercion does not succeed then field is ignored.
Name or path of the new field.
{
"type": "object",
"additionalProperties": false,
"properties": {
"regex_group_name": {
"type": "string",
"description": "Regex group name identifying the field."
},
"type": {
"type": "string",
"enum": [
"STRING",
"NUMBER",
"BOOLEAN"
],
"description": "Data type of match group field. If type coercion does not succeed then field is ignored."
},
"key": {
"type": "string",
"description": "Name or path of the new field."
}
},
"required": [
"regex_group_name",
"type",
"key"
]
}
Removes the source field after applying this substitution.
Fail if source fields do not match regex or are not found.
{
"type": "object",
"additionalProperties": false,
"description": "Substitutes event field value for another event field value. Note the source field and destination field can be the same.",
"properties": {
"type": {
"type": "string",
"enum": [
"RegexSubstitution"
],
"default": "RegexSubstitution"
},
"fail_dst_not_found": {
"type": "boolean",
"default": true,
"description": "Fail if destination field does not exist. This can be the case if attempting to insert into a nested destination which was not yet created."
},
"pattern": {
"type": "string",
"description": "Regex pattern with match groups. See https://docs.oracle.com/javase/8/docs/api/java/util/regex/Pattern.html"
},
"src_fields": {
"type": "array",
"items": {
"type": "string"
},
"description": "Source fields to pull value from and apply regex to. If multiple fields are provided the first non-null valued one is used."
},
"fields": {
"type": "array",
"items": {
"$ref": "#/definitions/RegexSubField"
},
"description": "List of fields to create from matching regex groups."
},
"remove_src_field": {
"type": "boolean",
"default": false,
"description": "Removes the source field after applying this substitution."
},
"fail_src_not_found": {
"type": "boolean",
"default": true,
"description": "Fail if source fields do not match regex or are not found."
}
},
"title": "RegexSubstitution",
"required": [
"type",
"pattern",
"src_fields",
"fields"
]
}
Substitutes event field value for list of event metadata proprties.
Name of the new field
Fail if destination field does not exist. This can be the case if attempting to insert into a nested destination which was not yet created.
List of metadata fields to add. If non-specified then all are added.
List of metadata fields to exclude. If non-specified then none are excluded.
{
"type": "object",
"additionalProperties": false,
"description": "Substitutes event field value for list of event metadata proprties.",
"properties": {
"type": {
"type": "string",
"enum": [
"MetadataSubstitution"
],
"default": "MetadataSubstitution"
},
"key": {
"type": "string",
"description": "Name of the new field"
},
"fail_dst_not_found": {
"type": "boolean",
"default": true,
"description": "Fail if destination field does not exist. This can be the case if attempting to insert into a nested destination which was not yet created."
},
"includes": {
"type": "array",
"items": {
"type": "string"
},
"description": "List of metadata fields to add. If non-specified then all are added."
},
"excludes": {
"type": "array",
"items": {
"type": "string"
},
"description": "List of metadata fields to exclude. If non-specified then none are excluded."
}
},
"title": "MetadataSubstitution",
"required": [
"type",
"key"
]
}
circular reference
{
"description": "_circular reference_"
}
Creates a new string using variable replacement.
Name of the new field
Fail if destination field does not exist. This can be the case if attempting to insert into a nested destination which was not yet created.
String with variable indices to substitue. See https://docs.oracle.com/javase/8/docs/api/java/text/MessageFormat.html
List of variables used in string substitution. Index of variable relates to index in 'format' string.
Source fields to pull value from. If multiple fields are provided the first non-null valued one is used. Note that if no fields are found the value will be set to null.
Fail if source field is not found.
Removes the source field when performing the substitution.
{
"type": "object",
"additionalProperties": false,
"properties": {
"type": {
"type": "string",
"enum": [
"FieldVariable"
],
"default": "FieldVariable"
},
"src_fields": {
"type": "array",
"items": {
"type": "string"
},
"description": "Source fields to pull value from. If multiple fields are provided the first non-null valued one is used. Note that if no fields are found the value will be set to null."
},
"fail_src_not_found": {
"type": "boolean",
"default": true,
"description": "Fail if source field is not found."
},
"remove_src_field": {
"type": "boolean",
"default": false,
"description": "Removes the source field when performing the substitution."
}
},
"title": "FieldVariable",
"required": [
"type",
"src_fields"
]
}
Value to substitute.
{
"type": "object",
"additionalProperties": false,
"properties": {
"type": {
"type": "string",
"enum": [
"StaticVariable"
],
"default": "StaticVariable"
},
"value": {
"type": "string",
"description": "Value to substitute."
}
},
"title": "StaticVariable",
"required": [
"type",
"value"
]
}
{
"type": "object",
"additionalProperties": false,
"description": "Creates a new string using variable replacement.",
"properties": {
"type": {
"type": "string",
"enum": [
"FormattedSubstitution"
],
"default": "FormattedSubstitution"
},
"key": {
"type": "string",
"description": "Name of the new field"
},
"fail_dst_not_found": {
"type": "boolean",
"default": true,
"description": "Fail if destination field does not exist. This can be the case if attempting to insert into a nested destination which was not yet created."
},
"format": {
"type": "string",
"description": "String with variable indices to substitue. See https://docs.oracle.com/javase/8/docs/api/java/text/MessageFormat.html"
},
"variables": {
"type": "array",
"description": "List of variables used in string substitution. Index of variable relates to index in 'format' string.",
"anyOf": [
{
"$ref": "#/definitions/FieldVariable"
},
{
"$ref": "#/definitions/StaticVariable"
}
]
}
},
"title": "FormattedSubstitution",
"required": [
"type",
"key",
"format",
"variables"
]
}
Substitutes event field value for a static value.
Name of the new field
Fail if destination field does not exist. This can be the case if attempting to insert into a nested destination which was not yet created.
Value of the new field.
{
"type": "object",
"additionalProperties": false,
"description": "Substitutes event field value for a static value.",
"properties": {
"type": {
"type": "string",
"enum": [
"StaticSubstitution"
],
"default": "StaticSubstitution"
},
"key": {
"type": "string",
"description": "Name of the new field"
},
"fail_dst_not_found": {
"type": "boolean",
"default": true,
"description": "Fail if destination field does not exist. This can be the case if attempting to insert into a nested destination which was not yet created."
},
"value": {
"type": "string",
"description": "Value of the new field."
}
},
"title": "StaticSubstitution",
"required": [
"type",
"key",
"value"
]
}
Substitutes event field value for list of lambda function context proprties.
Name of the new field
Fail if destination field does not exist. This can be the case if attempting to insert into a nested destination which was not yet created.
List of context fields to add. If non-specified then all are added.
List of context fields to exclude. If non-specified then none are excluded.
{
"type": "object",
"additionalProperties": false,
"description": "Substitutes event field value for list of lambda function context proprties.",
"properties": {
"type": {
"type": "string",
"enum": [
"ContextSubstitution"
],
"default": "ContextSubstitution"
},
"key": {
"type": "string",
"description": "Name of the new field"
},
"fail_dst_not_found": {
"type": "boolean",
"default": true,
"description": "Fail if destination field does not exist. This can be the case if attempting to insert into a nested destination which was not yet created."
},
"includes": {
"type": "array",
"items": {
"type": "string"
},
"description": "List of context fields to add. If non-specified then all are added."
},
"excludes": {
"type": "array",
"items": {
"type": "string"
},
"description": "List of context fields to exclude. If non-specified then none are excluded."
}
},
"title": "ContextSubstitution",
"required": [
"type",
"key"
]
}
{
"type": "object",
"additionalProperties": false,
"description": "The nested substitution helps build a Map object containing the result of other substitution.",
"properties": {
"type": {
"type": "string",
"enum": [
"NestedSubstitution"
],
"default": "NestedSubstitution"
},
"key": {
"type": "string",
"description": "Name of the new field"
},
"fail_dst_not_found": {
"type": "boolean",
"default": true,
"description": "Fail if destination field does not exist. This can be the case if attempting to insert into a nested destination which was not yet created."
},
"substitutions": {
"type": "array",
"description": "List of substitutions that will build up the final Map object.",
"anyOf": [
{
"$ref": "#/definitions/FieldSubstitutionConfig"
},
{
"$ref": "#/definitions/RegexSubstitutionConfig"
},
{
"$ref": "#/definitions/MetadataSubstitutionConfig"
},
{
"$ref": "#/definitions/NestedSubstitutionConfig"
},
{
"$ref": "#/definitions/FormattedSubstitutionConfig"
},
{
"$ref": "#/definitions/StaticSubstitutionConfig"
},
{
"$ref": "#/definitions/ContextSubstitutionConfig"
}
]
}
},
"title": "NestedSubstitution",
"required": [
"type",
"key"
]
}
Creates a new string using variable replacement.
Name of the new field
Fail if destination field does not exist. This can be the case if attempting to insert into a nested destination which was not yet created.
String with variable indices to substitue. See https://docs.oracle.com/javase/8/docs/api/java/text/MessageFormat.html
List of variables used in string substitution. Index of variable relates to index in 'format' string.
Source fields to pull value from. If multiple fields are provided the first non-null valued one is used. Note that if no fields are found the value will be set to null.
Fail if source field is not found.
Removes the source field when performing the substitution.
{
"type": "object",
"additionalProperties": false,
"properties": {
"type": {
"type": "string",
"enum": [
"FieldVariable"
],
"default": "FieldVariable"
},
"src_fields": {
"type": "array",
"items": {
"type": "string"
},
"description": "Source fields to pull value from. If multiple fields are provided the first non-null valued one is used. Note that if no fields are found the value will be set to null."
},
"fail_src_not_found": {
"type": "boolean",
"default": true,
"description": "Fail if source field is not found."
},
"remove_src_field": {
"type": "boolean",
"default": false,
"description": "Removes the source field when performing the substitution."
}
},
"title": "FieldVariable",
"required": [
"type",
"src_fields"
]
}
Value to substitute.
{
"type": "object",
"additionalProperties": false,
"properties": {
"type": {
"type": "string",
"enum": [
"StaticVariable"
],
"default": "StaticVariable"
},
"value": {
"type": "string",
"description": "Value to substitute."
}
},
"title": "StaticVariable",
"required": [
"type",
"value"
]
}
{
"type": "object",
"additionalProperties": false,
"description": "Creates a new string using variable replacement.",
"properties": {
"type": {
"type": "string",
"enum": [
"FormattedSubstitution"
],
"default": "FormattedSubstitution"
},
"key": {
"type": "string",
"description": "Name of the new field"
},
"fail_dst_not_found": {
"type": "boolean",
"default": true,
"description": "Fail if destination field does not exist. This can be the case if attempting to insert into a nested destination which was not yet created."
},
"format": {
"type": "string",
"description": "String with variable indices to substitue. See https://docs.oracle.com/javase/8/docs/api/java/text/MessageFormat.html"
},
"variables": {
"type": "array",
"description": "List of variables used in string substitution. Index of variable relates to index in 'format' string.",
"anyOf": [
{
"$ref": "#/definitions/FieldVariable"
},
{
"$ref": "#/definitions/StaticVariable"
}
]
}
},
"title": "FormattedSubstitution",
"required": [
"type",
"key",
"format",
"variables"
]
}
Substitutes event field value for a static value.
Name of the new field
Fail if destination field does not exist. This can be the case if attempting to insert into a nested destination which was not yet created.
Value of the new field.
{
"type": "object",
"additionalProperties": false,
"description": "Substitutes event field value for a static value.",
"properties": {
"type": {
"type": "string",
"enum": [
"StaticSubstitution"
],
"default": "StaticSubstitution"
},
"key": {
"type": "string",
"description": "Name of the new field"
},
"fail_dst_not_found": {
"type": "boolean",
"default": true,
"description": "Fail if destination field does not exist. This can be the case if attempting to insert into a nested destination which was not yet created."
},
"value": {
"type": "string",
"description": "Value of the new field."
}
},
"title": "StaticSubstitution",
"required": [
"type",
"key",
"value"
]
}
Substitutes event field value for list of lambda function context proprties.
Name of the new field
Fail if destination field does not exist. This can be the case if attempting to insert into a nested destination which was not yet created.
List of context fields to add. If non-specified then all are added.
List of context fields to exclude. If non-specified then none are excluded.
{
"type": "object",
"additionalProperties": false,
"description": "Substitutes event field value for list of lambda function context proprties.",
"properties": {
"type": {
"type": "string",
"enum": [
"ContextSubstitution"
],
"default": "ContextSubstitution"
},
"key": {
"type": "string",
"description": "Name of the new field"
},
"fail_dst_not_found": {
"type": "boolean",
"default": true,
"description": "Fail if destination field does not exist. This can be the case if attempting to insert into a nested destination which was not yet created."
},
"includes": {
"type": "array",
"items": {
"type": "string"
},
"description": "List of context fields to add. If non-specified then all are added."
},
"excludes": {
"type": "array",
"items": {
"type": "string"
},
"description": "List of context fields to exclude. If non-specified then none are excluded."
}
},
"title": "ContextSubstitution",
"required": [
"type",
"key"
]
}
{
"type": "object",
"additionalProperties": false,
"description": "This operation allows substituting event fields with different sources such as other fields, static values, or metadata.",
"properties": {
"type": {
"type": "string",
"enum": [
"SubstitutionOperation"
],
"default": "SubstitutionOperation"
},
"substitutions": {
"type": "array",
"description": "List of substitutions to perform.",
"anyOf": [
{
"$ref": "#/definitions/FieldSubstitutionConfig"
},
{
"$ref": "#/definitions/RegexSubstitutionConfig"
},
{
"$ref": "#/definitions/MetadataSubstitutionConfig"
},
{
"$ref": "#/definitions/NestedSubstitutionConfig"
},
{
"$ref": "#/definitions/FormattedSubstitutionConfig"
},
{
"$ref": "#/definitions/StaticSubstitutionConfig"
},
{
"$ref": "#/definitions/ContextSubstitutionConfig"
}
]
}
},
"title": "SubstitutionOperation",
"required": [
"type",
"substitutions"
]
}
Sets the partition information for the Event using fields from the deserialized object. When using JSON use JsonPath format to specify fields. See https://github.com/jayway/JsonPath
Configuration to specify object fields that are treated as partitions
Value to use as the key for the partition.
Fields to use for the value of the partition.
Interpreter to use on the partition value.
Java date format to use when using a time based interpreter.
Basic string formatting.
Rounds time down to value prior to formatting. For example 300 would round down to nearest 5 minutes. If set to 0 then rounding is disabled.
{
"type": "object",
"additionalProperties": false,
"properties": {
"name": {
"type": "string",
"description": "Value to use as the key for the partition."
},
"sources": {
"type": "array",
"items": {
"type": "string"
},
"description": "Fields to use for the value of the partition."
},
"interpreter": {
"type": "string",
"default": "STRING",
"enum": [
"STRING",
"MILLISECONDS",
"SECONDS",
"STATIC"
],
"description": "Interpreter to use on the partition value."
},
"format": {
"type": "string",
"description": "Java date format to use when using a time based interpreter."
},
"string_format": {
"type": "string",
"default": "NONE",
"enum": [
"TOLOWER",
"TOUPPER",
"NONE"
],
"description": "Basic string formatting."
},
"seconds_to_round": {
"type": "integer",
"minimum": 0,
"maximum": 86400,
"default": 0,
"description": "Rounds time down to value prior to formatting. For example 300 would round down to nearest 5 minutes. If set to 0 then rounding is disabled."
}
},
"required": [
"name",
"interpreter"
]
}
{
"type": "object",
"additionalProperties": false,
"description": "Sets the partition information for the Event using fields from the deserialized object. When using JSON use JsonPath format to specify fields. See https://github.com/jayway/JsonPath",
"properties": {
"type": {
"type": "string",
"enum": [
"PartitionOperation"
],
"default": "PartitionOperation"
},
"partition_specs": {
"type": "array",
"items": {
"$ref": "#/definitions/PartitionSpec"
},
"description": "Configuration to specify object fields that are treated as partitions"
}
},
"title": "PartitionOperation",
"required": [
"type"
]
}
Provided a path to a JSON array it will produce new events with payloads coorepsonding to elements of the array. For example [{"foo": 1}, {"bar": 2}] will be turned into two seperate events with payloads of {"foo": 1} and {"bar": 2}.
If an array is found and split, this can specify additional fields to keep in the new JSON object to preserve common contexts, such as timestamps, accountId, etc.
{
"type": "object",
"additionalProperties": false,
"description": "Provided a path to a JSON array it will produce new events with payloads coorepsonding to elements of the array. For example [{\"foo\": 1}, {\"bar\": 2}] will be turned into two seperate events with payloads of {\"foo\": 1} and {\"bar\": 2}.",
"properties": {
"type": {
"type": "string",
"enum": [
"JsonArraySplitOperation"
],
"default": "JsonArraySplitOperation"
},
"path": {
"type": "string",
"description": "Path to a JSON node which is an array. See https://github.com/jayway/JsonPath"
},
"fields_to_keep": {
"type": "array",
"items": {
"type": "string"
},
"description": "If an array is found and split, this can specify additional fields to keep in the new JSON object to preserve common contexts, such as timestamps, accountId, etc."
}
},
"title": "JsonArraySplitOperation",
"required": [
"type",
"path"
]
}
Performs a URL decode on the specified fields.
fields
Number of times to perform URL decode on field. This is helpful if a field is encoded multiple times.
{
"type": "object",
"additionalProperties": false,
"description": "Performs a URL decode on the specified fields.",
"properties": {
"type": {
"type": "string",
"enum": [
"UrlDecodeOperation"
],
"default": "UrlDecodeOperation"
},
"fields": {
"type": "array",
"items": {
"type": "string"
},
"description": "fields"
},
"times": {
"type": "integer",
"minimum": 1,
"maximum": 65535,
"default": 1,
"description": "Number of times to perform URL decode on field. This is helpful if a field is encoded multiple times."
}
},
"title": "UrlDecodeOperation",
"required": [
"type",
"fields"
]
}
Deletes the inputted field in the deserialized object. When using JSON use JsonPath format to specify field. See https://github.com/jayway/JsonPath
Name of field key to be deleted.
{
"type": "object",
"additionalProperties": false,
"description": "Deletes the inputted field in the deserialized object. When using JSON use JsonPath format to specify field. See https://github.com/jayway/JsonPath",
"properties": {
"type": {
"type": "string",
"enum": [
"DeleteFieldOperation"
],
"default": "DeleteFieldOperation"
},
"key_field": {
"type": "string",
"description": "Name of field key to be deleted."
}
},
"title": "DeleteFieldOperation",
"required": [
"type",
"key_field"
]
}
Provided a JSON object it will remove any keys which have array values. This is helpful for use with ElasticSearch which indexes arrays in a particular manner not conducive to exploration. See: https://www.elastic.co/guide/en/elasticsearch/guide/current/complex-core-fields.html
{
"type": "object",
"additionalProperties": false,
"description": "Provided a JSON object it will remove any keys which have array values. This is helpful for use with ElasticSearch which indexes arrays in a particular manner not conducive to exploration. See: https://www.elastic.co/guide/en/elasticsearch/guide/current/complex-core-fields.html",
"properties": {
"type": {
"type": "string",
"enum": [
"JsonDropArraysOperation"
],
"default": "JsonDropArraysOperation"
}
},
"title": "JsonDropArraysOperation",
"required": [
"type"
]
}
Sets the Event timestamp using the provided field in the deserialized object. When using JSON use JsonPath format to specify field. See https://github.com/jayway/JsonPath
Name of field to use as time field
How to interpret time field
{
"type": "object",
"additionalProperties": false,
"description": "Sets the Event timestamp using the provided field in the deserialized object. When using JSON use JsonPath format to specify field. See https://github.com/jayway/JsonPath",
"properties": {
"type": {
"type": "string",
"enum": [
"TimeOperation"
],
"default": "TimeOperation"
},
"time_field": {
"type": "string",
"description": "Name of field to use as time field"
},
"time_field_type": {
"type": "string",
"enum": [
"SECONDS",
"MILLISECONDS",
"ISO8601"
],
"description": "How to interpret time field"
}
},
"title": "TimeOperation",
"required": [
"type",
"time_field",
"time_field_type"
]
}
This operation is used to remove certain events from the stream before continuing on to the destination. Each event is assessed by applying a JsonPath to its payload and matching the value against a regex Pattern. If exclude is true, events that match this criteria will be filtered out. If exclude is false, any events not matching this criteria will be filtered out. For example, say these two events are in the stream: {\"data\": \"one\", \"type\": \"bar\"} and {\"data\": \"one\", \"type\": \"baz\"}. With config values: regex = \"(bar)\", path = \"$.type\", and exclude = true, after filtering, the stream will only hold {\"data\": \"one\", \"type\": \"baz\"}. If instead, exclude = false, the stream would only hold {\"data\": \"one\", \"type\": \"bar\"} after filtering.
Regex to be matched against JSON objects. See https://docs.oracle.com/javase/8/docs/api/java/util/regex/Pattern.html
If true, matches will be filtered out. If false, non-matches will be filtered out.
{
"type": "object",
"additionalProperties": false,
"description": "This operation is used to remove certain events from the stream before continuing on to the destination. Each event is assessed by applying a JsonPath to its payload and matching the value against a regex Pattern. If exclude is true, events that match this criteria will be filtered out. If exclude is false, any events not matching this criteria will be filtered out. For example, say these two events are in the stream: {\\\"data\\\": \\\"one\\\", \\\"type\\\": \\\"bar\\\"} and {\\\"data\\\": \\\"one\\\", \\\"type\\\": \\\"baz\\\"}. With config values: regex = \\\"(bar)\\\", path = \\\"$.type\\\", and exclude = true, after filtering, the stream will only hold {\\\"data\\\": \\\"one\\\", \\\"type\\\": \\\"baz\\\"}. If instead, exclude = false, the stream would only hold {\\\"data\\\": \\\"one\\\", \\\"type\\\": \\\"bar\\\"} after filtering.",
"properties": {
"type": {
"type": "string",
"enum": [
"RegexFilterOperation"
],
"default": "RegexFilterOperation"
},
"regex": {
"type": "string",
"description": "Regex to be matched against JSON objects. See https://docs.oracle.com/javase/8/docs/api/java/util/regex/Pattern.html"
},
"path": {
"type": "string",
"description": "JsonPath to the element to be matched against. See https://github.com/jayway/JsonPath"
},
"exclude": {
"type": "boolean",
"default": true,
"description": "If true, matches will be filtered out. If false, non-matches will be filtered out."
}
},
"title": "RegexFilterOperation",
"required": [
"type",
"regex",
"path"
]
}
Looks up geo location provided an IP address and adds a map field to the payload that optionally contains country_name, country_iso_code, subdivision_name, subdivision_iso_code, city_name, postal_code, and location. For example
input = {"ip": "8.8.8.8"}
output = {"ip": "8.8.8.8", "geo_ip": {"location": {"lat": 37.751, "lon": -97.822}}}
Note 1: This operation requires the MaxMind GeoLite2 City Database. It is available at https://dev.maxmind.com/geoip/geoip2/geolite2/. After you download the database, extract the tarball and uploaded the .mmdb file to an S3 bucket accessible by your lambda function.
Note 2: If your ip address field contains a comma separated list of ip addresses (ie forwarded web requests) the first address in the list is used. This is typically the source address.
Field containing an IP address string in the payload
Field name to save the geo data to
S3 path to the GeoLite2 City database file. Prefix with s3://
List of geo properties to include
If ip lookup fails also fail the operation. Doing so will filter out the event.
{
"type": "object",
"additionalProperties": false,
"description": "Looks up geo location provided an IP address and adds a map field to the payload that optionally contains country_name, country_iso_code, subdivision_name, subdivision_iso_code, city_name, postal_code, and location. For example \n\n input = {\"ip\": \"8.8.8.8\"} \n\n output = {\"ip\": \"8.8.8.8\", \"geo_ip\": {\"location\": {\"lat\": 37.751, \"lon\": -97.822}}}\n\n\n\nNote 1: This operation requires the MaxMind GeoLite2 City Database. It is available at https://dev.maxmind.com/geoip/geoip2/geolite2/. After you download the database, extract the tarball and uploaded the .mmdb file to an S3 bucket accessible by your lambda function.\n\nNote 2: If your ip address field contains a comma separated list of ip addresses (ie forwarded web requests) the first address in the list is used. This is typically the source address.",
"properties": {
"type": {
"type": "string",
"enum": [
"GeoIpOperation"
],
"default": "GeoIpOperation"
},
"src_field_name": {
"type": "string",
"description": "Field containing an IP address string in the payload"
},
"dst_field_name": {
"type": "string",
"default": "geo_ip",
"description": "Field name to save the geo data to"
},
"geo_lite_db": {
"type": "string",
"description": "S3 path to the GeoLite2 City database file. Prefix with s3://"
},
"geo_properties": {
"type": "array",
"items": {
"type": "string",
"enum": [
"COUNTRY_NAME",
"COUNTRY_ISO_CODE",
"SUBDIVISION_NAME",
"SUBDIVISION_ISO_CODE",
"CITY_NAME",
"POSTAL_CODE",
"LOCATION"
]
},
"description": "List of geo properties to include"
},
"fail_on_not_found": {
"type": "boolean",
"default": false,
"description": "If ip lookup fails also fail the operation. Doing so will filter out the event."
}
},
"title": "GeoIpOperation",
"required": [
"type",
"src_field_name",
"dst_field_name",
"geo_lite_db"
]
}
Provided a deeply nested JSON Object, it will flatten out the object into keys with a specific separator (dot by default). For example, if the input is {"foo": {"bar": {"baz": 1}}} the operation will produce {"foo.bar.baz": 1} as the new payload.
Separator to be used between nested key names (typically a dot(.))
{
"type": "object",
"additionalProperties": false,
"description": "Provided a deeply nested JSON Object, it will flatten out the object into keys with a specific separator (dot by default). For example, if the input is {\"foo\": {\"bar\": {\"baz\": 1}}} the operation will produce {\"foo.bar.baz\": 1} as the new payload.",
"properties": {
"type": {
"type": "string",
"enum": [
"FlattenOperation"
],
"default": "FlattenOperation"
},
"separator": {
"type": "string",
"default": ".",
"description": "Separator to be used between nested key names (typically a dot(.))"
}
},
"title": "FlattenOperation",
"required": [
"type"
]
}
If pass is set to false then events are filtered out.
If 'false', event is filtered out.
{
"type": "object",
"additionalProperties": false,
"description": "If pass is set to false then events are filtered out.",
"properties": {
"type": {
"type": "string",
"enum": [
"BasicFilterOperation"
],
"default": "BasicFilterOperation"
},
"pass": {
"type": "boolean",
"default": false,
"description": "If 'false', event is filtered out."
}
},
"title": "BasicFilterOperation",
"required": [
"type"
]
}
Uses a regex to find matching parts of a key name and replaces with value or drops the key/value if a match is found. This is typically used to sanitize key names.
If a match is found then replace match with this value. Defaults to emptry string.
If a match is found then drop the key/value.
{
"type": "object",
"additionalProperties": false,
"description": "Uses a regex to find matching parts of a key name and replaces with value or drops the key/value if a match is found. This is typically used to sanitize key names.",
"properties": {
"type": {
"type": "string",
"enum": [
"KeyNameReplacementOperation"
],
"default": "KeyNameReplacementOperation"
},
"regex": {
"type": "string",
"description": "Java regex. See https://docs.oracle.com/javase/8/docs/api/java/util/regex/Pattern.html"
},
"replacement": {
"type": "string",
"default": "",
"description": "If a match is found then replace match with this value. Defaults to emptry string."
},
"drop": {
"type": "boolean",
"default": false,
"description": "If a match is found then drop the key/value."
}
},
"title": "KeyNameReplacementOperation",
"required": [
"type",
"regex"
]
}
circular reference
{
"description": "_circular reference_"
}
Transforms JSON input into a GELF message format (See http://docs.graylog.org/en/2.4/pages/gelf.html details). Note that when selecting source fields use the JsonPath notation https://github.com/json-path/JsonPath.
Field containing host string
Field containing short_message string
Field containing full_message string
Field containing epoch timestamp in seconds with optional ms as decimal. Source field must be in numeric form (not a string) or Graylog will reject the input. Alternatively, use the TimeOperation to detect your timestamp field and properly parse it, and the GelfOperation will automatically use that to set the timestamp field properly. Use this setting only to override that behavior.
Field containing syslog level number
Field containing syslog facility string
Field containing the line in a file that caused the error (decimal); optional, deprecated. Send as additional field instead.
Field containing the file (with path if you want) that caused the error (string); optional, deprecated. Send as additional field instead.
{
"type": "object",
"additionalProperties": false,
"description": "Transforms JSON input into a GELF message format (See http://docs.graylog.org/en/2.4/pages/gelf.html details). Note that when selecting source fields use the JsonPath notation https://github.com/json-path/JsonPath.",
"properties": {
"type": {
"type": "string",
"enum": [
"GelfOperation"
],
"default": "GelfOperation"
},
"src_host_field": {
"type": "array",
"items": {
"type": "string"
},
"description": "Field containing host string"
},
"src_short_message_field": {
"type": "array",
"items": {
"type": "string"
},
"description": "Field containing short_message string"
},
"src_full_message_field": {
"type": "array",
"items": {
"type": "string"
},
"description": "Field containing full_message string"
},
"src_timestamp_field": {
"type": "array",
"items": {
"type": "string"
},
"description": "Field containing epoch timestamp in seconds with optional ms as decimal. Source field must be in numeric form (not a string) or Graylog will reject the input. Alternatively, use the TimeOperation to detect your timestamp field and properly parse it, and the GelfOperation will automatically use that to set the timestamp field properly. Use this setting only to override that behavior."
},
"src_level_field": {
"type": "array",
"items": {
"type": "string"
},
"description": "Field containing syslog level number"
},
"src_facility_field": {
"type": "array",
"items": {
"type": "string"
},
"description": "Field containing syslog facility string"
},
"src_line_number_field": {
"type": "array",
"items": {
"type": "string"
},
"description": "Field containing the line in a file that caused the error (decimal); optional, deprecated. Send as additional field instead."
},
"src_file_field": {
"type": "array",
"items": {
"type": "string"
},
"description": "Field containing the file (with path if you want) that caused the error (string); optional, deprecated. Send as additional field instead."
}
},
"title": "GelfOperation",
"required": [
"type",
"src_host_field",
"src_short_message_field"
]
}
Provided a JSON object it will recursively append the primitive type of the value to the key name. For example {"foo": "one", "bar": 2} will become {"foo__str": "one", "bar__long": 2}. The mapping is string:str, boolean:boolean, array:arr, number:long or __float. It also repalces "." with "_" in key names. This operation is particularily useful for modifying JSON which will be written to ElasticSearch which does not allow conflicting value types for keys or "." in key names.
{
"type": "object",
"additionalProperties": false,
"description": "Provided a JSON object it will recursively append the primitive type of the value to the key name. For example {\"foo\": \"one\", \"bar\": 2} will become {\"foo__str\": \"one\", \"bar__long\": 2}. The mapping is string:__str, boolean:__boolean, array:__arr, number:__long or __float. It also repalces \".\" with \"_\" in key names. This operation is particularily useful for modifying JSON which will be written to ElasticSearch which does not allow conflicting value types for keys or \".\" in key names.",
"properties": {
"type": {
"type": "string",
"enum": [
"JsonKeyNameOperation"
],
"default": "JsonKeyNameOperation"
}
},
"title": "JsonKeyNameOperation",
"required": [
"type"
]
}
Filter operation to evaluate events against.
This operation is used to remove certain events from the stream before continuing on to the destination. Each event is assessed by applying a JsonPath to its payload and matching the value against a regex Pattern. If exclude is true, events that match this criteria will be filtered out. If exclude is false, any events not matching this criteria will be filtered out. For example, say these two events are in the stream: {\"data\": \"one\", \"type\": \"bar\"} and {\"data\": \"one\", \"type\": \"baz\"}. With config values: regex = \"(bar)\", path = \"$.type\", and exclude = true, after filtering, the stream will only hold {\"data\": \"one\", \"type\": \"baz\"}. If instead, exclude = false, the stream would only hold {\"data\": \"one\", \"type\": \"bar\"} after filtering.
Regex to be matched against JSON objects. See https://docs.oracle.com/javase/8/docs/api/java/util/regex/Pattern.html
If true, matches will be filtered out. If false, non-matches will be filtered out.
{
"type": "object",
"additionalProperties": false,
"description": "This operation is used to remove certain events from the stream before continuing on to the destination. Each event is assessed by applying a JsonPath to its payload and matching the value against a regex Pattern. If exclude is true, events that match this criteria will be filtered out. If exclude is false, any events not matching this criteria will be filtered out. For example, say these two events are in the stream: {\\\"data\\\": \\\"one\\\", \\\"type\\\": \\\"bar\\\"} and {\\\"data\\\": \\\"one\\\", \\\"type\\\": \\\"baz\\\"}. With config values: regex = \\\"(bar)\\\", path = \\\"$.type\\\", and exclude = true, after filtering, the stream will only hold {\\\"data\\\": \\\"one\\\", \\\"type\\\": \\\"baz\\\"}. If instead, exclude = false, the stream would only hold {\\\"data\\\": \\\"one\\\", \\\"type\\\": \\\"bar\\\"} after filtering.",
"properties": {
"type": {
"type": "string",
"enum": [
"RegexFilterOperation"
],
"default": "RegexFilterOperation"
},
"regex": {
"type": "string",
"description": "Regex to be matched against JSON objects. See https://docs.oracle.com/javase/8/docs/api/java/util/regex/Pattern.html"
},
"path": {
"type": "string",
"description": "JsonPath to the element to be matched against. See https://github.com/jayway/JsonPath"
},
"exclude": {
"type": "boolean",
"default": true,
"description": "If true, matches will be filtered out. If false, non-matches will be filtered out."
}
},
"title": "RegexFilterOperation",
"required": [
"type",
"regex",
"path"
]
}
If pass is set to false then events are filtered out.
If 'false', event is filtered out.
{
"type": "object",
"additionalProperties": false,
"description": "If pass is set to false then events are filtered out.",
"properties": {
"type": {
"type": "string",
"enum": [
"BasicFilterOperation"
],
"default": "BasicFilterOperation"
},
"pass": {
"type": "boolean",
"default": false,
"description": "If 'false', event is filtered out."
}
},
"title": "BasicFilterOperation",
"required": [
"type"
]
}
{
"type": "object",
"additionalProperties": false,
"properties": {
"operations": {
"type": "array",
"description": "List of operations to perform.",
"anyOf": [
{
"$ref": "#/definitions/ForkOperationConfig"
},
{
"$ref": "#/definitions/LowerCaseKeyOperationConfig"
},
{
"$ref": "#/definitions/JsonRootNodeOperationConfig"
},
{
"$ref": "#/definitions/SubstitutionOperationConfig"
},
{
"$ref": "#/definitions/PartitionOperationConfig"
},
{
"$ref": "#/definitions/ArraySplitOperationConfig"
},
{
"$ref": "#/definitions/URLDecodeOperationConfig"
},
{
"$ref": "#/definitions/DeleteFieldOperationConfig"
},
{
"$ref": "#/definitions/DropArraysOperationConfig"
},
{
"$ref": "#/definitions/TimeOperationConfig"
},
{
"$ref": "#/definitions/RegexFilterOperationConfig"
},
{
"$ref": "#/definitions/GeoIpOperationConfig"
},
{
"$ref": "#/definitions/FlattenOperationConfig"
},
{
"$ref": "#/definitions/BasicFilterOperationConfig"
},
{
"$ref": "#/definitions/KeyNameReplacementOperationConfig"
},
{
"$ref": "#/definitions/ConditionalOperationConfig"
},
{
"$ref": "#/definitions/GelfOperationConfig"
},
{
"$ref": "#/definitions/KeyNameOperationConfig"
}
]
},
"condition": {
"oneOf": [
{
"$ref": "#/definitions/RegexFilterOperationConfig"
},
{
"$ref": "#/definitions/BasicFilterOperationConfig"
}
],
"description": "Filter operation to evaluate events against."
}
},
"required": [
"condition"
]
}
When true events that don't match any conditions will be filtered out.
{
"type": "object",
"additionalProperties": false,
"description": "The conditional operation allows for 'if else' style branches in the flow of Bender operations. Conditions are evaluated in order of appearance and the first matching condition will receive the event.",
"properties": {
"type": {
"type": "string",
"enum": [
"ConditionalOperation"
],
"default": "ConditionalOperation"
},
"conditions": {
"type": "array",
"items": {
"$ref": "#/definitions/Condition"
},
"description": "List of conditions and their operations."
},
"filter_non_match": {
"type": "boolean",
"default": false,
"description": "When true events that don't match any conditions will be filtered out."
}
},
"title": "ConditionalOperation",
"required": [
"type"
]
}
Transforms JSON input into a GELF message format (See http://docs.graylog.org/en/2.4/pages/gelf.html details). Note that when selecting source fields use the JsonPath notation https://github.com/json-path/JsonPath.
Field containing host string
Field containing short_message string
Field containing full_message string
Field containing epoch timestamp in seconds with optional ms as decimal. Source field must be in numeric form (not a string) or Graylog will reject the input. Alternatively, use the TimeOperation to detect your timestamp field and properly parse it, and the GelfOperation will automatically use that to set the timestamp field properly. Use this setting only to override that behavior.
Field containing syslog level number
Field containing syslog facility string
Field containing the line in a file that caused the error (decimal); optional, deprecated. Send as additional field instead.
Field containing the file (with path if you want) that caused the error (string); optional, deprecated. Send as additional field instead.
{
"type": "object",
"additionalProperties": false,
"description": "Transforms JSON input into a GELF message format (See http://docs.graylog.org/en/2.4/pages/gelf.html details). Note that when selecting source fields use the JsonPath notation https://github.com/json-path/JsonPath.",
"properties": {
"type": {
"type": "string",
"enum": [
"GelfOperation"
],
"default": "GelfOperation"
},
"src_host_field": {
"type": "array",
"items": {
"type": "string"
},
"description": "Field containing host string"
},
"src_short_message_field": {
"type": "array",
"items": {
"type": "string"
},
"description": "Field containing short_message string"
},
"src_full_message_field": {
"type": "array",
"items": {
"type": "string"
},
"description": "Field containing full_message string"
},
"src_timestamp_field": {
"type": "array",
"items": {
"type": "string"
},
"description": "Field containing epoch timestamp in seconds with optional ms as decimal. Source field must be in numeric form (not a string) or Graylog will reject the input. Alternatively, use the TimeOperation to detect your timestamp field and properly parse it, and the GelfOperation will automatically use that to set the timestamp field properly. Use this setting only to override that behavior."
},
"src_level_field": {
"type": "array",
"items": {
"type": "string"
},
"description": "Field containing syslog level number"
},
"src_facility_field": {
"type": "array",
"items": {
"type": "string"
},
"description": "Field containing syslog facility string"
},
"src_line_number_field": {
"type": "array",
"items": {
"type": "string"
},
"description": "Field containing the line in a file that caused the error (decimal); optional, deprecated. Send as additional field instead."
},
"src_file_field": {
"type": "array",
"items": {
"type": "string"
},
"description": "Field containing the file (with path if you want) that caused the error (string); optional, deprecated. Send as additional field instead."
}
},
"title": "GelfOperation",
"required": [
"type",
"src_host_field",
"src_short_message_field"
]
}
Provided a JSON object it will recursively append the primitive type of the value to the key name. For example {"foo": "one", "bar": 2} will become {"foo__str": "one", "bar__long": 2}. The mapping is string:str, boolean:boolean, array:arr, number:long or __float. It also repalces "." with "_" in key names. This operation is particularily useful for modifying JSON which will be written to ElasticSearch which does not allow conflicting value types for keys or "." in key names.
{
"type": "object",
"additionalProperties": false,
"description": "Provided a JSON object it will recursively append the primitive type of the value to the key name. For example {\"foo\": \"one\", \"bar\": 2} will become {\"foo__str\": \"one\", \"bar__long\": 2}. The mapping is string:__str, boolean:__boolean, array:__arr, number:__long or __float. It also repalces \".\" with \"_\" in key names. This operation is particularily useful for modifying JSON which will be written to ElasticSearch which does not allow conflicting value types for keys or \".\" in key names.",
"properties": {
"type": {
"type": "string",
"enum": [
"JsonKeyNameOperation"
],
"default": "JsonKeyNameOperation"
}
},
"title": "JsonKeyNameOperation",
"required": [
"type"
]
}
{
"type": "object",
"additionalProperties": false,
"properties": {
"name": {
"type": "string",
"description": "Source name"
},
"source_regex": {
"type": "string",
"default": ".*",
"description": "Pattern to match source trigger against"
},
"regex_patterns": {
"type": "array",
"items": {
"type": "string"
},
"description": "Regex patterns to filter events by prior to deserialization"
},
"contains_strings": {
"type": "array",
"items": {
"type": "string"
},
"description": "Filter events containing these Strings prior to deserialization"
},
"deserializer": {
"oneOf": [
{
"$ref": "#/definitions/RegexDeserializerConfig"
},
{
"$ref": "#/definitions/GenericJsonDeserializerConfig"
}
],
"description": "Deserializer configuration"
},
"operations": {
"type": "array",
"description": "Operation configuration",
"anyOf": [
{
"$ref": "#/definitions/ForkOperationConfig"
},
{
"$ref": "#/definitions/LowerCaseKeyOperationConfig"
},
{
"$ref": "#/definitions/JsonRootNodeOperationConfig"
},
{
"$ref": "#/definitions/SubstitutionOperationConfig"
},
{
"$ref": "#/definitions/PartitionOperationConfig"
},
{
"$ref": "#/definitions/ArraySplitOperationConfig"
},
{
"$ref": "#/definitions/URLDecodeOperationConfig"
},
{
"$ref": "#/definitions/DeleteFieldOperationConfig"
},
{
"$ref": "#/definitions/DropArraysOperationConfig"
},
{
"$ref": "#/definitions/TimeOperationConfig"
},
{
"$ref": "#/definitions/RegexFilterOperationConfig"
},
{
"$ref": "#/definitions/GeoIpOperationConfig"
},
{
"$ref": "#/definitions/FlattenOperationConfig"
},
{
"$ref": "#/definitions/BasicFilterOperationConfig"
},
{
"$ref": "#/definitions/KeyNameReplacementOperationConfig"
},
{
"$ref": "#/definitions/ConditionalOperationConfig"
},
{
"$ref": "#/definitions/GelfOperationConfig"
},
{
"$ref": "#/definitions/KeyNameOperationConfig"
}
]
}
},
"required": [
"name"
]
}
Wrapper configuration
Adds information about kinesis stream which the event came from. Included is partition, sequence number, source arn, event source, function name, function version, arrival timestsamp (ms since epoch), processing time (ms), processing delay (ms), processing timestsamp (ms since epoch).
{
"type": "object",
"additionalProperties": false,
"description": "Adds information about kinesis stream which the event came from. Included is partition, sequence number, source arn, event source, function name, function version, arrival timestsamp (ms since epoch), processing time (ms), processing delay (ms), processing timestsamp (ms since epoch).",
"properties": {
"type": {
"type": "string",
"enum": [
"KinesisWrapper"
],
"default": "KinesisWrapper"
}
},
"title": "KinesisWrapper",
"required": [
"type"
]
}
A meta wrapper that performs no wrapping and passes the payload through.
{
"type": "object",
"additionalProperties": false,
"description": "A meta wrapper that performs no wrapping and passes the payload through.",
"properties": {
"type": {
"type": "string",
"enum": [
"PassthroughWrapper"
],
"default": "PassthroughWrapper"
}
},
"title": "PassthroughWrapper",
"required": [
"type"
]
}
Includes the original payload, sha1 hash of the original event and event timestamp.
{
"type": "object",
"additionalProperties": false,
"description": "Includes the original payload, sha1 hash of the original event and event timestamp.",
"properties": {
"type": {
"type": "string",
"enum": [
"BasicWrapper"
],
"default": "BasicWrapper"
}
},
"title": "BasicWrapper",
"required": [
"type"
]
}
Adds information about function name, function version, processing time (in ms), processing delay (in ms), processing timestsamp (ms since epoch), sha1 hash of the original event, and s3 key, bucket, and key version from the source S3file read. Only works with the S3Handler and SNSS3Handler.
{
"type": "object",
"additionalProperties": false,
"description": "Adds information about function name, function version, processing time (in ms), processing delay (in ms), processing timestsamp (ms since epoch), sha1 hash of the original event, and s3 key, bucket, and key version from the source S3file read. Only works with the S3Handler and SNSS3Handler.",
"properties": {
"type": {
"type": "string",
"enum": [
"S3Wrapper"
],
"default": "S3Wrapper"
}
},
"title": "S3Wrapper",
"required": [
"type"
]
}
Serializer configuration
Serializes events into JSON format.
GSON field naming policy. See https://google.github.io/gson/apidocs/com/google/gson/FieldNamingPolicy.html
{
"type": "object",
"additionalProperties": false,
"description": "Serializes events into JSON format.",
"properties": {
"type": {
"type": "string",
"enum": [
"Json"
],
"default": "Json"
},
"field_naming_policy": {
"type": "string",
"default": "LOWER_CASE_WITH_UNDERSCORES",
"enum": [
"IDENTITY",
"UPPER_CAMEL_CASE",
"UPPER_CAMEL_CASE_WITH_SPACES",
"LOWER_CASE_WITH_UNDERSCORES",
"LOWER_CASE_WITH_DASHES",
"LOWER_CASE_WITH_DOTS"
],
"description": "GSON field naming policy. See https://google.github.io/gson/apidocs/com/google/gson/FieldNamingPolicy.html"
}
},
"title": "Json",
"required": [
"type"
]
}
Transport configuration
Writes events to a file on the local file system. Typically only used for testing and debugging.
Number of concurrent transporters allowed
Local filesystem file to write data to
{
"type": "object",
"additionalProperties": false,
"description": "Writes events to a file on the local file system. Typically only used for testing and debugging.",
"properties": {
"type": {
"type": "string",
"enum": [
"File"
],
"default": "File"
},
"threads": {
"type": "integer",
"default": 5,
"description": "Number of concurrent transporters allowed"
},
"filename": {
"type": "string",
"description": "Local filesystem file to write data to"
}
},
"title": "File",
"required": [
"type",
"filename"
]
}
Writes batches of events to S3. The output filename will either be the unique function inocation request id as specified by Lambda or a hash of the input filename when using the S3 handler. Required IAM permissions are: s3:AbortMultipartUpload, s3:PutObject, s3:ListMultipartUploadParts, s3:ListBucketMultipartUploads
Number of concurrent transporters allowed
Region of remote AWS service. Not required to be set if you are using a service within the Lambda's current region. Ensure a service is supported in the remote region.
S3 bucket name.
Path to append to S3 keys.
Compress files with bz2 compression.
When using compression optionally compress buffered data as it is serialized. This is optimal for no or low cardinality partitioning. When writing high cardinality data this may result in OOMs due to high overhead of having a compressor for each partition.
Amount of serialized data to hold in memory before forcing a write to S3. It is important to note that this value is per partition and having too many or too large buffers will lead to OOMs. Also, S3 requires multi-part files have parts of more than 5mb. When using compression and NOT compressing buffers you must take compression ratio into account. Meaning that when compressed the contents of your buffer must be more than 5mb or the put to S3 will fail. Set this value accordingly high to account for compression.
{
"type": "object",
"additionalProperties": false,
"description": "Writes batches of events to S3. The output filename will either be the unique function inocation request id as specified by Lambda or a hash of the input filename when using the S3 handler. Required IAM permissions are: s3:AbortMultipartUpload, s3:PutObject, s3:ListMultipartUploadParts, s3:ListBucketMultipartUploads",
"properties": {
"type": {
"type": "string",
"enum": [
"S3"
],
"default": "S3"
},
"threads": {
"type": "integer",
"default": 5,
"description": "Number of concurrent transporters allowed"
},
"region": {
"type": "string",
"enum": [
"GovCloud",
"US_GOV_EAST_1",
"US_EAST_1",
"US_EAST_2",
"US_WEST_1",
"US_WEST_2",
"EU_WEST_1",
"EU_WEST_2",
"EU_WEST_3",
"EU_CENTRAL_1",
"EU_NORTH_1",
"AP_EAST_1",
"AP_SOUTH_1",
"AP_SOUTHEAST_1",
"AP_SOUTHEAST_2",
"AP_NORTHEAST_1",
"AP_NORTHEAST_2",
"SA_EAST_1",
"CN_NORTH_1",
"CN_NORTHWEST_1",
"CA_CENTRAL_1",
"ME_SOUTH_1"
],
"description": "Region of remote AWS service. Not required to be set if you are using a service within the Lambda's current region. Ensure a service is supported in the remote region."
},
"bucket_name": {
"type": "string",
"description": "S3 bucket name."
},
"base_path": {
"type": "string",
"description": "Path to append to S3 keys."
},
"use_compression": {
"type": "boolean",
"default": true,
"description": "Compress files with bz2 compression."
},
"compress_buffer": {
"type": "boolean",
"default": false,
"description": "When using compression optionally compress buffered data as it is serialized. This is optimal for no or low cardinality partitioning. When writing high cardinality data this may result in OOMs due to high overhead of having a compressor for each partition."
},
"max_buffer_size": {
"type": "integer",
"minimum": 5242880,
"default": 6291456,
"description": "Amount of serialized data to hold in memory before forcing a write to S3. It is important to note that this value is per partition and having too many or too large buffers will lead to OOMs. Also, S3 requires multi-part files have parts of more than 5mb. When using compression and NOT compressing buffers you must take compression ratio into account. Meaning that when compressed the contents of your buffer must be more than 5mb or the put to S3 will fail. Set this value accordingly high to account for compression."
}
},
"title": "S3",
"required": [
"type",
"bucket_name"
]
}
Writes to an ElasticSearch cluster. When using AWS hosted ES do not use gzip compression.
Number of concurrent transporters allowed
HTTP endpoint hostname.
ElasticSearch HTTP endpoint port.
Use SSL connections (certificates are not validated).
Use GZIP compression on HTTP calls.
Maximum number of documents in api call.
Number of retries to make when a put failure occurs.
Initial delay between retries. If more than one retries specified exponential backoff is used.
Socket timeout on HTTP connection.
HTTP headers to include.
HTTP headers to include.
{
"type": "object",
"additionalProperties": {
"oneOf": [
{
"$ref": "#/definitions/KmsValueConfig"
},
{
"$ref": "#/definitions/StringValueConfig"
}
]
},
"description": "HTTP headers to include."
}
Authentication scheme.
{
"type": "object",
"additionalProperties": false,
"properties": {
"type": {
"type": "string",
"enum": [
"UrlSigningAuth"
],
"default": "UrlSigningAuth"
},
"region": {
"type": "string",
"enum": [
"GovCloud",
"US_GOV_EAST_1",
"US_EAST_1",
"US_EAST_2",
"US_WEST_1",
"US_WEST_2",
"EU_WEST_1",
"EU_WEST_2",
"EU_WEST_3",
"EU_CENTRAL_1",
"EU_NORTH_1",
"AP_EAST_1",
"AP_SOUTH_1",
"AP_SOUTHEAST_1",
"AP_SOUTHEAST_2",
"AP_NORTHEAST_1",
"AP_NORTHEAST_2",
"SA_EAST_1",
"CN_NORTH_1",
"CN_NORTHWEST_1",
"CA_CENTRAL_1",
"ME_SOUTH_1"
]
},
"service": {
"type": "string"
},
"http_interceptor": {}
},
"title": "UrlSigningAuth",
"required": [
"type",
"region",
"service"
]
}
KMS encrypted value.
AWS region associated with the KMS key used to encrypt the value.
{
"type": "object",
"additionalProperties": false,
"properties": {
"type": {
"type": "string",
"enum": [
"KmsValue"
],
"default": "KmsValue"
},
"value": {
"type": "string",
"description": "KMS encrypted value."
},
"region": {
"type": "string",
"enum": [
"GovCloud",
"US_GOV_EAST_1",
"US_EAST_1",
"US_EAST_2",
"US_WEST_1",
"US_WEST_2",
"EU_WEST_1",
"EU_WEST_2",
"EU_WEST_3",
"EU_CENTRAL_1",
"EU_NORTH_1",
"AP_EAST_1",
"AP_SOUTH_1",
"AP_SOUTHEAST_1",
"AP_SOUTHEAST_2",
"AP_NORTHEAST_1",
"AP_NORTHEAST_2",
"SA_EAST_1",
"CN_NORTH_1",
"CN_NORTHWEST_1",
"CA_CENTRAL_1",
"ME_SOUTH_1"
],
"description": "AWS region associated with the KMS key used to encrypt the value."
}
},
"title": "KmsValue",
"required": [
"type",
"value",
"region"
]
}
{
"type": "object",
"additionalProperties": false,
"properties": {
"type": {
"type": "string",
"enum": [
"StringValue"
],
"default": "StringValue"
},
"value": {
"type": "string"
}
},
"title": "StringValue",
"required": [
"type",
"value"
]
}
{
"type": "object",
"additionalProperties": false,
"properties": {
"type": {
"type": "string",
"enum": [
"UserPassAuth"
],
"default": "UserPassAuth"
},
"username": {
"type": "string"
},
"password": {
"oneOf": [
{
"$ref": "#/definitions/KmsValueConfig"
},
{
"$ref": "#/definitions/StringValueConfig"
}
]
}
},
"title": "UserPassAuth",
"required": [
"type",
"username",
"password"
]
}
Index to write to.
ElasticSearch document type.
ElasticSearch bulk api path including leading slash '/'.
Java time format to append to index name.
Name of the routing field. ElasticSearch 6+ changed the field name from _routing to routing.
{
"type": "object",
"additionalProperties": {
"type": "string"
}
}
Use hash id generated by Bender as document id.
Use partitions as a mechanism for routing records to Elasticsearch. Routing allows a document to be written to a specific shard within an index. Using routing can, if configured correctly, dramatically improve read and/or write performance when indexing or searching documents. However, this is largely based on your data, data source, and workload. For more details on routing consult:
https://www.elastic.co/blog/customizing-your-document-routing
https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-routing-field.html
Partition keys and values will be concatenated by "=" and separated by "/". Ifyour partitions are part1=foo and part2=bar the Elasticsearch "_routing" field will have the value "part1=foo/part2=bar".
A typical routing strategy is to use the data source context associated with a function's invocation. For instance with Kinesis this can be the shard-id and with S3 the source file. This approach ensures a single function invocation writes to only one shard. However, to avoid hot spots, it's also advised that you have a secondary partitioning key with low cardinality. For example bucketing by 5 minute intervals in addition to using shard-id.
{
"type": "object",
"additionalProperties": false,
"description": "Writes to an ElasticSearch cluster. When using AWS hosted ES do not use gzip compression.",
"properties": {
"type": {
"type": "string",
"enum": [
"ElasticSearch"
],
"default": "ElasticSearch"
},
"threads": {
"type": "integer",
"default": 5,
"description": "Number of concurrent transporters allowed"
},
"hostname": {
"type": "string",
"description": "HTTP endpoint hostname."
},
"port": {
"type": "integer",
"minimum": 1,
"maximum": 65535,
"default": 9200,
"description": "ElasticSearch HTTP endpoint port."
},
"use_ssl": {
"type": "boolean",
"default": false,
"description": "Use SSL connections (certificates are not validated)."
},
"use_gzip": {
"type": "boolean",
"default": false,
"description": "Use GZIP compression on HTTP calls."
},
"batch_size": {
"type": "integer",
"minimum": 1,
"maximum": 100000,
"default": 500,
"description": "Maximum number of documents in api call."
},
"retry_count": {
"type": "integer",
"minimum": 0,
"maximum": 10,
"default": 0,
"description": "Number of retries to make when a put failure occurs."
},
"retry_delay": {
"type": "integer",
"minimum": 1,
"maximum": 60000,
"default": 1000,
"description": "Initial delay between retries. If more than one retries specified exponential backoff is used."
},
"timeout": {
"type": "integer",
"minimum": 1000,
"maximum": 300000,
"default": 40000,
"description": "Socket timeout on HTTP connection."
},
"http_headers": {
"type": "object",
"additionalProperties": {
"oneOf": [
{
"$ref": "#/definitions/KmsValueConfig"
},
{
"$ref": "#/definitions/StringValueConfig"
}
]
},
"description": "HTTP headers to include."
},
"auth_config": {
"oneOf": [
{
"$ref": "#/definitions/UrlSigningAuthConfig"
},
{
"$ref": "#/definitions/BasicHttpAuthConfig"
}
],
"description": "Authentication scheme."
},
"index": {
"type": "string",
"description": "Index to write to."
},
"document_type": {
"type": "string",
"description": "ElasticSearch document type."
},
"bulk_api_path": {
"type": "string",
"default": "/_bulk",
"description": "ElasticSearch bulk api path including leading slash '/'."
},
"index_time_format": {
"type": "string",
"description": "Java time format to append to index name."
},
"routing_field_name": {
"type": "string",
"default": "_routing",
"description": "Name of the routing field. ElasticSearch 6+ changed the field name from `_routing` to `routing`."
},
"http_string_headers": {
"type": "object",
"additionalProperties": {
"type": "string"
}
},
"use_hashid": {
"type": "boolean",
"default": false,
"description": "Use hash id generated by Bender as document id."
},
"use_partitions_for_routing": {
"type": "boolean",
"default": false,
"description": "Use partitions as a mechanism for routing records to Elasticsearch. Routing allows a document to be written to a specific shard within an index. Using routing can, if configured correctly, dramatically improve read and/or write performance when indexing or searching documents. However, this is largely based on your data, data source, and workload. For more details on routing consult:\n\nhttps://www.elastic.co/blog/customizing-your-document-routing\nhttps://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-routing-field.html\n\nPartition keys and values will be concatenated by \"=\" and separated by \"/\". Ifyour partitions are part1=foo and part2=bar the Elasticsearch \"_routing\" field will have the value \"part1=foo/part2=bar\". \n\nA typical routing strategy is to use the data source context associated with a function's invocation. For instance with Kinesis this can be the shard-id and with S3 the source file. This approach ensures a single function invocation writes to only one shard. However, to avoid hot spots, it's also advised that you have a secondary partitioning key with low cardinality. For example bucketing by 5 minute intervals in addition to using shard-id."
}
},
"title": "ElasticSearch",
"required": [
"type",
"hostname",
"index",
"document_type"
]
}
Writes events to a Scalyr endpoint.
Number of concurrent transporters allowed
Scalyr HTTP endpoint hostname.
HTTP endpoint port.
Use SSL connections (certificates are not validated).
Use GZIP compression on HTTP calls.
Maximum number of documents in api call.
Number of retries to make when a put failure occurs.
Initial delay between retries. If more than one retries specified exponential backoff is used.
Socket timeout on HTTP connection.
HTTP headers to include.
HTTP headers to include.
{
"type": "object",
"additionalProperties": {
"oneOf": [
{
"$ref": "#/definitions/KmsValueConfig"
},
{
"$ref": "#/definitions/StringValueConfig"
}
]
},
"description": "HTTP headers to include."
}
Scalyr auth token.
KMS encrypted value.
AWS region associated with the KMS key used to encrypt the value.
{
"type": "object",
"additionalProperties": false,
"properties": {
"type": {
"type": "string",
"enum": [
"KmsValue"
],
"default": "KmsValue"
},
"value": {
"type": "string",
"description": "KMS encrypted value."
},
"region": {
"type": "string",
"enum": [
"GovCloud",
"US_GOV_EAST_1",
"US_EAST_1",
"US_EAST_2",
"US_WEST_1",
"US_WEST_2",
"EU_WEST_1",
"EU_WEST_2",
"EU_WEST_3",
"EU_CENTRAL_1",
"EU_NORTH_1",
"AP_EAST_1",
"AP_SOUTH_1",
"AP_SOUTHEAST_1",
"AP_SOUTHEAST_2",
"AP_NORTHEAST_1",
"AP_NORTHEAST_2",
"SA_EAST_1",
"CN_NORTH_1",
"CN_NORTHWEST_1",
"CA_CENTRAL_1",
"ME_SOUTH_1"
],
"description": "AWS region associated with the KMS key used to encrypt the value."
}
},
"title": "KmsValue",
"required": [
"type",
"value",
"region"
]
}
{
"type": "object",
"additionalProperties": false,
"properties": {
"type": {
"type": "string",
"enum": [
"StringValue"
],
"default": "StringValue"
},
"value": {
"type": "string"
}
},
"title": "StringValue",
"required": [
"type",
"value"
]
}
Scalyr String Parser.
{
"type": "object",
"additionalProperties": {
"type": "string"
}
}
{
"type": "object",
"additionalProperties": false,
"description": "Writes events to a Scalyr endpoint.",
"properties": {
"type": {
"type": "string",
"enum": [
"Scalyr"
],
"default": "Scalyr"
},
"threads": {
"type": "integer",
"default": 5,
"description": "Number of concurrent transporters allowed"
},
"hostname": {
"type": "string",
"default": "www.scalyr.com",
"description": "Scalyr HTTP endpoint hostname."
},
"port": {
"type": "integer",
"minimum": 1,
"maximum": 65535,
"default": 443,
"description": "HTTP endpoint port."
},
"use_ssl": {
"type": "boolean",
"default": false,
"description": "Use SSL connections (certificates are not validated)."
},
"use_gzip": {
"type": "boolean",
"default": false,
"description": "Use GZIP compression on HTTP calls."
},
"batch_size": {
"type": "integer",
"minimum": 1,
"maximum": 100000,
"default": 500,
"description": "Maximum number of documents in api call."
},
"retry_count": {
"type": "integer",
"minimum": 0,
"maximum": 10,
"default": 0,
"description": "Number of retries to make when a put failure occurs."
},
"retry_delay": {
"type": "integer",
"minimum": 1,
"maximum": 60000,
"default": 1000,
"description": "Initial delay between retries. If more than one retries specified exponential backoff is used."
},
"timeout": {
"type": "integer",
"minimum": 1000,
"maximum": 300000,
"default": 40000,
"description": "Socket timeout on HTTP connection."
},
"http_headers": {
"type": "object",
"additionalProperties": {
"oneOf": [
{
"$ref": "#/definitions/KmsValueConfig"
},
{
"$ref": "#/definitions/StringValueConfig"
}
]
},
"description": "HTTP headers to include."
},
"token": {
"oneOf": [
{
"$ref": "#/definitions/KmsValueConfig"
},
{
"$ref": "#/definitions/StringValueConfig"
}
],
"description": "Scalyr auth token."
},
"parser": {
"type": "string",
"default": "json",
"description": "Scalyr String Parser."
},
"http_string_headers": {
"type": "object",
"additionalProperties": {
"type": "string"
}
}
},
"title": "Scalyr",
"required": [
"type",
"token"
]
}
Writes logs into a Graylog TCP GELF Input
Number of concurrent transporters allowed
TCP endpoint hostname.
TCP endpoint port.
Use SSL/TLS connections.
Verify SSL/TLS certificates.
Maximum size (in bytes) in memory before triggering a write.
Number of retries to make when a write failure occurs.
Initial delay between retries. If more than one retries specified exponential backoff is used.
Socket timeout (in milliseconds) on TCP connection.
{
"type": "object",
"additionalProperties": false,
"description": "Writes logs into a Graylog TCP GELF Input",
"properties": {
"type": {
"type": "string",
"enum": [
"Gelf"
],
"default": "Gelf"
},
"threads": {
"type": "integer",
"default": 5,
"description": "Number of concurrent transporters allowed"
},
"hostname": {
"type": "string",
"description": "TCP endpoint hostname."
},
"port": {
"type": "integer",
"minimum": 1,
"maximum": 65535,
"description": "TCP endpoint port."
},
"use_ssl": {
"type": "boolean",
"default": true,
"description": "Use SSL/TLS connections."
},
"verify_ssl": {
"type": "boolean",
"default": true,
"description": "Verify SSL/TLS certificates."
},
"max_buffer_size": {
"type": "integer",
"minimum": 1024,
"maximum": 10485760,
"default": 10240,
"description": "Maximum size (in bytes) in memory before triggering a write."
},
"retry_count": {
"type": "integer",
"minimum": 0,
"maximum": 10,
"default": 0,
"description": "Number of retries to make when a write failure occurs."
},
"retry_delay": {
"type": "integer",
"minimum": 1,
"maximum": 60000,
"default": 1000,
"description": "Initial delay between retries. If more than one retries specified exponential backoff is used."
},
"timeout": {
"type": "integer",
"minimum": 1000,
"maximum": 60000,
"default": 30000,
"description": "Socket timeout (in milliseconds) on TCP connection."
}
},
"title": "Gelf",
"required": [
"type",
"hostname",
"port"
]
}
Writes events out to the console for testing and debugging
Number of concurrent transporters allowed
{
"type": "object",
"additionalProperties": false,
"description": "Writes events out to the console for testing and debugging",
"properties": {
"type": {
"type": "string",
"enum": [
"Stdout"
],
"default": "Stdout"
},
"threads": {
"type": "integer",
"default": 5,
"description": "Number of concurrent transporters allowed"
}
},
"title": "Stdout",
"required": [
"type"
]
}
HTTP transport that only supports HTTP1/1
Number of concurrent transporters allowed
HTTP endpoint hostname.
HTTP endpoint port.
Use SSL connections (certificates are not validated).
Use GZIP compression on HTTP calls.
Maximum number of documents in api call.
Number of retries to make when a put failure occurs.
Initial delay between retries. If more than one retries specified exponential backoff is used.
Socket timeout on HTTP connection.
HTTP headers to include.