Migrating to Version 3

Benthos version 3 comes with some breaking service and configuration changes as well as a few breaking API changes.

Service

Memory Map File Buffer Removed

The long deprecated mmap_file buffer has been removed. If you were still relying on this buffer implementation then please raise an issue.

Old Metrics Paths Removed

The following undocumented metrics paths have been removed:

All of these paths have remaining equivalents.

Metric Path Changes

The http_client output client metrics have been renamed from output.*.output.http_client to output.*.client.

Configuration

Metrics Prefix

The configuration field prefix within metrics has been moved from the root of the config object to individual types. E.g. when using statsd the field metrics.prefix should be replaced with metrics.statsd.prefix.

JSON Paths

Many components within Benthos use an unspecified "JSON dot path" syntax for querying and setting fields within JSON documents. The format of these paths has been formalised to make them clearer and more generally useful, but this potentially breaks your paths when they query against hierarchies that contain arrays.

The formal specification for v3 can be found in this document.

The following components are affected:

Migration Guide

In order to replicate the exact same behaviour as currently exists your paths should be updated to include the character * wherever an array exists. For example, the default value of sqs_body_path for the s3 input has been updated from Records.s3.object.key to Records.*.s3.object.key.

Process DAG Stage Names

The process_dag processor now only permits workflow stage names matching the following regular expression: [a-zA-Z0-9_-]+. The reasoning for this restriction is to potentially expand the features of process_dag in the future with custom root fields (e.g. $on_error).

Go API

Modules

Benthos now fully adheres to Go Modules, import paths must therefore now contain the major version (v3) like so:

import "github.com/Jeffail/benthos/v3/lib/processor"

It should be pretty quick to update your imports, either using a tool or just:

grep "Jeffail/benthos" . -Rl | grep -e "\.go$" | xargs -I{} sed -i 's/Jeffail\/benthos/Jeffail\/benthos\/v3/g' {}

Other