This document was generated with benthos --list-buffers

Benthos uses a transaction based model for guaranteeing delivery of messages without the need for a buffer. This ensures that messages are never acknowledged from a source until the message has left the target sink.

However, sometimes the transaction model is undesired, in which case there are a range of buffer options available which decouple input sources from the rest of the Benthos pipeline.

Buffers can therefore solve a number of typical streaming problems but come at the cost of weakening the delivery guarantees of your pipeline. Common problems that might warrant use of a buffer are:

If you believe that a problem you have would be solved by a buffer the next step is to choose an implementation based on the throughput and delivery guarantees you need. In order to help here are some simplified tables outlining the different options and their qualities:


Type Throughput Consumers Capacity
Memory Highest Parallel RAM
Mmap File High Single Disk

Delivery Guarantees

Event Shutdown Crash Disk Corruption
Memory Flushed* Lost Lost
Mmap File Persisted Lost Lost

* Makes a best attempt at flushing the remaining messages before closing gracefully.


  1. memory
  2. mmap_file
  3. none


type: memory
    byte_size: 0
      type: static
      static: false
    count: 0
    enabled: false
    period: ""
  limit: 524288000

The memory buffer stores messages in RAM. During shutdown Benthos will make a best attempt at flushing all remaining messages before exiting cleanly.

This buffer has a configurable limit, where consumption will be stopped with back pressure upstream if the total size of messages in the buffer reaches this amount. Since this calculation is only an estimate, and the real size of messages in RAM is always higher, it is recommended to set the limit significantly below the amount of RAM available.


It is possible to batch up messages sent from this buffer using a batch policy. Batches are considered complete and will be flushed downstream when either of the following conditions are met:

This is a more powerful way of batching messages than the batch processor, as it does not rely on new messages entering the pipeline in order to trigger the conditions.


type: mmap_file
  clean_up: true
  directory: ""
  file_size: 2.62144e+08
  reserved_disk_space: 1.048576e+08
  retry_period: 1s

DEPRECATED: This buffer type is due to be removed in V3.

The mmap file buffer type uses memory mapped files to perform low-latency, file-persisted buffering of messages.

To configure the mmap file buffer you need to designate a writeable directory for storing the mapped files. Benthos will create multiple files in this directory as it fills them.

When files are fully read from they will be deleted. You can disable this feature if you wish to preserve the data indefinitely, but the directory will fill up as fast as data passes through.

WARNING: This buffer currently wipes all metadata from message payloads. If you are using metadata in your pipeline you should avoid using this buffer, or preferably all buffers altogether.


type: none
none: {}

Selecting no buffer (default) means the output layer is directly coupled with the input layer. This is the safest and lowest latency option since acknowledgements from at-least-once protocols can be propagated all the way from the output protocol to the input protocol.

If the output layer is hit with back pressure it will propagate all the way to the input layer, and further up the data stream. If you need to relieve your pipeline of this back pressure consider using a more robust buffering solution such as Kafka before resorting to alternatives.