High performance FIFO buffer

Release: 0.13

→ preface → usage → download → history → todo → contact

Preface

This utility is designed to buffer data during real time processing like on the fly audio encoding or tape backups. It is optimized for high throughput, fast and reliable response and low CPU load. The data is not copied around in memory. It is not even touched. See examples for more details.

Usage

Requirements

OS/2 binary: The C runtime (libc061.dll) which can be downloaded from Netlabs by ftp.

Installation

Copy the executable somewhere to your path.

Parameters

buffer2 source destination[options]

source, destination must be one of the following:

An absolute or relative or UNC path to a file.

OS/2: A local or remote name of an existing or non-existing named pipe. In case the pipe does not exist it is created and connected exactly once.

A TCP/IP port following the syntax tcpip://[hostname]:port. Without a host name the port is turned into listening state on the local machine with bind address 0.0.0.0 and exactly one connection is accepted.

A device name like com1: or /dev/tape.

A - (dash) meaning stdin or stdout respectively.

Options

Command line
Description

-b=size Set the size of the FIFO-Buffer. size may be followed by k, M or G to give the size in kiB, MiB or GiB, i.e. 1024¹, 1024² or 1024³ bytes. It is strongly recommended to use a multiple of 1kiB as buffer size. Otherwise the performance may degrade significantly. By default the buffer size is 64kiB.

-p=size Set the size of the Pipe-Buffer if a pipe is created by buffer2. size may be followed by k to give the size in kiB, i.e. 1024 bytes. Note that the pipe buffer must not exceed 64kiB. This is a limitation of OS/2. By default the pipe buffer size is 8kiB.

-r=size Maximum request size. size may be followed by k, M or G to give the size in kiB, MiB or GiB, i.e. 1024¹, 1024² or 1024³ bytes. By default the request size is automatically calculated. For buffer sizes below 256kiB one quarter of the buffer size is used, for buffers sizes of 256kiB or more one eighth of the buffer size is used. It is recommended that the maximum request size is a factor of the buffer size to avoid fragmentation. Furthermore the request size should not be larger than half of the buffer size. Otherwise the buffer becomes very ineffective because large parts of the buffer may be blocked by either the input or the output side.

-h=level High water mark. Once the buffer gets empty the output side of the buffer will stop working until the buffer is filled up to the high water mark. level may be followed by k, M or G to give the high water mark in kiB, MiB or GiB, i.e. 1024¹, 1024² or 1024³ bytes. If level is followed by % it counts relative to the buffer size in percent. The default value is zero. That causes the output side to resume as soon as any data is available.
This parameter may be used to defragment streaming data in pipes or to avoid unnecessary stops of tape drives or CD writers. In the latter cases a value of 100% may be reasonable. This causes nothing to be written until the buffer is full (or the end of the input stream is reached, of course).

-l=level Low water mark. This is the opposite of the high water mark. Once the buffer gets full it will not resume to fill the buffer before the buffer level drops below the low water mark. level may be followed by k, M or G to give the high water mark in kiB, MiB or GiB, i.e. 1024¹, 1024² or 1024³ bytes. If level is followed by % it counts relative to the buffer size in percent. The default value is 100%. This causes the input operation to resume as soon as there is any space in the buffer.

-c Enable file system caching (if source or destination is an ordinary file). This may or may not increase the throughput. Normally it is no good choice to use the file system's cache here, because the FIFO is in fact a kind of cache. Generally this will only increase the load on the CPU to memory interface.

-si Print statistics from the input side of the FIFO to stderr. This option is for diagnostic purposes. Using it may stop the FIFO from spooling data, because the print functions may block, e.g. if one hits the "pause" key.

-so Print statistics from the output side of the FIFO to stderr. This option is for diagnostic purposes. Using it may stop the FIFO from spooling data, because the print functions may block, e.g. if one hits the "pause" key.

Examples

playrec /r con | sp f4 o buffer2 - - /b=64M /h=32k | toolame - -b 320 -m s -x "filename.mp2"

Record an audio stream and encode it on the fly. To avoid any interrupt in the audio stream the encoder process must run at reasonable high priority and must not stop because of a high I/O load or something like that. Here buffer2 comes into play. With a data buffer of a few minutes capacity this can be relaxed significantly and the encoder may run at normal priority level. (sp f4 o increases the priority)

machine1>sp f4 o buffer2 /b=64M /b=60k /h=100% \\machine1\pipe\to_tape tape_device machine2>tar c ... -f \\pipe\to_tape

Make a backup of a workstation with tar or something like that over the network to a DLT drive or another media which requires a high sustained data rate. The high water mark of 100% ensures that the tape will not stop before at least one buffer size of data is written - usually much more since the buffer will be still filled at the same time.

Download

This utility is freeware.

ZIP file with the source, an OS/2 executable and this guide: Release 0.13

History

Version 0.13 (27.06.2006)

Unix port (posix).
Minor bug fixes.

Version 0.12 (22.02.2006)

Added TCP/IP support.

Version 0.11 (16.01.2006)

Fixed command line parser. V0.1 accidentally had invalid old source code.
Changed SimpleDrain interface. (Currently unused code.)

Version 0.1 (11.01.2006)

first release

ToDo, known issues

Add 64 bit support: When writing from or to files there is a 2GB limit, because buffer2 does not support the 64 bit file API.
Hang if the data source is a listening TCP/IP port or a listening pipe and the destination fails to initialize: When the output fails to open and the input is not yet connected the input thread does not receive the termination signal.

Contact

Suggestions, help, complaints (but not too much:-):

Original homepage: http://www.maazl.de/project/misc/buffer2.html

Command line	Description
`-b=size`	Set the size of the FIFO-Buffer. `size` may be followed by `k`, `M` or `G` to give the size in kiB, MiB or GiB, i.e. 1024¹, 1024² or 1024³ bytes. It is strongly recommended to use a multiple of 1kiB as buffer size. Otherwise the performance may degrade significantly. By default the buffer size is 64kiB.
`-p=size`	Set the size of the Pipe-Buffer if a pipe is created by `buffer2`. `size` may be followed by `k` to give the size in kiB, i.e. 1024 bytes. Note that the pipe buffer must not exceed 64kiB. This is a limitation of OS/2. By default the pipe buffer size is 8kiB.
`-r=size`	Maximum request size. `size` may be followed by `k`, `M` or `G` to give the size in kiB, MiB or GiB, i.e. 1024¹, 1024² or 1024³ bytes. By default the request size is automatically calculated. For buffer sizes below 256kiB one quarter of the buffer size is used, for buffers sizes of 256kiB or more one eighth of the buffer size is used. It is recommended that the maximum request size is a factor of the buffer size to avoid fragmentation. Furthermore the request size should not be larger than half of the buffer size. Otherwise the buffer becomes very ineffective because large parts of the buffer may be blocked by either the input or the output side.
`-h=level`	High water mark. Once the buffer gets empty the output side of the buffer will stop working until the buffer is filled up to the high water mark. `level` may be followed by `k`, `M` or `G` to give the high water mark in kiB, MiB or GiB, i.e. 1024¹, 1024² or 1024³ bytes. If `level` is followed by `%` it counts relative to the buffer size in percent. The default value is zero. That causes the output side to resume as soon as any data is available. This parameter may be used to defragment streaming data in pipes or to avoid unnecessary stops of tape drives or CD writers. In the latter cases a value of `100%` may be reasonable. This causes nothing to be written until the buffer is full (or the end of the input stream is reached, of course).
`-l=level`	Low water mark. This is the opposite of the high water mark. Once the buffer gets full it will not resume to fill the buffer before the buffer level drops below the low water mark. `level` may be followed by `k`, `M` or `G` to give the high water mark in kiB, MiB or GiB, i.e. 1024¹, 1024² or 1024³ bytes. If `level` is followed by `%` it counts relative to the buffer size in percent. The default value is `100%`. This causes the input operation to resume as soon as there is any space in the buffer.
`-c`	Enable file system caching (if source or destination is an ordinary file). This may or may not increase the throughput. Normally it is no good choice to use the file system's cache here, because the FIFO is in fact a kind of cache. Generally this will only increase the load on the CPU to memory interface.
`-si`	Print statistics from the input side of the FIFO to stderr. This option is for diagnostic purposes. Using it may stop the FIFO from spooling data, because the print functions may block, e.g. if one hits the "pause" key.
`-so`	Print statistics from the output side of the FIFO to stderr. This option is for diagnostic purposes. Using it may stop the FIFO from spooling data, because the print functions may block, e.g. if one hits the "pause" key.