0.9.1

New major features

• Broker-based balanced High-Level KafkaConsumer (broker version >= 0.9)
• SASL support (client authentication) - requires Cyrus libsasl
• SSL support (client authentication, transport encryption) - requires OpenSSL
• Native Win32 support (Visual Studio)
• Broker-based quota support (KIP-13)
• Relative offsets support (KIP-31)
• Message timestamp support (KIP-32)

New enhancements

• Broker API version query support for brokers >=0.10.0 with fallbacks for older brokers. (KIP-35)
  • api.version.request - enable/disable ApiVersionRequests
  • api.version.fallback.ms - how long to use the fallback version after ApiVersionRequest failure
  • broker.version.fallback - fallback version to adapt API usage to.
  • More info: https://github.com/edenhill/librdkafka/wiki/Broker-version-compatibility
• Added broker connection max in-flight enforcement: max.in.flight.requests.per.connection
• Added config aliases "retries", "acks" (#157)
• request.required.acks/acks: added support for "all" (-1)
• Flag config properties now support explicit +addx,-dely and "all"
• Added rd_kafka_last_error() alternative to errno for legacy APIs (#571)
• Added rd_kafka_get_err_descs() to retrieve full list of error codes.
• Added rd_kafka_partitions_pause/_resume()
  • Consumer: flush local fetchq for partition(s) and stop fetching
  • Producer: dont send any ProduceRequests to broker for partition(s)
    This is also used when a rebalance_cb is registered between the time of
    rebalancing and until the application has called assign() to synchronized state.
    Prevents app from receiving messages after a rebalance but before assign()
• Keep track of app_offset: last message+1 provided to application (issue #106). This is the "next offset" for the application.
• Added log.connection.close to turn off logging of broker-triggered disconnects (issue #437)
• Added rd_kafka_mem_free()
• Formal C and C++ API docs (doxygen)
• Added RdKafka::Message::topic_name().
  Allows retreiving message topic (name) for auto consumed topics where we dont have a local Topic object
• Add C++ Metadata API (@eugpermar and @incessantier)
• Changed topic.metadata.refresh.interval.ms to 5m, must be lower than server idle timeout
• Alias some new java client properties (bootstrap.servers, enable.auto.commit)
• Added topic-level "compression.codec" configuration
• Added "builtin.features" property to allow applications to query for built-in features (e.g., gzip, snappy, zlib)
• Add Consumer::OffsetTail(). (@bwelling)
• Removed enforce.isr.cnt - should be handled by broker now and request.required.acks=-1
• Added rd_kafka_topic_opaque()
• Added rd_kafka_*_conf_get() for querying conf property values on a conf object
• New high-level balanced KafkaConsumer (requires broker version >= 0.9)
• Add consumer queue support for C++. (@rthalley and @macisamuele)
• Fixed alignment errors on Sparc, et.al.
• Increased topic.metadata.refresh.interval.ms to the official default 10 minutes (#363)
• Added config property 'socket.blocking.max.ms' to control max socket poll time
• Added lazy producer message offset reporting
  When 'produce.offset.report' is off, produced offsets will still be reported or the last message in each produced batch.
  This means an application can still keep track of offsets (though a bit more roughly) without the cost of per-message offset reports.
• Added rd_kafka_yield() to break out of callback loops
• Propagate UNKNOWN_TOPIC and UNKNOWN_PARTITION to consumers (issue #348)
• Handle broker name and nodeid updates (issue #343)
  This also migrates bootstrap brokers to proper brokers if they can be exactly matched by name.
• Added consume.callback.max.messages (issue #248)
• Added C++ Consumer::consume_callback as front-end of rd_kafka_consume_callback (kodekarl)
• Add a built in consistent hashing partitioner (@NZKoz)
• Add ability to specify a CRL for checking remote certificates (@vincentbernat)

Fixes

• Preserve configured partition.assignment.strategy order
• Final offset commits on consumer close() are now limited to session.timeout.ms
• rd_kafka_log_print should print to stderr, not stdout
• C++ unassign() assigned an empty list, which is not an unassign..
• Dont fail/retry unsent requests on broker failure, leave them in queue.
• Portability fixes (FreeBSD, Win32, MUSL libc, SunOS, AIX)
• C++: added virtual destructors to all classes that have a virtual function (@danny794)
• Consumer: fix race condition with start(logical_offset) quickly followed by seek(abs_offset)
• Maintain per-partition op-version in Fetch requests (issue #586)
• Richer CONFIGURATION.md format (supported values, value type)
• Fixed a number of possible hang-on-termination scenarios.
• Improved handling of failed broker requests (exposed by #529)
• Fixed cases where a timeout_ms of 0 would serve precisely timed events forever
• Fixed buffer retry counting
• Dont rely on partition leader broker to be set when querying metadata (issue #518)
• Ignore intermittent Topic Metadata failures while brokers are rebalancing. (issue #513)
• rd_kafka_topic_partition_available() returned 1 for leader-less partitions (issue #513). Regression since 0.8.6
• Backoff sleep on broker UP -> DOWN state transition was never called. This could trigger busy-looping when connecting to a broken broker.
• Added configurable reconnect delay with jitter: reconnect.backoff.jitter.ms (issue #403)
• Snappy endian swapping for older glibc 2.9 (issue #505, #407)
• Fix possible crash on buf retries when broker is unavailable
• Consumer errors are now rkmessage-based rather than error_cb
• Added RD_KAFKA_OFFSET_INVALID to indicate invalid offsets in various APIs
• Minimize default address caching time to 1s (issue #494)
• Added rd_kafka_errno() (issue #489)
• Proper generation and installing of pkg-config .pc files for both C and C++ libraries
• Report unsupported compression codecs (on consume) as CONSUMER_ERR ops
• RdKafka::Conf->set "socket_cb" and "open_cb" checked for wrong Conf type
• Deprecate rd_kafka_set_logger() in favour of rd_kafka_conf_set_log_cb()
• Dont reset partition count to zero on metadata topic error
  A metadata partition count of 0 is returned by the broker with errors such as "No leader broker" for a topic, since these errors are
  typically intermittent we hold on to the existing partition count.
• Emit ALL_BROKERS_DOWN if none of the initial bootstrap brokers could be added.
• Consumer offset fixes (#352, #276, #227)
• Dont loose messages when partition goes away (and there is no UA partition)
  Previously any messages in a partition's queue would be silently lost if a partition went away (and there was no UA partition).
  These messages are now properly delivery reported with UNKNOWN_PARTITION.
• Provide rkmessage->rkt topic in dr_msg_cb (#333)
• Fixed wait in rd_kafka_consume_callback_queue().
  It returned immediately if no messages were available (timeout_ms was not used).
• Fix occasional invalid json on statistics buffers (@alfred-landrum)
• dr_msg_cb: rkmessage->partition is now properly set (issue #233)
• Renamed position() -> committed() and added position() that returns next offset. According to the Java client.
• Workaround Java client and broker Offset metadata parsing bugs (issue #577)
• Added rd_kafka_query_watermark_offsets() and rd_kafka_get_watermark_offsets() (issue #503, #530)
• C++: Added commitSync and commitAsync APIs taking a list of offsets (issue #528)
• Added enable.auto.offset.store (issue #528)
• sasl.kerberos.kinit.cmd now takes a command+args string template (issue #635)
• Send consumer group user_data as empty instead of NULL to workarund broker 0.9.0.0 bug (issue #660)

0.8.6

Mainly a maintenance release with 15 enhancements and 25 fixes.

Enhancements

• NULL messages (and keys) are now supported (@secretmike)
• Switched to use zlib crc functions for increased performance (@tburmeister)
• Added 'group.id' config property to topic config (overrides global 'group.id')
• Include group.id in offset file (issue #199). Escape offset filenames.
• C++: Implemented Message::key() (@KodeKarl).
• Enabled OSX builds for travis-ci
• Added "consumer_lag", "lo_offset", "hi_offset", "stored_offset" to stats output.
• Added conf: delivery.report.only.error: only trigger dr_cb() for errored messages.
• Added "internal.termination.signal" property (default off)
• rdkafka_performance: added latency measurement mode (-l) and offset reporting (-O)
• rdkafka_performance: Added '-A ': write per-message latency to file
• rdkafka_performance: send NULL key when -s 0
• rdkafka_performance: print key
• rdkafka_performance: Make -c work in consumer mode to limit number of messages consumed
• rdkafka_performance: Print received message content when verbosity is >= 3

Fixes

• Offset storage fixes (issue #191)
• Extern C-ilize rdkafka.h (issue #216)
• Revert request.required.acks limit (-1..1) back to (-1..1000) (issue #208)
• Avoid excessive topic leader queries when a broker goes down
• Renamed RdKafka::Producer::MSG_COPY and MSG_FREE to RK_MSG_COPY and RK_MSG_FREE (issue #198)
• rdkafka_performance: Better usage error reporting
• Avoid re-inclusion of rdkafkacpp.h (issue #195)
• rdkafka_performance: proper stats_fp closing
• Fix mismatches in broker_keep and broker_destroy that cause brokers to leak (#193) (@thelamb)
• Silence recurring broker resolve failures (#186)
• Added target dependencies for "make -j". (@KodeKarl)
• Dont log broker signal interrupts when terminating
• Fixed double-free of conf (defect added in ce8cd60)
• mklove update for mktemp (issue #176)
• Dont leak topic_conf memory when already added topic is passed to rd_kafka_topic_new()
• NULL conf pointer after free (@oldmantaiter)
• remove "executable" attribute from source files (@rgerhards)
• Dont require auto.commit.enable for stored offsetting (issue #171)
• Drop earlier (unwanted) messages from fetched messagesets
• Increased message.max.retries to allow long-time offline queueing.
• Library ordering (fixes Ubuntu ppc builds)
• Require linkage
• Make sure to store EOF offset on termination (issue #164)
• If RD_KAFKA_OFFSET_TAIL() rewinds past offset 0, cap it to 0.
• OSX requires linking required libs when creating dylib (issue #161)
• Use proper .dylib install_name on OSX

Thanks to @tburmeister, @thelamb, @KodeKarl, @rgerhards, @oldmantaiter, @secretmike

0.8.5

Enhancements and bugfixes:

• Dont link C++ library with C library - avoids symbol leakage
• Fixed endian.h check (for kFreeBSD, thanks to paravoid)
• rd_kafka_topic_new() was not completely thread-safe
• Unlock timer lock when calling timer callback (and honour rk_terminate)
• Store EOF offset for empty partitions on termination (previously did not store any offset)
• Off-by-one error when calculating tv_nsec - by dalegaard
• Fix metadata cache purge when topic.metadata.refresh.interval.ms=-1 (issue #156)
• Allow 0 value for fetch.error.backoff.ms
• Use a random broker for metadata requests
• Dont crazy-spin on topic.metadata.refresh.interval.ms==-1 (issue #156)
• Fixed _BIG_ENDIAN check on Solaris Sparc - thanks to shivashis
• Now compiles on FreeBSD - thanks to rtyler
• Sparse metadata retrieval mode (consumes less bandwidth) (issue #138)
  Config property: topic.metadata.refresh.sparse = true or false
• Added "protocol" debugging (sent and received message types)

0.8.4

The 0.8.4 release introduces the new C++ interface, a configure script for increased portability and a large number of enhancements and bug fixes.

New features and enhancement

• Native C++ interface - see src-cpp/rdkafkacpp.h
• ./configure support using the mklove framework: https://github.com/edenhill/mklove
• pkg-config rdkafka.pc file is now generated during build
• Broker-based offset storage using OffsetCommit and OffsetFetch
• Added Metadata API (issues #90 and #94)
• rdkafka_example: Added -L (list) mode to list metadata
• Added batch producer rd_kafka_produce_batch() for even more performant producing (issue #112)
• Added queued.max.messages.kbytes (local consumer queue limit) (issue #81)
• Added enforce.isr.cnt to locally enforce that this value is <= cluster ISR cnt (issue #91)
• rdkafka_performance: Added -G <pipe command> to pipe stats output directly to command.
• rdkafka_performance: Added -r <msgs/s> producer rate limiting
• Added broker.address.family config property (limits broker connectivity to v4 or v6)
• Added experimental msg_dr_cb() passing a complex message struct (NOT API SAFE just yet)
• Added rd_kafka_opaque() to return the opaque pointer set by set_opaque
• Added socket_cb and open_cb config properties (issue #77) (for CLOEXEC)
• Added RPM package spec file for building RPM package - see rpm/librdkafka.spec
• Added "msgs" to toppar stats output
• rdkafka_performance: added tabular output, and more..
• Solaris/SunOS support
• Bart Warmerdam added an example how to integrate with ZooKeeper in examples/rdkafka_zookeeper_example.c
• Support for infinite message timeouts message.timeout.ms=0 (thanks to Bart Warmerdam)
• Added socket.keepalive.enable config property to enable SO_KEEPALIVE
• Added socket.max.fails to disconnect from broker on request timeouts (defaults to 3)
• Added produce.offset.report config property to propagate message offset of produced message back to application.
• Added RD_KAFKA_OFFSET_TAIL(cnt) - start consuming cnt messages from end.
• New consumer queue re-routing to pull messages from multiple topics+partitions simultaneously on the same thread with one single call. (rd_kafka_consume_start_queue())

Bug fixes

• All signals are now properly blocked by rdkafka threads - no need to set up SIGPIPE handlers
  in the application anymore.
• Fixed compilation errors and warnings on various platforms
• rdkafka_performance: use message.timeout.ms default value (issue #74)
• Added redundant but correctly spelled "committed_offset" field to stats (issue #80)
• Dont retry failed Fetch requests, wait for next intervalled Fetch (issue #84)
• rd_kafka_*conf_dump() did not properly return all types of values (namely string2int mappings)
• More consistent handling of unknown topics and partitions (issue #96)
• Handle transient partition_cnt=0 during topic auto creation (issue #99)
• Add pthread_attr_destroy() (thanks to Bart Warmerdam)
• Consuming from a cluster with more than 500 partitions would crash rdkafka, this is now fixed. (issue #111)
• Fixed MacOSX 10.9 compilation
• Use .dylib extension for shared library on MacOSX
• Fixed dead-lock race condition on metadata update (issue #131)
• Topics were not destroyed in example code (issue #130)

The OpenSuse build service is now used to ensure portability, please see
https://build.opensuse.org/package/show/home:edenhill/librdkafka
for build results and packages.

Sponsors

The fix for issue #131 (dead-lock race condition on metadata update) was sponsored by
the redBorder project (http://redBorder.org)

Big thanks goes to LonelyCoder (https://github.com/andoma) for his help in conceiving the C++ interface and throwing down Changs, sometimes simultaneously - kap kom kap!

0.8.3

New features and enhancements:

• Added fetch.message.max.bytes and receive.message.max.bytes properties.
• Added rd_kafka_errno2err() for converting errno to proper rdkafka errors (issue #39)
• Trigger RD_KAFKA_RESP_ERR__ALL_BROKERS_DOWN error when all brokers are down (issue #64)
• Immediate failure on unknown topic (issue #39)
• rdkafka_example: Added raw output (-A) and quiet (-q) options, suitable for generic output to other programs.
• Added "rxcorriderrs" to stats output (unknown correlation ids)

Fixes:

• Receiving from multiple well fed partitions will no longer cause "Invalid message size .." errors.
• Topic refcount fix - rdkafka would not fully terminate under some circumstances (issue #72)
• rd_kafka_wait_destroyed() now also waits for all rk's to go away.
• Fix Cygwin compilation, again
• Proper errcode for broker transport failures, not just __FAIL (issue #64)
• Fixed out-of-order problem for produced messages under some circumstances
• Internal topic count was not properly decreased on topic destroy
• Plugged memory leak on exit: internal broker ops queue
• Added friendly hint when message.max.bytes is configured too low.
• Fixed duplicate broker names in broker fail logs
• message.send.max.retries did not have effect, no messages were retried (for issue #74)
• Avoid flooding the broker with topic-specific metadata requests when produce fails

0.8.2

New features:

• Added automatic consumer offset management (local file)
• The application no longer needs to install a SIGPIPE handler
• Added more configuration API functions (copy, destroy, dump, ...)
• examples: Added support for canonical "end","beginning","stored" to -o offset arg
• rdkafka_example: Added -X prop=val
• Now builds on MacOSX 10.9
• Now builds on RHEL 5
• Now builds on Cygwin
• Added -I (idle) switch to rdkafka_performance

Fixes:

• Proper librdkafka.a dependency for examples
• Added 'make check' target to check availability of all build artifacts
• Dont attempt to gzip compress 0 length message chunks (issue #60)
• The application no longer needs to install a SIGPIPE handler
• Compilation warning fixes on various platforms and gcc versions
• Allow producing and consuming empty messages (issue #60)
• rdkafka_performance: -X topic... properties did not work
• Fixed decoding of snappy-java compression framing (issue #55)
• Fixed flaky Gzip decompression (issue #54)
• Fail messages for unknown partitions immediately (issue #39)
• Clarify available compression support (none, gzip, snappy)
• Allow NULL 'conf' argument to rd_kafka_topic_new()
• Use 'void *' for produce payload type instead of 'char *'
• rdkafka_performance: Count immediate produce() failures as failed messages too
• Handle per topic+partition produce errors (issue #40)
• Print error string, not number, in metadata request failure log message
• Only generate statistics if statistics.interval.ms is non-zero
• Correctly calculate number of topics for filtered Metadata requests

0.8.1

Bugfix and enhancement release.

Bugfixes:

• Correctly handle IPv6 broker addresses added by application (issue #23)
• Let produce API honour "message.max.bytes" (issue #24)
• Dont re-query metadata immediately if leader was lost (issue #28)
• Fixed recursive locking on broker fail+buf retry (issue #27)
• Compilation warning fixes on various platforms and setups
• Dont reuse broker ERR_REQUEST_TIMED_OUT if timeout was local.

Enhancements:

• rdkafka_performance consumer: call poll to handle stats callbacks.
• Reformatted stats JSON output not to contain arrays
• Added broker request RTT statistic
• Added rd_kafka_version() and rd_kafka_version_str()
• Added socket.send.buffer.bytes and socket.receive.buffer.bytes configuration properties

0.8.0

Full support for Apache Kafka 0.8:

• producer
• consumer
• compression
• broker failover support
• message reliability
• high performance