0.3.4

WARNING: please upgrade to v0.3.5 to address the broken nsqadmin binary.

Upgrading from 0.3.3: Binaries contain no backwards incompatible changes.

This is a quick bug fix release to fix the outdated go-nsq dependency in v0.3.3 for the bundled utilities (see 6e8504e).

0.3.3

WARNING: please upgrade to v0.3.5 to address the outdated go-nsq dependency for the
bundled utilities and the broken nsqadmin binary.

Upgrading from 0.3.2: Binaries contain no backwards incompatible changes.

This release is primarily a bug fix release after cleaning up and reorganizing the codebase. nsqadmin is now importable, which paves the way for completing #323. The bundled utilities received a few feature additions and bug fixes (mostly from bug fixes on the go-nsq side).

Features:

• #569 - nsqadmin: re-org into importable package
• #562 - nsq_to_{nsq,http}: add epsilon-greedy mode (thanks @twmb)
• #547 - nsqd: adds start_time to /stats (thanks @ShawnSpooner)
• #544 - nsq_to_http: accept any 200 response as success (thanks @mikedewar)
• #548 - nsq_to_http: read entire request body (thanks @imgix)
• #552/#554/#555/#556/#561 - code cleanup and /internal package re-org (thanks @cespare)

Bugs:

• #573 - nsqd: don't persist metadata upon startup (thanks @xiaost)
• #560 - nsqd: do not print EOF error when client closes cleanly (thanks @twmb)
• #557 - nsqd: fix --tls-required=tcp-https with --tls-client-auth-policy (thanks @twmb)
• #545 - enable shell expansion in official Docker image (thanks @paddyforan)

NOTE: the bundled utilities are built against go-nsq v1.0.4 and include all of those features/fixes.

0.3.2

Upgrading from 0.3.1: Binaries contain no backwards incompatible changes however as of this release we've updated our official Docker images.

We now provide a single Docker image nsqio/nsq that includes all of the NSQ binaries. We did this for several reasons, primarily because the tagged versions in the previous incarnation were broken (and did not actually pin to a version!). The new image is an order of magnitude smaller, weighing in around 70mb.

In addition, the impetus for this quick release is to address a slew of reconnect related bug fixes in the utility apps (nsq_to_nsq, nsq_to_file, etc.), for details see the go-nsq v1.0.3 release notes.

Features:

• #534/#539/#540 - single Dockerfile approach (thanks @paddyforan)

Bugs:

• #529 - nsqadmin: fix more #ephemeral topic deletion issues
• #530 - nsqd: fix the provided sample config file (thanks @jnewmano)
• #538 - nsqd: fix orphaned ephemeral channels (thanks @adamsathailo)

0.3.1

Upgrading from 0.3.0: No backwards incompatible changes.

This release contains minor bug fixes and feature additions.

There are a number of functionality improvements to the nsq_stat and nsq_to_file helper applications (and general support for #ephemeral topics, broken in 0.2.30).

Additionally, the TLS options continue to improve with support for setting --tls-min-version and a work-around for a bug relating to TLS_FALLBACK_SCSV (to be fixed in Go 1.5).

Features:

• #527 - nsq_stat: deprecate --status-every in favor of --interval
• #524 - nsq_stat: add --count option (thanks @nordicdyno)
• #518 - nsqd: set defaults for --tls-min-version and set TLS max version to 1.2
• #475/#513/#518 - nsqd: --tls-required can be disabled for HTTP / add --tls-min-version (thanks @twmb)
• #496 - nsq_to_file: add <PID> to filename and rotation by size/interval (thanks @xiaost)
• #507 - nsq_stat: add rates (thanks @xiaost)
• #505 - nsqd: speed up failure path of BytesToBase10 (thanks @iand)

Bugs:

• #522 - nsqadmin: fix #ephemeral topic deletion issues
• #509 - nsqd: fix diskqueue atomic rename on Windows (thanks @allgeek)
• #479 - nsqd: return output_buffer_* resolved settings in IDENTIFY response (thanks @tj)

0.3.0

Upgrading from 0.2.31: No backwards incompatible changes.

This release includes a slew of bug fixes and few key feature additions.

The biggest functional change is that nsqd no longer decrements its RDY count for clients. This means that client libraries no longer have to periodically re-send RDY. For some context, nsqd already provided back-pressure due to the fact that a client must respond to messages before receiving new ones. The decremented RDY count only made the implementation of the server and client more complex without additional benefit. Now the RDY command can be treated as an "on/off" switch. For details see #404 and the associated changes in nsqio/go-nsq#83 and nsqio/pynsq#98.

The second biggest change (and oft-requested feature!) is #ephemeral topics. Their behavior mirrors that of channels. This feature is incredibly useful for situations where you're using topics to "route" messages to consumers (like RPC) or when a backlog of messages is undesirable.

There are now scripts in the bench directory that automate the process of running a distributed benchmark. This is a work-in-progress, but it already provides a closer-to-production setup and therefore more accurate results. There's much work to do here!

A whole bunch of bugs were fixed - notably all were 3rd-party contributions! Thanks!

• #305 - #ephemeral topics
• #404/#459 - don't decr RDY / send RDY before FIN/REQ
• #472 - improve nsqd diskqueue sync strategies
• #488 - ability to filter topics by regex in nsq_to_file (thanks @lxfontes)
• #438 - distributed pub-sub benchmark scripts
• #448 - better nsqd IOLoop logging (thanks @rexposadas)
• #458 - switch to gpm for builds

Bugs:

• #493 - ensure all nsqd Notify() goroutines have exited prior to shutdown (thanks @allgeek)
• #492 - ensure diskqueue syncs at end of benchmarks (thanks @Dieterbe)
• #490 - de-flake TestPauseMetadata (thanks @allgeek)
• #486 - require ports to be specified for daemons (thanks @jnewmano)
• #482 - better bash in dist.sh (thanks @losinggeneration)
• #480 - fix panic when nsqadmin checks stats for missing topic (thanks @jnewmano)
• #469 - fix panic when misbehaving client sends corrupt command (thanks @prio)
• #461 - fix panic when nsqd decodes corrupt message data (thanks @twmb)
• #454/#455 - fix 32-bit atomic ops in nsq_to_nsq/nsq_to_http (thanks @leshik)
• #451 - fix go get compatibility (thanks @adams-sarah)

0.2.31

Upgrading from 0.2.30: No backwards incompatible changes.

This release includes a few key changes. First, we improved feedback and back-pressure when nsqd writes to disk. Previously this was asynchronous and would result in clients not knowing that their PUB had failed. Interestingly, this refactoring improved performance of PUB by 41%, by removing the topic's goroutine responsible for message routing in favor of N:N Go channel communication. For details see #437.

@paddyforan contributed official Dockerfiles that are now built automatically via Docker Hub. Please begin to use (and improve these) as the various older images we had been maintaining will be deprecated. See the updated docs.

The utility apps deprecated the --reader-opt flag in favor of --consumer-opt and nsq_to_nsq and to_nsq received a --producer-opt flag, for configuring details of the connection publishing to nsqd. Additionally, it is now possible to configure client side TLS certificates via tls_cert and tls_key opts.

As usual, we fixed a few minor bugs, see below for details.

New Features / Enhancements:

• #422/#437 - nsqd: diskqueue error feedback/backpressure (thanks @boyand)
• #412 - official Dockerfiles for nsqd, nsqlookupd, nsqadmin (thanks @paddyforan)
• #442 - utilities: add --consumer-opt alias for --reader-opt and
  add --producer-opt to nsq_to_nsq (also support configuration
  of tls_cert and tls_key)
• #448 - nsqd: improve IOLoop error messages (thanks @rexposadas)

Bugs:

• #440 - nsqd: fixed statsd GC stats reporting (thanks @jphines)
• #434/#435 - refactored/stabilized tests and logging
• #429 - nsqd: improve handling/documentation of --worker-id (thanks @bschwartz)
• #428 - nsqd: IDENTIFY should respond with materialized msg_timeout (thanks @visionmedia)

0.2.30

Upgrading from 0.2.29: No backwards incompatible changes.

IMPORTANT: this is a quick bug-fix release to address a panic in nsq_to_nsq and nsq_to_http, see #425.

New Features / Enhancements:

• #417 - nsqadmin/nsqd: expose TLS connection state
• #425 - nsq_to_nsq/nsq_to_file: display per-destination-address timings

Bugs:

• #425 - nsq_to_nsq/nsq_to_file: fix shared mutable state panic

0.2.29

Upgrading from 0.2.28: No backwards incompatible changes.

This release includes a slew of new features and bug fixes, with contributions from 8 members of the community, thanks!

The most important new feature is authentication (the AUTH command for nsqd), added in #356. When nsqd is configured with an --auth-http-address it will require clients to send the AUTH command. The AUTH command body is opaque to nsqd, it simply passes it along to the configured auth daemon which responds with well formed JSON, indicating which topics/channels and properties on those entities are accessible to that client (rejecting the client if it accesses anything prohibited). For more details, see the spec or the nsqd guide.

Additionally, we've improved performance in a few areas. First, we refactored in-flight handling in nsqd to reduce garbage creation and improve baseline performance 6%. End-to-end processing latency calculations are also significantly faster, thanks to improvements in the perks package.

HTTP response formats have been improved (removing the redundant response wrapper) and cleaning up some of the endpoint namespaces. This change is backwards compatible. Clients wishing to move towards the new response format can either use the new endpoint names or send the following header:

Accept: application/vnd.nsq; version=1.0

Other changes including officially bumping the character limit for topic and channel names to 64 (thanks @svmehta), making the REQ timeout limit configurable in nsqd (thanks @AlphaB), and compiling static asset dependencies into nsqadmin to simplify deployment (thanks @crossjam).

Finally, to_nsq was added to the suite of bundled apps. It takes a stdin stream and publishes to nsqd, an extremely flexible solution (thanks @matryer)!

As for bugs, they're mostly minor, see the pull requests referenced in the section below for details.

New Features / Enhancements:

• #304 - apps: added to_nsq for piping stdin to NSQ (thanks @matryer)
• #406 - nsqadmin: embed external static asset dependencies (thanks @crossjam)
• #389 - apps: report app name and version via user_agent
• #378/#390 - nsqd: improve in-flight message handling (6% faster, GC reduction)
• #356/#370/#386 - nsqd: introduce AUTH
• #358 - increase topic/channel name max length to 64 (thanks @svmehta)
• #357 - remove internal go-nsq dependencies (GC reduction)
• #330/#366 - version HTTP endpoints, simplify response format
• #352 - nsqd: make REQ timeout limit configurable (thanks @AlphaB)
• #340 - nsqd: bump perks dependency (E2E performance improvement, see 25086e4)

Bugs:

• #384 - nsqd: fix statsd GC time reporting
• #407 - nsqd: fix double TOUCH and use of client's configured msg timeout
• #392 - nsqadmin: fix HTTPS warning (thanks @juliangruber)
• #383 - nsqlookupd: fix race on last update timestamp
• #385 - nsqd: properly handle empty FIN
• #365 - nsqd: fix IDENTIFY msg_timeout response (thanks @visionmedia)
• #345 - nsq_to_file: set proper permissions on new directories (thanks @bschwartz)
• #338 - nsqd: fix windows diskqueue filenames (thanks @politician)

0.2.28

Upgrading from 0.2.27: No backwards incompatible changes. We've deprecated the short_id and long_id options in the IDENTIFY command in favor of client_id and hostname, which more accurately reflect the data typically used.

This release includes a few important new features, in particular enhanced nsqd TLS support thanks to a big contribution by @chrisroberts.

You can now require that clients negotiate TLS with --tls-required and you can configure a client certificate policy via --tls-client-auth-policy (require or require-verify):

• require - the client must offer a certificate, otherwise rejected
• require-verify - the client must offer a valid certificate according to the default CA or
  the chain specified by --tls-root-ca-file, otherwise rejected

This can be used as a form of client authentication.

Additionally, nsqd is now structured such that it is importable in other Go applications via github.com/bitly/nsq/nsqd, thanks to @kzvezdarov.

Finally, thanks to @paddyforan, nsq_to_file can now archive multiple topics or optionally archive all discovered topics (by specifying no --topic params and using --lookupd-http-address).

New Features / Enhancements:

• #334 - nsq_to_file can archive many topics (thanks @paddyforan)
• #327 - add nsqd TLS client certificate verification policy, ability
  to require TLS, and HTTPS support (thanks @chrisroberts)
• #325 - make nsqd importable (github.com/bitly/nsq/nsqd) (thanks @kzvezdarov)
• #321 - improve IDENTIFY options (replace short_id and long_id with
  client_id and hostname)
• #319 - allow path separator in nsq_to_file filenames (thanks @jsocol)
• #324 - display memory depth and total depth in nsq_stat

Bug Fixes:

• nsqio/go-nsq#19 and nsqio/go-nsq#29 - fix deadlocks on nsq.Reader connection close/exit, this
  impacts the utilities packaged with the NSQ binary
  distribution such as nsq_to_file, nsq_to_http,
  nsq_to_nsq and nsq_tail.
• #329 - use heartbeat interval for write deadline
• #321/#326 - improve benchmarking tests
• #315/#318 - fix test data races / flakiness

0.2.27

Upgrading from 0.2.26: No backwards incompatible changes. We deprecated --max-message-size in favor of --max-msg-size for consistency with the rest of the flag names.

IMPORTANT: this is another bug-fix release to address an issue in nsqadmin where templates were incompatible with older versions of Go (pre-1.2).

• #306 - fix nsqadmin template compatibility (and formatting)
• #310 - fix nsqadmin behavior when E2E stats are disabled
• #309 - fix nsqadmin INVALID_ERROR on node page tombstone link
• #311/#312 - fix nsqd client metadata race condition and test flakiness
• #314 - fix nsqd test races (run w/ -race and GOMAXPROCS=4) deprecate --max-message-size