Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

E pub failed #20

Open
wants to merge 71 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
71 commits
Select commit Hold shift + click to select a range
8ea1a4f
port to netty 4
brainlag Feb 28, 2015
9638708
clean up code left, right and center
brainlag Mar 1, 2015
5d05d76
more code clean up
brainlag Mar 21, 2015
bc013fa
add missing files from last commit
brainlag Mar 21, 2015
a59e336
refactor connections to not close at all if possible
brainlag Apr 2, 2015
5f57ef6
add connection pooling for producers
brainlag Apr 5, 2015
52c0cfa
add builder pattern to consumer and producer
brainlag Apr 5, 2015
54176b7
add identify
brainlag Apr 20, 2015
b88fb76
prepare for release
brainlag Apr 23, 2015
ce2b1ed
prepare for release
brainlag Apr 23, 2015
38c7555
update post release
brainlag Apr 27, 2015
3e49356
add test for mpub
brainlag Apr 27, 2015
099ad94
fix parallel producer test to be actually parallel
brainlag May 1, 2015
df6f33f
move NSQLookup back to a interface so jackson json parser is not forc…
brainlag May 1, 2015
13c161b
simple retry logic when a producer doesn't get a connection
brainlag May 1, 2015
a129038
add support for ssl
brainlag May 23, 2015
3b4f3ae
add ssl + compression
brainlag May 26, 2015
fe2fc50
refactor message processing for backoff
brainlag May 2, 2015
581195e
implement backoff
brainlag Jul 8, 2015
efcf84d
test backoff
brainlag Jul 12, 2015
2366fad
Consumers were needlessly creating connections every tick. Fixed set…
moonpolysoft Sep 7, 2015
520d48e
Merge pull request #2 from moonpolysoft/fix-connection-leak
brainlag Sep 7, 2015
5dd0f61
Adding exception handling/throwing for unreachable nsqdlookup servers
Oct 1, 2015
d59c6bf
Fixing typo
Oct 1, 2015
63f9fdf
Allow NSQConsumer to add to timer
Oct 2, 2015
36970c9
Ability to add callback methods to be run with scheduler to consumer
Oct 8, 2015
e3960c6
should not fail if nsqlookupd disappears or topic does not exist yet
brainlag Oct 12, 2015
4be05ae
update dependencies
brainlag Oct 12, 2015
e3ba495
cleanup tests
brainlag Oct 12, 2015
b4e16fa
update version
brainlag Oct 12, 2015
5144eae
The EventLoopGroup can now be configured using NSQConfig.
brainlag Dec 8, 2015
82a4346
Expose the connection pool of NSQProducer because someone might want …
brainlag Dec 8, 2015
ee0df3a
Update the method shutdown
Loveflowers Dec 10, 2015
3d85e92
Merge pull request #11 from Loveflowers/Loveflowers-patch-1
brainlag Dec 10, 2015
b066545
update README
sundy-li Dec 14, 2015
3fc3f65
Merge pull request #14 from sundy-li/patch-1
brainlag Dec 14, 2015
0408f62
Improve logging.
brainlag Dec 14, 2015
e3a8b8f
Implement Closable interface for consumer
brainlag Feb 1, 2016
d5b615b
update dependencies
brainlag Feb 1, 2016
4e6318f
ship RC2
brainlag Feb 1, 2016
b38bf15
Interrupt exceptions should not lead to NPEs. Fixes #16.
brainlag Feb 27, 2016
068c11a
Don't fail silently when the consumer can't connect. There was also a…
brainlag Mar 23, 2016
fe7309e
change test resource client.key from pkcs#1 to pkcs#8, which makes al…
rukyzhc Mar 30, 2016
ab9841d
Merge pull request #18 from rukyzhc/fix-ssl
brainlag Mar 30, 2016
83bbeca
Adds URL encoding to DefaultNSQLookup to properly handle ephemeral to…
klucar May 19, 2016
4849b02
Merge pull request #22 from klucar/ephemeral_topic
brainlag May 19, 2016
269fafe
add heartbeat interval exceeded check
icanfly Sep 7, 2016
47dfc26
add topic in debug logs when lookup fails
Sep 9, 2016
02f3455
minor change on log debug
Sep 9, 2016
c0d8d4f
Parse ByteBufs correctly if no array is available.
dpapworth-qc Dec 13, 2016
46eea2f
Fix odd behaviour on reading a ByteBuf in MessageFrame.
dpapworth-qc Dec 13, 2016
b29255f
Update Netty to 4.0.39.Final.
dpapworth-qc Dec 13, 2016
6f7c065
Merge pull request #29 from garnaud/master
brainlag Dec 17, 2016
3fa8d23
Merge pull request #28 from icanfly/master
brainlag Dec 17, 2016
68217e0
Merge pull request #31 from dpapworth/fix_netty
brainlag Dec 17, 2016
855a6e6
Added the `TOUCH` command to NSQMessage to allow long running consume…
pfeairheller Jul 2, 2016
a35650e
Allowing NSQD and NSQLOOKUPD hostnames to be discovered from the envi…
pfeairheller Jul 16, 2016
c9dceaf
Use Executor instead of ExecutorService for simplier use
Jul 18, 2016
19f4f83
Refactor NSQCommand creation
Dec 26, 2016
d6aa4c8
DefaultNSQLookup: accept an ObjectMapper
Dec 26, 2016
3994e23
NSQCommand: only allow valid instances with creators
Dec 26, 2016
9af016c
Merge pull request #32 from esiqveland/refactor-nsqcommand
brainlag Jan 6, 2017
7266d9a
minor code cleanup/fixes
brainlag Jan 6, 2017
6947bff
fix byte buffer leak
brainlag Feb 20, 2017
bf4e041
ship RC3
brainlag Feb 20, 2017
329b747
Limit max in flight
Feb 21, 2017
9f5d767
37: provide compatibility with nsq1.0.0-compact, ensure backward comp…
vikash-tiwari Mar 31, 2017
d0e6612
ship RC4
brainlag Apr 5, 2017
6fc435f
close connection after consumer close
Sep 29, 2018
c005044
Update NSQProducer.java
shaymos Mar 10, 2019
8c96bd5
Merge pull request #1 from shaymos/shaymos-patch-1
shaymos Mar 10, 2019
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 0 additions & 11 deletions .classpath

This file was deleted.

11 changes: 5 additions & 6 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -1,10 +1,9 @@
*.class

build/classes
build/jar
build/build.num
target
.idea
*.iml
.classpath
.project

# Package Files #
*.war
*.ear
pom.xml.versionsBackup
17 changes: 0 additions & 17 deletions .project

This file was deleted.

1 change: 1 addition & 0 deletions LICENSE
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
The MIT License (MIT)

Copyright (c) 2013 Dustin Norlander
Copyright (c) 2015 Peter Nimmervoll

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
Expand Down
80 changes: 22 additions & 58 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,87 +1,51 @@
## TrendrrNSQClient
## JavaNSQClient

A fast netty-based java client for [NSQ][nsq] initially developed and maintained by [@dustismo](https://github.com/dustismo)
for use in various places in the trendrr.tv and curatorr.com stacks.
A (fast?) netty-based Java8 client for [NSQ](https://nsq.io)
heavily forked of TrendrrNSQClient.

**I ([@mreiferson](https://github.com/mreiferson)) am currently maintaining this repo to facilitate
bug fixes and minor updates and am interested in finding a longer-tem maintainer.**
## Artifact

```
<dependency>
<groupId>com.github.brainlag</groupId>
<artifactId>nsq-client</artifactId>
<version>1.0.0.RC4</version>
</dependency>
```

## Consumer

Example usage:

```
NSQLookup lookup = new NSQLookupDynMapImpl();
lookup.addAddr("localhost", 4161);
NSQConsumer consumer = new NSQConsumer(lookup, "speedtest", "dustin", new NSQMessageCallback() {

@Override
public void message(NSQMessage message) {
NSQLookup lookup = new DefaultNSQLookup();
lookup.addLookupAddress("localhost", 4161);
NSQConsumer consumer = new NSQConsumer(lookup, "speedtest", "dustin", (message) -> {
System.out.println("received: " + message);
//now mark the message as finished.
message.finished();

//or you could requeue it, which indicates a failure and puts it back on the queue.
//message.requeue();
}
@Override
public void error(Exception x) {
//handle errors
log.warn("Caught", x);
}
});

consumer.start();
```


## Producer

Example usage:

```
NSQProducer producer = new NSQProducer().addAddress("localhost", 4150, 1);
producer.start();
for (int i=0; i < 50000; i++) {
producer.produce("speedtest", ("this is a message" + i).getBytes());
}
NSQProducer producer = new NSQProducer().addAddress("localhost", 4150).start();
producer.produce("TestTopic", ("this is a message").getBytes());
```

The producer also has a Batch collector that will collect messages until some threshold is reached (currently maxbytes or maxmessages) then send as a MPUB request. This gives much greater throughput then producing messages one at a time.
## Backoff

```
producer.configureBatch("speedtest",
new BatchCallback() {
@Override
public void batchSuccess(String topic, int num) {
}
@Override
public void batchError(Exception ex, String topic, List<byte[]> messages) {
ex.printStackTrace();
}
},
batchsize,
null, //use default maxbytes
null //use default max seconds
);
By default Backoff does not kick in and a consumer will eat all your memory
and CPU. To enable Backoff you have to set your own thread pool executer with:

producer.start();
for (int i=0; i < iterations; i++) {
producer.produceBatch("speedtest", ("this is a message" + i).getBytes());
}
```


## Dependancies

* [netty][netty]
* [slf4j][slf4j]
* [trendrr-oss][trendrr-oss]

Note: the trendrr-oss dependancy can easily be swapped out by implementing the com.trendrr.nsq.NSQLookup interface using a different json parser


[nsq]: https://github.com/bitly/nsq
[netty]: http://netty.io/
[slf4j]: http://www.slf4j.org/
[trendrr-oss]: https://github.com/trendrr/java-oss-lib
consumer.setExecutor(...);
```
78 changes: 0 additions & 78 deletions build.xml

This file was deleted.

Binary file removed build/nsq-client.jar
Binary file not shown.
Binary file removed lib/trendrr-oss.jar
Binary file not shown.
Loading