pulsar-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Apache Pulsar Slack" <apache.pulsar.sl...@gmail.com>
Subject Slack digest for #general - 2019-01-15
Date Tue, 15 Jan 2019 09:11:04 GMT
2019-01-14 16:14:03 UTC - Siva Prasad Rao Janapati: @Siva Prasad Rao Janapati has joined the
2019-01-14 16:15:14 UTC - Siva Prasad Rao Janapati: Hi, Is there any way I can consume messages
in batch from Pulsar topic? I knew Pulsar client supports batch publish.
2019-01-14 16:16:07 UTC - Matteo Merli: Messages are internally received in batches, though
at the API level are presented individually 
2019-01-14 16:17:17 UTC - Matteo Merli: If you want to do some bulk operation, you could just
receive a bunch of messages (within some small time window) and process them at once
2019-01-14 16:18:18 UTC - Siva Prasad Rao Janapati: I am using Pulsar Java client library.
In that I use comsumer's receive method which will give only one message at a time
2019-01-14 16:19:27 UTC - Siva Prasad Rao Janapati: I don't see any API to get batch of messages
available in topic from a single call to Pulsar
2019-01-14 16:20:11 UTC - Matteo Merli: Yes, I was meaning you can call receive multiple times
and then process a collection of messages at once
2019-01-14 16:21:30 UTC - Siva Prasad Rao Janapati: I did the same way as you suggested. In
my code I applied iterative logic to get specified number of messages from Pulsar. Essentially
I am making thse may calls to Pulsar.
2019-01-14 16:21:47 UTC - Siva Prasad Rao Janapati: Is that the only way I can achieve batch
consumption from Pulsar topic?
2019-01-14 16:23:50 UTC - Matteo Merli: Currently, yes. The thing is that since all these
messages are prefetched already, the calls to receive become very cheap
2019-01-14 16:25:00 UTC - Matteo Merli: I don’t think there will be a noticeable perf advantage
in getting a batch of messages in this case
2019-01-14 16:27:43 UTC - Siva Prasad Rao Janapati: ok
2019-01-14 16:30:18 UTC - Alex Mironov: @Alex Mironov has joined the channel
2019-01-14 16:32:35 UTC - Alex Mironov: Hey folks, looks like sha512 signature RPM links for
cpp client (<https://pulsar.apache.org/docs/en/client-libraries-cpp/>) are broken on
some mirrors, e.g. a mirror link for devel rpm is leading to 404: <http://ftp.tudelft.nl/apache/pulsar/pulsar-2.2.1/RPMS/apache-pulsar-client-devel-2.2.1-1.x86_64.rpm.sha512>
2019-01-14 16:42:26 UTC - Matteo Merli: I think the mirrors don’t host the signatures (these
should only be fetched from Apache servers to guarantee authenticity). If we have a link pointing
to the mirrors, we have to fix it in our website. Can you open a github issue for it? Thanks
2019-01-14 16:47:00 UTC - Alex Mironov: yep, done: <https://github.com/apache/pulsar/issues/3370>
2019-01-14 16:55:03 UTC - Ryan Samo: Hey guys and gals, have any of you looked into segregation
of ingress traffic to egress traffic? Basically making sure that producers are not sending
data into your Pulsar cluster on the same NIC as your consumers, keeping performance high.
Just wondering 
2019-01-14 17:50:30 UTC - Fredrick P Eisele: @Fredrick P Eisele has joined the channel
2019-01-14 19:16:38 UTC - Ezequiel Lovelle: @Siva Prasad Rao Janapati I think that calling
a receive method with the desired number of message to fetch from this in a single operation
vs having multiple receive() calls on different threads would have the same overhead (due
to the reasons explained by @Matteo Merli), the only thing here might be simplicity at client
side performing just one receive call, anyway, I don't think it's worth the effort to add
this functionality.
2019-01-14 19:29:29 UTC - Grant Wu: <https://pulsar.apache.org/api/client/org/apache/pulsar/client/api/ConsumerBuilder.html>
there are a bunch of tunables here
2019-01-14 19:30:25 UTC - Grant Wu: <https://pulsar.apache.org/api/client/org/apache/pulsar/client/api/ConsumerBuilder.html#maxTotalReceiverQueueSizeAcrossPartitions-int->
may be interesting
2019-01-14 19:39:22 UTC - Grant Wu: @Jerry Peng I just noticed the existence of <https://pulsar.apache.org/api/client/org/apache/pulsar/client/api/ConsumerBuilder.html#patternAutoDiscoveryPeriod-int->
Does this imply that there could be, at worst, latency between a message showing up on a new
topic that matches a topic pattern, and the message being processed by the Pulsar Function
of up to the specified period?
2019-01-14 19:56:49 UTC - Grant Wu: @Jonathan Budd
2019-01-14 19:58:29 UTC - Jerry Peng: @Grant Wu that is correct
2019-01-14 19:58:53 UTC - Grant Wu: Okay, good to know
2019-01-15 00:34:43 UTC - Vardhman: @Vardhman has joined the channel
2019-01-15 00:44:35 UTC - Vardhman: Hi I am trying to use the pulsar/pulsar-client-cpp for
a project written in C. I have a CMakeList.txt file that has a `add_subdirectory(pulsar/pulsar-client-cpp)`
to compile the pulsar-client-cpp using its CMakeList.txt file. However, when I call make (after
cmake .) in my project it gives me an error `pulsar/pulsar-client-cpp/lib/MessageRouterBase.h:25:41:
fatal error: pulsar/MessageRoutingPolicy.h: No such file or directory`. I think when it tries
to compile files that have `#include &lt;pulsar/someheader.h&gt;`  imports, it fails
since it does not know where `pulsar/...` is. Can someone help me with what needs to be modified
in the CMakeList.txt files to fix this issue? Any help would be really appreciated. Thanks!
2019-01-15 01:46:45 UTC - Grant Wu: @Jerry Peng out of curiosity does that imply that messages
sent to the topic within the first N seconds (where N is between 0 and 60, depending on the
timing) would not be processed by the Pulsar function?
2019-01-15 01:47:05 UTC - Grant Wu: Or does Pulsar have logic to retrieve the previous messages?
2019-01-15 01:47:36 UTC - Grant Wu: I know with normal subscriptions Pulsar by default always
starts from messages that arrive after the consumer is created.
2019-01-15 02:54:16 UTC - Jerry Peng: That is correct. A function will have the same behavior
as a consumer in regards with subscribing to a topic pattern 
View raw message