pulsar-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Apache Pulsar Slack" <apache.pulsar.sl...@gmail.com>
Subject Slack digest for #general - 2019-01-25
Date Fri, 25 Jan 2019 09:11:02 GMT
2019-01-24 11:02:58 UTC - Ericko Samudera: @Ericko Samudera has joined the channel
2019-01-24 11:10:49 UTC - Ericko Samudera: Hi guys, we are looking to modify the reader to
be able to move the cursor in the reversed direction (to retrieve "chat history"). Is it on
the roadmap or could we send a PR? Thanks!
+1 : Sijie Guo
2019-01-24 11:16:25 UTC - Sijie Guo: @Ericko Samudera: it is not on the roadmap. you are welcome
to contribute the feature. if you already have the PR, send it and we can have discussion
over there; if you just start with the requirement, it would be good to send a PIP and raise
a proposal on dev@ mailing list
2019-01-24 11:17:44 UTC - Ericko Samudera: okay, we will send the PIP soon, thanks
100 : Sijie Guo
+1 : Sijie Guo
2019-01-24 14:26:03 UTC - Vincent Ngan: Can anyone explain to me the purpose of using a Pulsar
Proxy? The documentation also suggests the use of frontend load-balancer such as HAProxy.
This makes it quite confusing to me because HAProxy can also be used as a proxy to load-balance
the backend servers. So, why don’t we just configure the HAProxy to load-balance the brokers
directly. In this case, we do not need an additional Pulsar Proxy.
2019-01-24 14:48:30 UTC - dba: Hi. We have a group of Python developers (I'm not one of them)
that would like to give Pulsar Functions a try. They are developing on Windows, but when trying
to install the pulsar-client lib from PIP they cant't, because it is built on the C++ lib,
which is Linux only.
Any plans in regards to compiling a version for Windows or is there a better option?
2019-01-24 15:01:48 UTC - Ivan Kelly: @Vincent Ngan pulsar proxy gives you a single endpoint
to talk to. if you talk directly to the brokers, if a topic you want to use is assigned to
another broker, you then have to talk to that broker
2019-01-24 15:03:27 UTC - Ivan Kelly: whereas, with the proxy, you can talk directly to one
instance the whole time
2019-01-24 15:32:09 UTC - Vincent Ngan: So, that means pulsar proxy is a smart proxy which
knows where the topics are and route the requests to the correct brokers without requiring
the client to know exactly where the topics are. Originally, I thought this is already handled
by the brokers. So the location transparency of the topics is provided by the pulsar proxy
and not by the brokers. Thanks for your help.
2019-01-24 15:40:46 UTC - Grant Wu: Have you considered the Windows Subsystem for Linux?
2019-01-24 16:46:47 UTC - Ezequiel Lovelle: @Ericko Samudera Hi! I don't know your specific
use case but you have the option to reset a given position of a consumer to a previous in
time position, maybe this fits your needs.
2019-01-24 16:59:43 UTC - Joe Francis: Thats not exactly correct. Lookup is handled by client
library + brokers provided they are in the same network address space.  Any broker can then
redirect a client  to the  broker serving that topic. But if the brokers are in a private
network, and the client outside of that, then the client cannot directly access the brokers,
because the addresses are private.  Pulsar proxy,  running  on a gateway host, acts as a network
proxy  to bridge  networks
2019-01-24 17:42:53 UTC - Ambud Sharma: @Ambud Sharma has joined the channel
2019-01-24 17:55:04 UTC - Matteo Merli: @Vincent Ngan The purpose of pulsar proxy is for deployments
where client don’t have direct connectivity with brokers.

One such example is:
 * Pulsars cluster deployed in Kubernetes
 * Clients connecting from outside

In these cases, the proxy will act as a completely stateless frontend to Pulsar and it can
be exposed through a LB/ DNS endpoint.

It cannot be a generic TCP proxy, because it needs to understand pulsar protocol to route
to appropriate broker. After the initial handshake, the Pulsar proxy will degrade itself to
a simple TCP proxy.
2019-01-24 17:59:01 UTC - Matteo Merli: I think @Ericko Samudera is referring to have a reader
that goes backward, if I’m not mistaken.

It would be an interesting addition for sure! :slightly_smiling_face:
2019-01-24 18:14:21 UTC - Ezequiel Lovelle: Yeah! of course!
2019-01-24 20:03:59 UTC - Emma Pollum: I want to see how much performance increase we get
by forcing clients to use the pulsar protocol. I was hoping to use the openmessaging framework.
Is there a way to force it to use http over pulsar?
2019-01-24 20:05:50 UTC - Matteo Merli: When you specify `<pulsar://localhost:6650>`
or `<http://localhost:8080>` the only difference is in the service discovery phase.
After that, the data path will always go to pulsar protocol
2019-01-24 20:23:31 UTC - Grant Wu: I think all the official clients always use the Pulsar
2019-01-24 20:24:31 UTC - Matteo Merli: Yes, also that server only support Pulsar protocol
:slightly_smiling_face: with the exception of proxies that can be used in front of it.
2019-01-24 21:43:30 UTC - Yu Yang: @Yu Yang has joined the channel
2019-01-25 02:41:55 UTC - bossbaby: i have 1 topic, many broker in the world and many consumer
&amp; producer in many server in the world connect, throughtput in each producer and consumser
are differently. What is the best solution to optimal my system?
2019-01-25 03:09:14 UTC - Ali Ahmed: @bossbaby can you clarify your question with more details
2019-01-25 03:22:08 UTC - bossbaby: ( my server can't use partition topic becase of not support
ackcumulative to optimal). i have a topic and
1 topic only 1 broker, so if my service have a many many consumer &amp; producer connect
and produce &amp; consume, bottleneck will happen. What is the best solution to solve
2019-01-25 04:02:31 UTC - Matteo Merli: I’d suggest to just use individual acknowledgments
with a partitioned topic
2019-01-25 04:10:25 UTC - bossbaby: The partition topic has many restrictions such as can't
seek with earlier, ack ackcumulative and acknowledgments will cause the system to be slow.
2019-01-25 04:12:27 UTC - Matteo Merli: I don’t think there’s a particular difference
in performance between ack and cumulative acks 
2019-01-25 04:12:53 UTC - bossbaby: because there are some messages that do not need confirmation
2019-01-25 04:13:04 UTC - Matteo Merli: There are several ways in the individual acks are
grouped together for efficiency 
2019-01-25 04:13:15 UTC - Matteo Merli: Both in client and in broker side 
2019-01-25 04:13:28 UTC - Matteo Merli: And both are configurable
2019-01-25 04:15:01 UTC - Matteo Merli: By default client will batch acks at a 100ms group
2019-01-25 04:18:10 UTC - bossbaby: each time sending a ack, the consumer will spam the server,
this is problem
2019-01-25 04:19:54 UTC - bossbaby: i am setting, each 5s consumer will send ack to server
2019-01-25 04:20:49 UTC - Matteo Merli: What I’m saying is that you can automatically get
the same behavior with individual acks
2019-01-25 04:21:34 UTC - Matteo Merli: Take a look at <http://pulsar.apache.org/api/client/org/apache/pulsar/client/api/ConsumerBuilder.html#acknowledgmentGroupTime-long-java.util.concurrent.TimeUnit->
2019-01-25 04:24:02 UTC - Matteo Merli: That will make the client to send the acks in batches
2019-01-25 04:26:37 UTC - bossbaby: sorry @Matteo Merli, i am using pulsar client c++ and
it not support acknowledgmentGroupTime
2019-01-25 04:27:02 UTC - Matteo Merli: Oh, I see
2019-01-25 04:28:37 UTC - bossbaby: acknowledgmentGroupTime will be available in pulsar client
c++ 2.3.0 ?
2019-01-25 04:32:24 UTC - Matteo Merli: We don’t have time to do it for 2.3
2019-01-25 04:32:40 UTC - Matteo Merli: Though it could be done in2.4
2019-01-25 04:50:31 UTC - bossbaby: thanks you merlimat, i think i must use Pulsar Geo-Replication
to solve problem with normal topic
2019-01-25 05:20:15 UTC - Toshiaki Irie: @Toshiaki Irie has joined the channel
2019-01-25 07:21:50 UTC - dba: No, I think we were hoping the C++ lib was written in standard
C++, so it could easily be compiled for Windows and that Pulsar would be interested in doing
so :slightly_smiling_face:
2019-01-25 07:29:05 UTC - Ali Ahmed: @dba you can try using a linux docker image for development,
there are windows users who do that. we also have mac binaries , supporting windows is an
expensive proposition which we don’t have enough people for at this time.
2019-01-25 07:36:37 UTC - dba: @Ali Ahmed I don't think I understand fully, so maybe you can
help me. If the C++ code is 100% Standard C++, wouldn't it then just be a matter of compiling
and releasing the C++ lib for Windows? That would make the Pulsar experience for C++, Go and
Python developers on Windows so much better.
Currently I am working on a C#/.NET lib, which will open Pulsar for the 10 million .NET developers
out there, but it would be nice if Pulsar could be more cross platform by embracing Windows.
2019-01-25 07:40:30 UTC - Ali Ahmed: that’s part of it’s also a question of testing .
We currently have linux machines for development and testing, I don’t know if any of the
committers are currently using windows.
2019-01-25 08:32:02 UTC - dba: Would it be worth it to find out if you could squeeze Windows
support in? Maybe some committers have a Windows machine and willing to test it? Given that
the C++ client is standard C++ and can be compiled for Windows of course.
2019-01-25 08:32:26 UTC - bossbaby: i have a question that
By default, ttl in topic = 0 but in dashboard, why backlog not detele?
2019-01-25 08:34:37 UTC - Ali Ahmed: I would recommend proposing this to the dev mailing list
see if there a support for this.
+1 : dba
View raw message