pulsar-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Apache Pulsar Slack" <apache.pulsar.sl...@gmail.com>
Subject Slack digest for #general - 2019-01-09
Date Wed, 09 Jan 2019 09:11:02 GMT
2019-01-08 09:57:54 UTC - bossbaby: i found different in library pulsar java and c++. In ack
message by consumer, message will schedule and resend but in c++, message will resend when
system started again so that message will be lost
----
2019-01-08 11:17:44 UTC - Yuvaraj Loganathan: @Yuvaraj Loganathan set the channel topic: Pulsar
graduated as a Top-Level project :tada: - Pulsar release 2.2.1 - <https://pulsar.apache.org/release-notes/#2.2.1>
+1 : Sijie Guo, Guangzhong Yao, Bogdan BUNECI, Edward, Karthik Ramasamy
100 : Sijie Guo
tada : Sijie Guo
----
2019-01-08 11:40:03 UTC - Sijie Guo: Can you explain more about this?
----
2019-01-08 14:08:54 UTC - eolivelli: Hi, I have a question about getting last message id <https://lists.apache.org/thread.html/1e8ab9ad9718b35944440513eccba7cbbc44ac09df2086669694300d@%3Cusers.pulsar.apache.org%3E>


Can anyone help please?
----
2019-01-08 14:21:47 UTC - Ivan Kelly: @eolivelli there was an admin command added recently
to get it
----
2019-01-08 14:22:29 UTC - eolivelli: Do you have any pointer? I am still new to Pulsar. Thanks
----
2019-01-08 14:23:26 UTC - Ivan Kelly: it's only in master, <https://github.com/apache/pulsar/commit/f4936763af1c1dab54d031ae355e40d66d3a602e>
----
2019-01-08 14:23:46 UTC - Ivan Kelly: on further inspection, it looks like they added the
rest endpoint and not the admin api itsel
----
2019-01-08 14:23:49 UTC - Ivan Kelly: :confused:
----
2019-01-08 14:27:35 UTC - Sijie Guo: @Ivan Kelly the admin api was added in a subsequent change
if I remembered correctly
----
2019-01-08 14:30:33 UTC - Sijie Guo: <https://github.com/apache/pulsar/commit/94a942384b4d1e07fba4dfb1fe2cc0c06bc0a2ea>
----
2019-01-08 14:31:26 UTC - Ivan Kelly: ah, wasn't noted on the original issue
----
2019-01-08 15:52:43 UTC - Grant Wu: Is that a question?  No, it uses a more complicated scheme;
<https://zookeeper.apache.org/doc/current/zookeeperOver.html> might help?
----
2019-01-08 15:57:13 UTC - bossbaby: i run my consumer client and get message from topic.
In java, im testing by consumer get message "A" and not commit ack to broker then about 30s,
consumer receive message "A" again and continue it unless consumer commit ack
In c++, same as java but consumer will receive message "A" (not ack to broker) when client
reconnect to broker
----
2019-01-08 19:20:15 UTC - Ryan Samo: Have you guys ever used Digicert as a root CA instead
of creating your own certificate authority?
----
2019-01-08 19:23:01 UTC - Matteo Merli: Mostly used with custom created CA. We had the idea
to include a set of default CA certs to ship with client library (eg: LetsEncrypt) to simplify
TLS setup on client side
----
2019-01-08 19:30:16 UTC - Ryan Samo: I guess I’m just wondering if it is more secure by
using your own root CA via the Pulsar documentation vs a Digicert since anyone can get a Digicert
root and create a client with a CNAME match and their in? Is that right?
----
2019-01-08 19:31:09 UTC - Matteo Merli: Never used Digicert so not sure how’s that model
----
2019-01-08 19:33:06 UTC - Ryan Samo: Enterprise security always turns their nose up self signed
certs so just wondered if you guys had any experience with that. Pulsar came out of Yahoo
right? I guess they were comfortable with self signed :)
----
2019-01-08 19:33:36 UTC - Matteo Merli: That’s not self-signed though. You have an internal
CA that’s created
----
2019-01-08 19:33:49 UTC - Matteo Merli: all certificates are created and signed with that
CA
----
2019-01-08 19:34:28 UTC - Matteo Merli: Self-signed means no CA is involved (and there’s
no way to verify the identity of the peer)
----
2019-01-08 19:35:11 UTC - Ryan Samo: Ok makes sense, that’s the ammo we need perfect, makes
sense. 
----
2019-01-08 19:36:09 UTC - Matteo Merli: The question is only that maintaining that CA and
generating TLS certs with OpenSSL is a bit painful.. but that’s how it is
----
2019-01-08 19:37:30 UTC - Ryan Samo: Yup you’re right, but works like a champ
----
2019-01-08 20:03:55 UTC - Karthik Palanivelu: Team, I am launching broker on a Kubernetes
Platform. Can you please let me know how I can expose hostIP as Advertised address in a Deployment
mode? PodStatus.hostIP is not working.
----
2019-01-08 20:05:11 UTC - Matteo Merli: There are several way to expose brokers outside Kubernetes
cluster.

The easiest way it use Pulsar proxy and attach a load-balancer to the proxy service.
----
2019-01-08 20:05:36 UTC - Matteo Merli: Then client can just use the LB url as the Pulsar
service URL
----
2019-01-08 20:09:59 UTC - Karthik Palanivelu: I understand, but I am trying Sync Replication
between two K8s clusters in which broker has to be exposed via the discoverable IP. So I am
using hostPort, which will have a broker per node on a port 6651. But still it is using PodIp
and is connect time out to reach to the other broker. If I expose broker on host IP, I should
be able to reach. Is there a way to do in Deployment Kind, it is possible in Daemonsets using
status.hostIP.
----
2019-01-08 20:11:13 UTC - Karthik Palanivelu: In this case I do not need proxy
----
2019-01-08 20:18:31 UTC - Matteo Merli: Oh, ok. In that case, you need to pass down the hostIP
and tell the broker to advertise that address
----
2019-01-08 20:19:07 UTC - Matteo Merli: This is done by setting `advertisedAddress=1.2.3.4`
in `broker.conf`.
----
2019-01-08 20:19:16 UTC - Matteo Merli: Let me search if I can find an example
----
2019-01-08 20:34:53 UTC - Karthik Palanivelu: Is there a way to pass this hostIP via the yaml
file like env variables from K8s?
----
2019-01-08 20:58:37 UTC - Matteo Merli: @Karthik Palanivelu Take a look at <https://github.com/apache/pulsar/blob/master/deployment/kubernetes/generic/bookie.yaml#L78>
----
2019-01-08 20:58:46 UTC - Matteo Merli: It’s for bookies, but it’s the same thing
----
2019-01-08 21:01:56 UTC - Matteo Merli: to be applied for brokers
----
2019-01-08 21:29:41 UTC - Karthik Palanivelu: Yes if I change to daemonset it will work but
I am trying as deployment 
----
2019-01-08 21:51:35 UTC - Grant Wu: ~Out of curiosity, is there a reason why the default cluster
name is called “pulsar” when running Pulsar in the clustered mode, but it’s “standalone”
when running Pulsar in standalone mode?~
----
2019-01-08 21:52:15 UTC - Grant Wu: ~This is causing some headaches where we’re trying to
automatically create some new namespaces; we need to then figure out whether or not the deployment
is running in clustered or standalone mode in order to figure out what the correct cluster
name is.~
----
2019-01-08 21:55:28 UTC - Grant Wu: Er, hold up, it seems like this might be an artifact of
some customizations we’re doing.
----
2019-01-08 22:33:48 UTC - Grant Wu: Can someone using the Helm charts available on GitHub
tell me what the default value of `pulsar-admin namespaces get-clusters public/default` is?
----
2019-01-09 00:04:23 UTC - Matteo Merli: Sure, for brokers it will also work with a “deployment”
----
2019-01-09 00:04:52 UTC - Matteo Merli: You just need to expose the host IP in the env:

```
                env:
                  - name: advertisedAddress
                    valueFrom:
                        fieldRef:
                            fieldPath: status.hostIP
```
----
2019-01-09 02:52:06 UTC - bossbaby: i was deploy 2 VMs, each VM is 1 broker, 1 bookie, 1 zookeer.
How to setup that 1 cluster with 2 broker use 2 VMs?
----
2019-01-09 03:29:23 UTC - Vincent Ngan: Is there any way I can explicitly delete a message
from a topic so that the message will never appear again in any subscriptions to the topic?
----
2019-01-09 03:32:35 UTC - Vincent Ngan: The reason I ask this is that the acknowledgement
of the consumption of a message only applicable to the subscription the consumer is subscribing.
If another consumer subscribes to the same topic using a different subscription, that message
will be delivered again to that new consumer.
----
2019-01-09 03:33:56 UTC - Sijie Guo: Unfortunately, there is no way to do so at this moment.
Do you mind sharing more details about your use case?
----
2019-01-09 03:34:22 UTC - Vincent Ngan: But I have a use case that I need to delete (or acknowledge)
a message so that it will never be delivered again to any consumer using any subscriptions.
----
2019-01-09 03:37:39 UTC - Vincent Ngan: I explained this in another message please check.
----
2019-01-09 04:08:58 UTC - Matteo Merli: @Vincent Ngan wouldn’t a shared subscription work
for your case? Or do you want some of the messages to be delivered to all consumers?
----
2019-01-09 04:15:03 UTC - Vincent Ngan: The consumers do not share messages. Each of them
needs to see all the messages in the topic with the exception that for certain messages, when
one consumer acknowledge them, they will not appear to other consumers.
----
2019-01-09 04:16:27 UTC - Matteo Merli: (Just speculating here) couldn’t you use a different
topic for these messages?
----
2019-01-09 04:17:00 UTC - Matteo Merli: Eg, keep the 2 kind of messages separated
----
2019-01-09 04:22:33 UTC - Vincent Ngan: Actually, what I mean is that they do share messages
but not in the sense Pulsar shares messages across consumers with the same subscription. Pulsar
sharing means the messages split to multiple consumers in round-robin way. What I mean is
that I want multiple consumers to receive identical messages. I can achieve this by having
the consumers subscribing to the topic with different subscription names.
----
2019-01-09 04:24:59 UTC - Vincent Ngan: But, because the consumers are in different subscriptions,
a consumer will not be aware of the acknowledgement made by  other consumers.
----
2019-01-09 04:25:38 UTC - Matteo Merli: Correct, the “model” is decided at the subscription
level and not per each single message
bananadance : Yuvaraj Loganathan
----
2019-01-09 04:26:40 UTC - Matteo Merli: In your case you want to have multiple subscriptions
for some messages and 1 single subscription for some others 
----
2019-01-09 04:28:14 UTC - Matteo Merli: That’s why I was suggesting if it’s possible to
isolate the different types of messages between 2 different topics. On one you can use 1 subscription
per consumer and in the other 1 single shared subscription across all consumers
----
2019-01-09 04:32:53 UTC - Vincent Ngan: No, the messages can not be classified into different
types. Any message can be acknowledged and that message once been acknowledged by a particular
consumer, it should not appear again to other consumers. But for those messages which have
not been acknowledged will be able to be delivered ( or redelivered) to other consumers.
----
2019-01-09 04:33:57 UTC - Matteo Merli: ...delivered to 1 other consumer or to all other consumers?
----
2019-01-09 04:36:38 UTC - Matteo Merli: In the first case it is exactly how a shared subscription
works. You can specify an ”ack timeout” after which if a message was not acked it will
be resent to another consumer
----
2019-01-09 04:37:52 UTC - Matteo Merli: If the message is acked by one consumer, it will not
be resent again
----
2019-01-09 04:47:23 UTC - xuyi: @xuyi has joined the channel
----
2019-01-09 05:16:32 UTC - Vincent Ngan: So, can I set ack timeout to zero?
----
2019-01-09 05:19:22 UTC - Matteo Merli: That would mean to disable the ack timeout 
----
2019-01-09 05:19:38 UTC - Matteo Merli: You can set to a small time 
----
2019-01-09 05:21:49 UTC - Vincent Ngan: It this is that case, then it almost solves my problem.
I will do some tests to verify it. Thanks a lot for your help.
----
2019-01-09 05:23:04 UTC - Matteo Merli: :+1:
----
Mime
View raw message