pulsar-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Apache Pulsar Slack" <apache.pulsar.sl...@gmail.com>
Subject Slack digest for #general - 2018-12-14
Date Fri, 14 Dec 2018 09:11:04 GMT
2018-12-13 10:05:20 UTC - B Senthil Kumar: @B Senthil Kumar has joined the channel
----
2018-12-13 14:58:38 UTC - Grégory Guichard: Hello ! I have a cluster of 6 bookies, when I
try to set the ensemble size of my namespace at 6, I get the message "Bookkeeper-Ensemble
must be &lt;= 5" with the pulsar-admin cli. Why ?
----
2018-12-13 15:05:46 UTC - Sijie Guo: there is a setting `managedLedgerMaxEnsembleSize` in
broker conf controlling the max bookkeeper ensemble size. by default it is 5, you can change
it.

(it is probably missing from conf/broker.conf. you can manually add it `managedLedgerMaxEnsembleSize=&lt;size&gt;`)
----
2018-12-13 15:14:09 UTC - Grégory Guichard: It works, thanks !
----
2018-12-13 19:16:43 UTC - Grant Wu: How can I get the last message published in a topic? 
The documentation seems to not be very clear on this.
I immediately thought to reach for the Reader interface, however:
<https://pulsar.apache.org/api/client/org/apache/pulsar/client/api/ReaderBuilder.html#startMessageId-org.apache.pulsar.client.api.MessageId->
this states:
&gt; MessageId.latest : Start reading from the end topic, only getting messages published
after the reader was created
which is not quite what I want.

Now <http://pulsar.apache.org/docs/en/concepts-clients/#reader-interface> has an image
which states that readers read from the latest unacked message, which doesn’t seem to match
up with the Javadoc.

And neither appears to fit my ideal, which is something that always gets the latest message,
whether or not it’s acked.
----
2018-12-13 19:19:24 UTC - Matteo Merli: @Grant Wu the website doc is a bit imprecise in this
case. The Javadoc reflects the real behavior.
----
2018-12-13 19:19:58 UTC - Matteo Merli: There’s currently no straightforward way to just
receive the last message published in the topic :confused:
----
2018-12-13 19:20:05 UTC - Grant Wu: @Jonathan Budd
sob : Jonathan Budd
----
2018-12-13 19:20:11 UTC - Grant Wu: Okay, good to know
----
2018-12-13 19:20:30 UTC - Matteo Merli: not that it’s impossible, or hard, to implement
----
2018-12-13 19:20:52 UTC - Jonathan Budd: Right, I would imagine we would just need to store
the message ID of the latest message somewhere
----
2018-12-13 19:21:26 UTC - Grant Wu: There might be some place in the Pulsar Admin interface
to get it too
----
2018-12-13 19:21:51 UTC - Matteo Merli: For sure in the stats-internal
----
2018-12-13 19:23:17 UTC - Matteo Merli: So, ideally in your case, you’d like to have a way
to position the Reader on last published message and get that and then all the other messages
after that. Correct?
----
2018-12-13 19:23:37 UTC - Grant Wu: Uh, not sure if we need the second part
----
2018-12-13 19:23:39 UTC - Matteo Merli: Something like `MessageId.latestIncluded` ?
----
2018-12-13 19:23:45 UTC - Grant Wu: But something like that
----
2018-12-13 19:24:00 UTC - Matteo Merli: well.. you can always just read one and close
----
2018-12-13 19:24:07 UTC - Grant Wu: Sure
----
2018-12-13 19:30:58 UTC - Tobias Gustafsson: Sounds more or less exactly like the use case
I have. I wrote an issue for it a couple of days ago: <https://github.com/apache/pulsar/issues/3162>
----
2018-12-13 19:37:29 UTC - Tobias Gustafsson: I have a question about the `Reader.HasNext()`
method in the Go client. I'm not really sure how to interpret the documentation. If this method
returns false, does it mean that there is no more published message on the topic or that there
simply is no more message in the local buffer ready for immediate delivery? The javadoc for
the corresponding method is the exact same btw.
What I'm after is the former but it seems to me like its the latter since it sometimes returns
false even though I know that there are messages published.
----
2018-12-13 19:45:04 UTC - Matteo Merli: @Tobias Gustafsson `Reader.HasNext()` can be used
to read all the messages until the reader is caught up with the producers.
----
2018-12-13 19:45:20 UTC - Matteo Merli: Of course, producers can still publish messages after
that
----
2018-12-13 19:45:48 UTC - Matteo Merli: but when `HasNext() == False`, at least for a moment
the reader will be have read everything
----
2018-12-13 19:50:29 UTC - Tobias Gustafsson: OK, then I think I've found a bug in the Go client.
I've created a test case where I repeatedly create a Reader instance starting from `pulsar.EarliestMessage`,
 and consume all messages on it until `HasNext()` returns false. After that the reader is
closed. Every now and then, rougly 1 -5 times / 100 reads, `HasNext()` returns false on the
first call.
----
2018-12-13 19:51:34 UTC - Tobias Gustafsson: I know that there are messages on the topic since
I'm reading from the exact same topic every time. There have not been any new messages published
to the topic in the meantime, it's basically static.
----
2018-12-13 19:52:48 UTC - Matteo Merli: Yes, looks like a bug
----
2018-12-13 19:53:04 UTC - Matteo Merli: please open an issue
----
2018-12-13 19:53:04 UTC - Tobias Gustafsson: OK, I'll file an issue, thanks
----
2018-12-13 19:53:09 UTC - Matteo Merli: :+1:
----
2018-12-13 21:06:24 UTC - Sundaram Surampudi: @Sundaram Surampudi has joined the channel
----
2018-12-13 23:00:53 UTC - Craig Yoshioka: @Craig Yoshioka has joined the channel
----
2018-12-14 00:17:45 UTC - Yang Cui: @Yang Cui has joined the channel
----
2018-12-14 00:33:41 UTC - Craig Yoshioka: I have a newb question... I'm confused by the "path"
syntax for creating topics in the python client.  All the documentation I've found refers
to topic urls being of the form: `<persistent://tenant/namespace/topic>`, but if I try
to use that from python: `client.create_producer('<persistent://team/app1/events>')`
I get a "Error Checking/Getting Partition Metadata while creating producer".
----
2018-12-14 00:34:25 UTC - Sijie Guo: did you create tenant `team` and namespace `app1`?
----
2018-12-14 00:34:37 UTC - Craig Yoshioka: yup
----
2018-12-14 00:35:50 UTC - Craig Yoshioka: `pulsar-admin namespaces list team` returns: team/app1
----
2018-12-14 00:36:37 UTC - Sijie Guo: what is the version of python client are you using?
----
2018-12-14 00:36:59 UTC - Craig Yoshioka: 2.2.0
----
2018-12-14 00:39:05 UTC - Sijie Guo: what is the error message after “while creating producer”?
----
2018-12-14 00:40:43 UTC - Craig Yoshioka: 2018-12-13 16:40:15.663 INFO  ConnectionPool:63
| Created connection for <pulsar://pncc-storage-1:6650>
2018-12-13 16:40:15.666 INFO  ClientConnection:285 | [10.125.80.69:58762 -&gt; 10.125.80.69:6650]
Connected to broker
2018-12-13 16:40:15.674 ERROR ClientConnection:709 | [10.125.80.69:58762 -&gt; 10.125.80.69:6650]
Failed partition-metadata lookup req_id: 1 error: 1
2018-12-13 16:40:15.674 ERROR ClientImpl:178 | Error Checking/Getting Partition Metadata while
creating producer on <persistent://team/app1/raw> -- 5
Traceback (most recent call last):
  File "./push-file-stream.py", line 64, in &lt;module&gt;
    main()
  File "./push-file-stream.py", line 55, in main
    writer = client.create_producer('<persistent://team/app1/raw>')
  File "/storage/sw/staging/conda-env/lib/python3.7/site-packages/pulsar/__init__.py", line
408, in create_producer
    p._producer = self._client.create_producer(topic, conf)
Exception: Pulsar error: ConnectError
----
2018-12-14 00:42:27 UTC - Craig Yoshioka: hmm, the python docs have an example using:
----
2018-12-14 00:42:28 UTC - Craig Yoshioka: '<persistent://sample/standalone/ns/my-topic>'
----
2018-12-14 00:42:34 UTC - Sijie Guo: `ConnectError` seems to indicate it can not connect.
----
2018-12-14 00:42:46 UTC - Sijie Guo: that is the v1 name pattern
----
2018-12-14 00:43:29 UTC - Craig Yoshioka: ok, that was one source of confusion
----
2018-12-14 00:43:34 UTC - Sijie Guo: (fyi, topic name changes between v1 and v2 : <http://pulsar.apache.org/docs/en/pulsar-2.0/#topic-names>)
----
2018-12-14 00:43:37 UTC - Craig Yoshioka: so don't use that
----
2018-12-14 00:43:52 UTC - Sijie Guo: it also works
----
2018-12-14 00:43:59 UTC - Sijie Guo: but it is just v1 naming pattern
----
2018-12-14 00:46:04 UTC - Craig Yoshioka: hmmm, so I changed the python code to: <persistent://team/standalone/app1/raw>
and now it seems to work...
----
2018-12-14 00:46:29 UTC - Sijie Guo: are you using standalone?
----
2018-12-14 00:47:09 UTC - Sijie Guo: how did you create the tenant?
----
2018-12-14 00:47:54 UTC - Craig Yoshioka: i am using standalone
----
2018-12-14 00:48:04 UTC - Craig Yoshioka: I created the tenant via pulsar-admin
----
2018-12-14 00:48:12 UTC - Sijie Guo: what is your full command?
----
2018-12-14 00:48:22 UTC - Sijie Guo: did you specify the cluster when creating tenant?
----
2018-12-14 00:48:49 UTC - Craig Yoshioka: hmm, I didn't but when I checked the tenant config
it listed standalone
----
2018-12-14 00:48:53 UTC - Craig Yoshioka: pulsar-admin tenants create team
----
2018-12-14 00:49:09 UTC - Craig Yoshioka: {
  "adminRoles" : [ ],
  "allowedClusters" : [ "standalone" ]
}
----
2018-12-14 00:49:21 UTC - Sijie Guo: okay
----
2018-12-14 00:50:04 UTC - Craig Yoshioka: I'm going to try and build the python client from
source... suspicious that the v1 pattern is working but the 2 is not
----
2018-12-14 00:51:09 UTC - Sijie Guo: v2 should be working. let me double check
----
2018-12-14 00:57:33 UTC - Nicolas Ha: on memory limits: <https://github.com/apache/pulsar/blob/master/deployment/kubernetes/helm/pulsar/values.yaml#L194-L204>
this specifies 15G of memory and 3 replicas. Is it 3 * 15G or memory that I need, or will
each replica get 5G?
----
2018-12-14 00:58:33 UTC - Craig Yoshioka: thanks for the help.  I gotta go commute, but I'll
check back in later
----
2018-12-14 00:58:44 UTC - Sijie Guo: the memory setting is per pod. so it is the former one
slightly_smiling_face : Nicolas Ha
----
2018-12-14 00:59:56 UTC - Nicolas Ha: thanks again :slightly_smiling_face:
----
2018-12-14 01:03:48 UTC - Nicolas Ha: Also, is it expected to only have 1 bookie? <https://github.com/apache/pulsar/blob/master/deployment/kubernetes/generic/bookie.yaml>
I do not see `replicaCount` in this file
----
2018-12-14 01:08:04 UTC - Sijie Guo: @Craig Yoshioka: so I think you forgot to specify cluster
when creating namespace. you should do `bin/pulsar-admin namespaces create --clusters standalone
team/app`.
----
2018-12-14 01:09:42 UTC - Sijie Guo: the generic one is using daemonset deployment. so it
deploys to one pod per node.

if you are deploying to a cloud environment, I would suggest you using helm chart or <https://github.com/apache/pulsar/tree/master/deployment/kubernetes/google-kubernetes-engine>,
as it will use statefulset for deploying bookkeeper.
----
2018-12-14 01:12:32 UTC - Nicolas Ha: ah I see :slightly_smiling_face: I am deploying using
the `generic` because I am on “bare metal” kubernetes
----
2018-12-14 01:12:49 UTC - Sijie Guo: oh gotcha
----
2018-12-14 02:00:59 UTC - Craig Yoshioka: thanks, that worked,  guess I assumed it would inherit
from the parent tenant.  thanks again!
----
Mime
View raw message